Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/18849
> This is how we behave in CREATE TABLE.
Yes, and I'm not advocating changing that. That is fine and that is correct.
The problem is what to do after the table has already been created. At that
point, "Hive compatibility" is already a property of the table. If you break
it, you might break a Hive application that was able to read from the table
before. So it's wrong, in my view, to change compatibility at that point.
If that is not the point of "Hive compatibility", then there is no point in
creating data source tables in a Hive compatible way to start with. Just always
create them as "not Hive compatible" because then Spark is free to do whatever
it wants with them.
At best, you could implement the current fallback behavior, but only if
it's a data source table. It is just wrong to fallback to the exception
handling case for normal Hive tables. But even then, that sort of make the case
for storing data source tables as Hive-compatible rather flimsy.
> In addition, we should avoid introducing a flag just for fixing a
specific scenario.
The flag is not for fixing this specific scenario. The flag is for checking
the Hive compatibility property of the table, so that code can make the correct
decisions when Hive compatibility is an issue - like it's the case for "alter
table".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
