[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18849 Thanks! Merging to master/2.2 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18849 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18849 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80935/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18849 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18849 **[Test build #80935 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80935/testReport)** for PR 18849 at commit [`cef66ac`](https://github.com/apache/spark/commit/cef66ac783a096cc5c623fa52ac9276321e554f9). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18849 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18849 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80936/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18849 **[Test build #80936 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80936/testReport)** for PR 18849 at commit [`c41683a`](https://github.com/apache/spark/commit/c41683a907cfd9dad9b16d462e7d8f6b5f78c200). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18849 **[Test build #80936 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80936/testReport)** for PR 18849 at commit [`c41683a`](https://github.com/apache/spark/commit/c41683a907cfd9dad9b16d462e7d8f6b5f78c200). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18849 **[Test build #80935 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80935/testReport)** for PR 18849 at commit [`cef66ac`](https://github.com/apache/spark/commit/cef66ac783a096cc5c623fa52ac9276321e554f9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18849 Sure, I will put it in my to-do list. Thank you very much! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/18849 That's fine if you want to do it, I'm just not signing up for actually doing it. I'm more worried about Spark actually working with a 2.1 metastore, which it currently doesn't in a few scenarios. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18849 I can see the value of maintaining hive compatibility for users who use Hive/Spark SQL together. We can do it in a separate PR. We also need to change `CREATE TABLE` for such usage scenarios. WDYT? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/18849 Alright, I give up. If you don't think it's important to maintain Hive compatibility once it's been set, and it's ok to create tables that have completely messed up metadata (from Hive's perspective) as long as they're data source tables, I'll do that. I'd rather fix the actual problem that actually happens when using Hive 2.x than keep a long discussion about what does it mean to be compatible... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18849 > If that is not the point of "Hive compatibility", then there is no point in creating data source tables in a Hive compatible way to start with. Just always create them as "not Hive compatible" because then Spark is free to do whatever it wants with them. For most usage scenarios of Spark native file source tables, they do not use Hive to query the tables. Thus, breaking/maintaining Hive compatibility will not affect them. Their DDL commands on the data source tables should not be blocked even if Hive metastore complains it. For Hive users who want to query Spark native file source tables, we can introduce the property like `DATASOURCE_HIVE_COMPATIBLE` for ensuring the Hive compatibility will not be broken in the whole life cycle of these tables. This property has to be manually set by users, instead of adding by Spark SQL. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/18849 > This is how we behave in CREATE TABLE. Yes, and I'm not advocating changing that. That is fine and that is correct. The problem is what to do after the table has already been created. At that point, "Hive compatibility" is already a property of the table. If you break it, you might break a Hive application that was able to read from the table before. So it's wrong, in my view, to change compatibility at that point. If that is not the point of "Hive compatibility", then there is no point in creating data source tables in a Hive compatible way to start with. Just always create them as "not Hive compatible" because then Spark is free to do whatever it wants with them. At best, you could implement the current fallback behavior, but only if it's a data source table. It is just wrong to fallback to the exception handling case for normal Hive tables. But even then, that sort of make the case for storing data source tables as Hive-compatible rather flimsy. > In addition, we should avoid introducing a flag just for fixing a specific scenario. The flag is not for fixing this specific scenario. The flag is for checking the Hive compatibility property of the table, so that code can make the correct decisions when Hive compatibility is an issue - like it's the case for "alter table". --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18849 When Hive complains it, we should still let users update the Spark-native file source tables. In Spark SQL, we do our best to make the native data source tables Hive compatible. However, we should not block users just because Hive metastore complained it. This is how we behave in [CREATE TABLE](https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala#L364-L374). If users really require reading our Spark-native data source tables from Hive, we should introduce a SQLConf or table-specific option and update the corresponding part in `CREATE TABLE` too. In addition, we should avoid introducing a flag just for fixing a specific scenario. Thus, I still think comparing the table schemas is preferred for such a fix. Could you show an example that could break it? cc @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/18849 > If ALTER TABLE makes the hive compatibility broken, the value of this flag becomes misleading. That's the whole point of the flag and what the current changes do! It takes different paths when handling alter table depending on whether the table is compatible. So if the table was compatible, it will remain compatible (or otherwise Hive should complain about the updated table, as it does in certain cases). So I really do not understand what is it you're not understanding about the patch. > When Hive metastore complained about it, we should also set it to false. Absolutely not. If you have a Hive compatible table and you try to update its schema with something that Hive complains about, YOU SHOULD GET AN ERROR. And that's what the current patch does. You should not try to mess up the table even further. The old code was just plain broken in this regard. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18849 If `ALTER TABLE` makes the hive compatibility broken, the value of this flag becomes misleading. Currently, the naming of this flag is pretty general. I expect this flag could be used for the other places in the future (besides `ALTER TABLE ADD COLUMN`). Introducing a flag is simple but maintaining the flag needs more works. That is why we do not want to introduce the extra new flags if they are not required. If we want to introduce such a flag, we also need to ensure the value is always true. That means, we need to follow [what we are doing in the CREATE TABLE code path](https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala#L364-L374). When Hive metastore complained about it, we should also set it to `false`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/18849 If the flag is set to true, then whenever an "alter table" command is executed, it will follow the "Hive compatible" path, which lets the Hive metastore decide whether the change is valid or not. So, to the best of Spark's knowledge, compatibility is maintained because Hive did not complain about it. No other table metadata (e.g. storage info) is changed by that command. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18849 If the new flag `DATASOURCE_HIVE_COMPATIBLE` is set to `true` when creating a table, are we sure it can be `true` forever? Is it a reliable flag we can trust? Is that possible the `ALTER TABLE` commands by the previous/current/future versions of Spark SQL might also change the hive compatibility? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/18849 I think there is still a lot of confusion around here about what this is fixing. I see a bunch of comments related to testing the schema for compatibility. That does not work. Schema compatibility is not the issue here; the issue is whether the table was *initially* created as Hive-compatible or not. This is the Hive metastore, not Spark, complaining, so the Spark-side schema for non-compatible tables is pretty irrelevant. The schema by itself does not provide enough information to detect whether a table is compatible or not. Even if the schema is Hive compatible, the data source may not have a Hive counterpart, or the table might have been initially created in a case sensitive session and have conflicting column names when case is ignore, or a few other things, all of which are checked at table creation time. The same checks *cannot* be done later, and should not be done. If the table was non-compatible it should remain non-compatible, and vice-versa. The only thing that is needed is a way to detect that single property of the table. You cannot do that just from the schema as has been proposed a few times here. There are two options: - use an explicit option, which is the approach I took - use some combination of metadata written by old Spark versions that tells you whether the table is compatible or not. The only thing that exists for the second one is the serde field in the storage descriptor. Spark sets it to either `None` or some placeholder that does not match the datasource serde. I use that fact as a fallback for when the property does not exist, but I think it's safer to have an explicit property for that instead of relying on these artifacts. Hope that clarifies things. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18849 Will review it today. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/18849 Ping? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18849 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80694/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18849 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18849 **[Test build #80694 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80694/testReport)** for PR 18849 at commit [`4a05b55`](https://github.com/apache/spark/commit/4a05b55b5c23755cba384cf85a0af2b802b8a9bd). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18849 **[Test build #80694 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80694/testReport)** for PR 18849 at commit [`4a05b55`](https://github.com/apache/spark/commit/4a05b55b5c23755cba384cf85a0af2b802b8a9bd). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18849 My main concern is, we should not change the write path if not necessary. For backward compatibility, the read path needs to handle all possible cases, and changing write path just adds more complexity to the read path. To be clear, if there are some new features that have to be implemented by some write path changes, we should do it. If you only wanna clean something up, changing the write path can only make it worse. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18849 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80463/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18849 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18849 **[Test build #80463 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80463/testReport)** for PR 18849 at commit [`abd6cf1`](https://github.com/apache/spark/commit/abd6cf119c0b400fda9c96fcb6432b090c2505bf). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18849 **[Test build #80463 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80463/testReport)** for PR 18849 at commit [`abd6cf1`](https://github.com/apache/spark/commit/abd6cf119c0b400fda9c96fcb6432b090c2505bf). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18849 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80432/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18849 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18849 **[Test build #80432 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80432/testReport)** for PR 18849 at commit [`7b777ed`](https://github.com/apache/spark/commit/7b777edccddf91f2dfb99b30855265284188e00b). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18849 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18849 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80423/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18849 **[Test build #80423 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80423/testReport)** for PR 18849 at commit [`6824e35`](https://github.com/apache/spark/commit/6824e358d231b7c56031c946d983cde0b89fd574). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18849 **[Test build #80432 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80432/testReport)** for PR 18849 at commit [`7b777ed`](https://github.com/apache/spark/commit/7b777edccddf91f2dfb99b30855265284188e00b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18849 **[Test build #80423 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80423/testReport)** for PR 18849 at commit [`6824e35`](https://github.com/apache/spark/commit/6824e358d231b7c56031c946d983cde0b89fd574). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18849 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18849 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80414/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18849 **[Test build #80414 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80414/testReport)** for PR 18849 at commit [`0b27209`](https://github.com/apache/spark/commit/0b272094c8a490e16066e85fc48ad59c4ccf0468). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18849 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80405/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18849 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18849 **[Test build #80405 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80405/testReport)** for PR 18849 at commit [`2f57a3c`](https://github.com/apache/spark/commit/2f57a3c1db2b4f8e58456b48bbc62ef01fa14633). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18849 **[Test build #80414 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80414/testReport)** for PR 18849 at commit [`0b27209`](https://github.com/apache/spark/commit/0b272094c8a490e16066e85fc48ad59c4ccf0468). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18849 **[Test build #80405 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80405/testReport)** for PR 18849 at commit [`2f57a3c`](https://github.com/apache/spark/commit/2f57a3c1db2b4f8e58456b48bbc62ef01fa14633). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18849 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18849 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80358/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18849 **[Test build #80358 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80358/testReport)** for PR 18849 at commit [`40ebc96`](https://github.com/apache/spark/commit/40ebc966ff8e9ad742f2cdbf631a289b8388560a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18849 **[Test build #80358 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80358/testReport)** for PR 18849 at commit [`40ebc96`](https://github.com/apache/spark/commit/40ebc966ff8e9ad742f2cdbf631a289b8388560a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18849 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80270/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18849 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18849 **[Test build #80270 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80270/testReport)** for PR 18849 at commit [`7ccf474`](https://github.com/apache/spark/commit/7ccf4743024a8a447a4b05369f6ebf237cf88c4f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18849 **[Test build #80270 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80270/testReport)** for PR 18849 at commit [`7ccf474`](https://github.com/apache/spark/commit/7ccf4743024a8a447a4b05369f6ebf237cf88c4f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/18849 This is a corrected version of #18824 after I tracked the actual failure and looked at the suggested code paths in the original review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
