[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14962 closing it as most of them were fixed in other PRs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14962 I'll update this PR after the global temp view --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14962 Maybe update this PR now? The only issue left is the atomicity issues. It sounds like we do not need to backport the fix, since the temporary view is session-specific. Right? We have to be very careful when implementing the global temporary views. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14962 Is it possible to first have a PR to fix the bugs? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14962 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65186/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14962 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14962 **[Test build #65186 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65186/consoleFull)** for PR 14962 at commit [`f421355`](https://github.com/apache/spark/commit/f42135586cdb9b71d44b9294a7f0d502a1da56ff). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14962 **[Test build #65186 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65186/consoleFull)** for PR 14962 at commit [`f421355`](https://github.com/apache/spark/commit/f42135586cdb9b71d44b9294a7f0d502a1da56ff). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14962 wow... we are allowed to insert data into temporary views... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14962 uh... yeah, that is another existing bug. : ) Let me write a test suite for all the DDL statements when the temp view with the same name exists. We might hit more bugs... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14962 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65153/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14962 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14962 **[Test build #65153 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65153/consoleFull)** for PR 14962 at commit [`e0d2427`](https://github.com/apache/spark/commit/e0d2427c114e008add4a7df65b3fb4edbb526ae8). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14962 @gatorsmile It's actually another bug in previous code. When we use `DataFrameWriter.saveAsTable` with overwrite mode, we may mistakenly drop a same-name temp view. I have updated the PR description to include it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14962 **[Test build #65153 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65153/consoleFull)** for PR 14962 at commit [`e0d2427`](https://github.com/apache/spark/commit/e0d2427c114e008add4a7df65b3fb4edbb526ae8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/14962
```
val tableName = "tab1"
withTable(tableName) {
Seq((1, 2), (3, 4)).toDF("i", "j").createOrReplaceTempView(tableName)
sql(s"select * from $tableName where i < 2")
.write.mode(SaveMode.Overwrite).saveAsTable(tableName)
sql(s"select * from $tableName").show()
}
```
@gatorsmile , why it's an issue? In this case, we create a temp view called
`tab1` with 2 rows first. Then we create a table also called `tab1` with 1 row.
Finally in select we use the name `tab1`, which refers to the temp view, it has
2 rows.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14962 > the target metastore table exists and thus tableExists returns true, but the following catalog.lookupRelation actually returns a temporary table. Before this PR, yeah, it behaves the same. After this PR, we expect `catalog.lookupRelation` returns a metastore table. However, it returns a temporary table. Thus, we might misse the detection. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14962
Found another issue we need to fix.
```
val tableName = "tab1"
withTable(tableName) {
Seq((1, 2), (3, 4)).toDF("i", "j").createOrReplaceTempView(tableName)
sql(s"select * from $tableName where i < 2")
.write.mode(SaveMode.Overwrite).saveAsTable(tableName)
sql(s"select * from $tableName").show()
}
```
Before this PR, the results are correct
```
+---+---+
| i| j|
+---+---+
| 1| 2|
+---+---+
```
After this PR, the results is incorrect.
```
+---+---+
| i| j|
+---+---+
| 1| 2|
| 3| 4|
+---+---+
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14962 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65102/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14962 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14962 **[Test build #65102 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65102/consoleFull)** for PR 14962 at commit [`a8ce27f`](https://github.com/apache/spark/commit/a8ce27f802e7ce22d2959f432ecd55f685677b76). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14962 **[Test build #65102 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65102/consoleFull)** for PR 14962 at commit [`a8ce27f`](https://github.com/apache/spark/commit/a8ce27f802e7ce22d2959f432ecd55f685677b76). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14962 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14962 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65101/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14962 **[Test build #65101 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65101/consoleFull)** for PR 14962 at commit [`01659dc`](https://github.com/apache/spark/commit/01659dc626311611173be94f62c8363ddae0860a). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14962 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14962 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65100/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14962 **[Test build #65100 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65100/consoleFull)** for PR 14962 at commit [`cec6b3e`](https://github.com/apache/spark/commit/cec6b3ea89a78997aacb38a8ee38410de7d710ae). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14962 **[Test build #65101 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65101/consoleFull)** for PR 14962 at commit [`01659dc`](https://github.com/apache/spark/commit/01659dc626311611173be94f62c8363ddae0860a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14962 **[Test build #65100 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65100/consoleFull)** for PR 14962 at commit [`cec6b3e`](https://github.com/apache/spark/commit/cec6b3ea89a78997aacb38a8ee38410de7d710ae). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14962 > the target metastore table exists and thus tableExists returns true, but the following catalog.lookupRelation actually returns a temporary table. isn't it true before this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14962 Sorry, I did not explain it clearly. In the analyzer rule [`PreWriteCheck`](https://github.com/apache/spark/blob/6d86403d8b252776effcddd71338b4d21a224f9b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala#L309-L311), the target metastore table exists and thus `tableExists` returns true, but the following `catalog.lookupRelation` actually returns a temporary table. Thus, it could report a wrong error or miss some errors. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14962 > Found a common bug in the following ALTER TABLE commands: let's fix them in separated PRs. > In addition, we need to carefully review all the places that are using these three functions. For example, the analyzer rule PreWriteCheck is using both lookupRelation and tableExists. We will have a bug after this PR. yea, we should review them carefully(I have walked through all of them). `PreWriteCheck` does have a bug, but not caused by this PR. I'm thinking about if we should also separate `lookUpRelation` to a metastore version and a temp view version. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14962 The biggest external change is in the table resolution order. Now, three common `SessionCatalog` functions `tableExists`, `getTableMetadata` and `lookupRelation` are not using the same order. `tableExists` and `getTableMetadata` do not check the existence of temp table, but `lookupRelation` always first checks the temp table. This inconsistency could be risky. That means, we allow users to create a table with the same name like a temp table. When we query the table, we will get the temp view at first. In addition, we have review all the places that are using these three functions. For example, the analyzer rule `PreWriteCheck` is [using both `lookupRelation` and `tableExists`](https://github.com/apache/spark/blob/6d86403d8b252776effcddd71338b4d21a224f9b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala#L309-L311). We will have a bug after this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14962
Found a common bug in the following ALTER TABLE commands:
```
| ALTER TABLE tableIdentifier (partitionSpec)?
SET SERDE STRING (WITH SERDEPROPERTIES tablePropertyList)?
#setTableSerDe
| ALTER TABLE tableIdentifier (partitionSpec)?
SET SERDEPROPERTIES tablePropertyList
#setTableSerDe
| ALTER TABLE tableIdentifier ADD (IF NOT EXISTS)?
partitionSpecLocation+
#addTablePartition
| ALTER VIEW tableIdentifier ADD (IF NOT EXISTS)?
partitionSpec+
#addTablePartition
| ALTER TABLE tableIdentifier
from=partitionSpec RENAME TO to=partitionSpec
#renameTablePartition
| ALTER TABLE tableIdentifier
DROP (IF EXISTS)? partitionSpec (',' partitionSpec)* PURGE?
#dropTablePartitions
| ALTER VIEW tableIdentifier
DROP (IF EXISTS)? partitionSpec (',' partitionSpec)*
#dropTablePartitions
| ALTER TABLE tableIdentifier partitionSpec? SET locationSpec
#setTableLocation
| ALTER TABLE tableIdentifier RECOVER PARTITIONS
#recoverPartitions
```
We need to issue an exception when the tableType is `VIEW`. This is not
introduced by this PR. Should we fix it here? or create a separate PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14962 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14962 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64987/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14962 **[Test build #64987 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64987/consoleFull)** for PR 14962 at commit [`0075f9d`](https://github.com/apache/spark/commit/0075f9db78579b90cebcc16da1a7381ac15d03ea). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class TempViewAlreadyExistsException(name: String)` * `class TempViewManager ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14962 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64984/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14962 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14962 **[Test build #64984 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64984/consoleFull)** for PR 14962 at commit [`7314a33`](https://github.com/apache/spark/commit/7314a33e1d5455bf551594474ff787d0ce91dcec). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class TempViewAlreadyExistsException(name: String)` * `class TempViewManager ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14962 **[Test build #64987 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64987/consoleFull)** for PR 14962 at commit [`0075f9d`](https://github.com/apache/spark/commit/0075f9db78579b90cebcc16da1a7381ac15d03ea). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14962 **[Test build #64984 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64984/consoleFull)** for PR 14962 at commit [`7314a33`](https://github.com/apache/spark/commit/7314a33e1d5455bf551594474ff787d0ce91dcec). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14962 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64973/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14962 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14962 **[Test build #64973 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64973/consoleFull)** for PR 14962 at commit [`8eb6c42`](https://github.com/apache/spark/commit/8eb6c420e4f696f015d8e5cd263814e9565291dc). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class TempViewAlreadyExistsException(name: String)` * `class TempViewManager ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14962 **[Test build #64973 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64973/consoleFull)** for PR 14962 at commit [`8eb6c42`](https://github.com/apache/spark/commit/8eb6c420e4f696f015d8e5cd263814e9565291dc). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14962 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64943/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14962 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14962 **[Test build #64943 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64943/consoleFull)** for PR 14962 at commit [`ebed732`](https://github.com/apache/spark/commit/ebed732367a9949fe3f3d3df53a6798a63670064). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class TempViewAlreadyExistsException(name: String)` * `class TempViewManager ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14962 also cc @srinathshankar , it will be very easy to implement global temp view after this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14962 **[Test build #64943 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64943/consoleFull)** for PR 14962 at commit [`ebed732`](https://github.com/apache/spark/commit/ebed732367a9949fe3f3d3df53a6798a63670064). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14962: [SPARK-17402][SQL] separate the management of temp views...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14962 cc @yhuai @gatorsmile @liancheng --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
