[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22466 @sandeep-katta Can you update the PR title? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22466 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22466 **[Test build #96929 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96929/testReport)** for PR 22466 at commit [`7577a5a`](https://github.com/apache/spark/commit/7577a5aecd92368cc5ac1b7035d3a1a571dea3fa). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22466 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96929/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22466 **[Test build #96929 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96929/testReport)** for PR 22466 at commit [`7577a5a`](https://github.com/apache/spark/commit/7577a5aecd92368cc5ac1b7035d3a1a571dea3fa). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22466 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22466 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96766/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22466 **[Test build #96766 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96766/testReport)** for PR 22466 at commit [`86e4d50`](https://github.com/apache/spark/commit/86e4d50d27ecec914ef534e84dcc75965456a32b). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22466 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96768/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22466 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22466 **[Test build #96768 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96768/testReport)** for PR 22466 at commit [`3c2831b`](https://github.com/apache/spark/commit/3c2831bf70d4ed83414625fd3e4b7607624d09fc). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22466 **[Test build #96768 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96768/testReport)** for PR 22466 at commit [`3c2831b`](https://github.com/apache/spark/commit/3c2831bf70d4ed83414625fd3e4b7607624d09fc). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22466 **[Test build #96766 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96766/testReport)** for PR 22466 at commit [`86e4d50`](https://github.com/apache/spark/commit/86e4d50d27ecec914ef534e84dcc75965456a32b). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22466 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22466 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96750/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22466 **[Test build #96750 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96750/testReport)** for PR 22466 at commit [`63dc53a`](https://github.com/apache/spark/commit/63dc53aba545fecfe5301c7ea0b8305e60e20496). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22466 **[Test build #96750 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96750/testReport)** for PR 22466 at commit [`63dc53a`](https://github.com/apache/spark/commit/63dc53aba545fecfe5301c7ea0b8305e60e20496). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22466 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96736/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22466 **[Test build #96736 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96736/testReport)** for PR 22466 at commit [`f05d7c1`](https://github.com/apache/spark/commit/f05d7c14f3942743485c8b78ac40ee7e1d3bafa9). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22466 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22466 **[Test build #96736 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96736/testReport)** for PR 22466 at commit [`f05d7c1`](https://github.com/apache/spark/commit/f05d7c14f3942743485c8b78ac40ee7e1d3bafa9). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22466 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22466 **[Test build #96730 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96730/testReport)** for PR 22466 at commit [`7d28da0`](https://github.com/apache/spark/commit/7d28da044ca80cd2c06c7d089786bec2657d175f). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22466 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96730/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22466 **[Test build #96730 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96730/testReport)** for PR 22466 at commit [`7d28da0`](https://github.com/apache/spark/commit/7d28da044ca80cd2c06c7d089786bec2657d175f). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user sandeep-katta commented on the issue: https://github.com/apache/spark/pull/22466 I am running the same test case with hive version **1.2.1.spark2** and it is passing,can I know with what hive version CI is running and how org.apache.hive.jdbc.HiveStatement and external catalog are linked,I don't see any such code. cc @cloud-fan @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22466 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96689/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22466 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22466 **[Test build #96689 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96689/testReport)** for PR 22466 at commit [`943b7fd`](https://github.com/apache/spark/commit/943b7fddcbb52687af821aa6ca176e74af512ce4). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22466 **[Test build #96689 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96689/testReport)** for PR 22466 at commit [`943b7fd`](https://github.com/apache/spark/commit/943b7fddcbb52687af821aa6ca176e74af512ce4). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22466 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22466 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96619/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22466 **[Test build #96619 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96619/testReport)** for PR 22466 at commit [`597f8e6`](https://github.com/apache/spark/commit/597f8e6130965ffbfbdd94bfb4ce3e9c90eed794). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22466 **[Test build #96619 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96619/testReport)** for PR 22466 at commit [`597f8e6`](https://github.com/apache/spark/commit/597f8e6130965ffbfbdd94bfb4ce3e9c90eed794). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22466 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22466 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96601/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22466 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22466 **[Test build #96601 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96601/testReport)** for PR 22466 at commit [`597f8e6`](https://github.com/apache/spark/commit/597f8e6130965ffbfbdd94bfb4ce3e9c90eed794). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22466 **[Test build #96601 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96601/testReport)** for PR 22466 at commit [`597f8e6`](https://github.com/apache/spark/commit/597f8e6130965ffbfbdd94bfb4ce3e9c90eed794). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22466 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user sandeep-katta commented on the issue: https://github.com/apache/spark/pull/22466 cc @cloud-fan @srowen I have updated the code,Please review --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user sandeep-katta commented on the issue: https://github.com/apache/spark/pull/22466 seems @cloud-fan comments are valid as it will not result in any behavior change, I will update the PR accordingly WDYT @srowen --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22466 This is a behavior change and makes us different from Hive. However I can't find a strong reason to do it. It's like importing a database, but we can't automatically create table entries in the metastore when creating a database with an existing location. To me a more reasonable behavior is, fail earlier when creating a database with an existing and non-empty location. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22466 Owp, I've been misreading that several times. Right. Well by analogy, if a database has a non default LOCATION then so do it's tables, and they are treated like EXTERNAL tables. Dropping the DB means dropping the tables, and dropping those tables doesn't delete data. So should the same happen for DBs? Seems sensible, because the DB directory might not even be empty. Still I feel like I'm missing something if it only comes up in the case that two DBs have the same location, which is going to cause a bunch of other problems. But is it the right change simply because it's consistent with how dropping tables works? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22466 > That link says Hive does support EXTERNAL. What am I missing? Hive supports `EXTERNAL` only for tables, not databases. The CREATE TABLE syntax: ``` CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name ... ``` The CREATE DATABASE syntax: ``` CREATE (DATABASE|SCHEMA) [IF NOT EXISTS] database_name ... ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22466 That link says Hive does support EXTERNAL. What am I missing? Well, in any event we aren't contemplating a behavior change here. If you delete a table with LOCATION specified, what should happen? Hive would delete it I guess... unless it's EXTERNAL. Looks like Spark would delete it even when it's in an 'external' location. If these are conflated I guess we weigh the surprise in not deleting 'local' locations for data when a table is dropped, vs surprise in deleting 'external' locations for data. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22466 yea, in Spark we conflate the two and treat a table as external if location is specified. However, Hive doesn't have external database, see: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Create/Drop/Alter/UseDatabase I don't want to introduce unnecessary behavior difference from hive, and I feel it's not very useful to have external database. Although your table files can be in an existing folder, but LIST TABLES will not work. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/22466
There is ... see
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ManagedandExternalTables
I think Spark conflates the two. It's rare (?) but possible to specify a
custom location of a managed table, but, typically occurs for `EXTERNAL`
tables. So maybe this is OK.
```
private def createTable(tableIdent: TableIdentifier): Unit = {
val storage =
DataSource.buildStorageFormatFromOptions(extraOptions.toMap)
val tableType = if (storage.locationUri.isDefined) {
CatalogTableType.EXTERNAL
} else {
CatalogTableType.MANAGED
}
```
And in `SqlParser`:
```
// If location is defined, we'll assume this is an external table.
// Otherwise, we may accidentally delete existing data.
val tableType = if (external || location.isDefined) {
CatalogTableType.EXTERNAL
} else {
CatalogTableType.MANAGED
}
```
So if `LOCATION` implies `EXTERNAL` in Spark, then I get this. `EXTERNAL`
tables shouldn't be deleted.
I agree that the Hive impl doesn't seem to take this into account, on the
code paths that call `dropDatabase`. CC @andrewor14 in case he is available to
comment on the original implementaiton. WDYT @cloud-fan
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22466 I'm not sure if there is a concept called "external database" in Hive... --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22466 We should look at Spark documentation, and Hive, if any, to figure out what the right behavior is here. Spark generally follows Hive. See https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-CreateTableCreate/Drop/TruncateTable I think this is, further, conflating what `LOCATION` and `EXTERNAL` does. I agree that external DB files shouldn't be deleted, but not simply those specified by `LOCATION`. At least that is my understanding. @yhuai or @cloud-fan or @clockfly might know more. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user sandeep-katta commented on the issue: https://github.com/apache/spark/pull/22466 > See JIRA, I don't think this should be merged. I have referred Databricks doc https://docs.databricks.com/spark/latest/spark-sql/language-manual/create-database.html and implemented accordingly.Let me know if any suggesstion --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user sandeep-katta commented on the issue: https://github.com/apache/spark/pull/22466 Yes I agree 2 database should not point to same path,**currently this is the loop hole in spark which is required to fix**.If this solution is not okay ,then we can append the dbname.db to the location given by the user for e.g create database db1 location /user/hive/warehouse then the location of the DB should be /user/hive/warehouse/db1.db --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22466 See JIRA, I don't think this should be merged. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user sujith71955 commented on the issue: https://github.com/apache/spark/pull/22466 @srowen @HyukjinKwon I think this can be a risk if the location of the newly created database points to an existing one, if user drop the db both the db data will be lost . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
