[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21398 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21398 I didn't make the original change so I'm refraining from making a decision about whether this should go in. Perhaps another fix is desired, perhaps the current behavior is desired, but either way this isn't an area where I'm comfortable making these kinds of decisions. (CDH is another thing altogether.) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/21398 @vanzin any decision on this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21398 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21398 I don't disagree with what you guys are saying, but I also don't have the context of why the original change was made. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user blue666man commented on the issue: https://github.com/apache/spark/pull/21398 To weigh in a bit, we're one of the larger enterprises that hit this bug, which we consider a regression going from Spark 1.6.3 (on Mesos, attempting to perform DDL to a Hive instance on CDH) to >= 2.1.x. I think many more large enterprises will hit this with Spark on K8S to any Sentry-secured Hive databases. IMO, [this commit](https://github.com/apache/spark/commit/ce13c2672318242748f7520ed4ce6bcfad4fb428), while well intended in removing some of the hacks for 2.1.0, was a reversal of Spark's sort-of-implicit-commitment to have Sentry/Hive DDL work. I agree this should be fixed upstream in Sentry too, but that would be a long wait for us: creating tables in Mesos Spark 2.x is blocking 100's of devs from upgrading to Spark 2. Basically, the hacks / workarounds had existed in the past and were working. Please add a workaround back until Sentry can fix the location issue on their side. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/21398 @vanzin @cloud-fan. Going back to the facts. The fix/work-around in Cloudera's distro obviously exists because the distro integrates with Sentry anyway. Removing the location did fix the Spark Job at the customer side. To re-phrase the last request from @cloud-fan, if we remove the explicit location what are the consequences, when do you need the explicit location for managed tables? Can this be a default behavior in general? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21398 I don't think working around a bug in Sentry is a good reason to add a hack in Spark. `HiveExternalCatalog` is already full of hacks... If you have a proposal to clean that up and can also work around the Sentry bug, that will be great. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21398 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/21398
I think you were confused by my example. It was not meant to be a
workaround for Spark users (since currently Spark will just add the location if
it's not there), it was meant to be used if creating the Sentry bug report
("this command should work but fails, and the corresponding command without the
explicit location works").
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user yuchaoran2011 commented on the issue: https://github.com/apache/spark/pull/21398 I'm not confused about anything. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21398 You said: "even when LOCATION is not specified, Sentry would still not allow the table to be created." I said that if that's the case, then your user doesn't have the needed permission. Because all my patch does is remove the location. I'm still confused about what it is that you're confused about. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user yuchaoran2011 commented on the issue: https://github.com/apache/spark/pull/21398 You said "If that's the case then your user does not have permission to create the table." What I'm saying is the user did have the permission. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21398 Yes, that's exactly what I said. I'm not sure what part is confusing you. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user yuchaoran2011 commented on the issue: https://github.com/apache/spark/pull/21398 I can double check. But I remember clearly that the user did have all the necessary privileges. Without this patch, a permission error was thrown. With this patch it worked fine. The CREATE TABLE statement didn't involve a LOCATION. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21398 If that's the case then your user does not have permission to create the table. The issue that my patch works around is a bug in Sentry where creating a table *with the location specified to the same location as it would have been automatically assigned* does not work. My patch removes the explicit location in that case, and things work. But if you don't have permissions, you'll get an error regardless of my patch. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user yuchaoran2011 commented on the issue: https://github.com/apache/spark/pull/21398 @vanzin But in Spark SQL, even when LOCATION is not specified, Sentry would still not allow the table to be created. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21398 I'm kinda neutral on what should happen, except maybe for filing a Sentry bug. I'll let @cloud-fan decide whether this makes sense in Spark. For the Sentry bug, there's an easy way to reproduce this, just run the following (either in Spark or using the old Hive CLI - not beeline nor jdbc): ``` USE some_db; CREATE TABLE foo(idx int) LOCATION 'hdfs://blah.example.com/user/hive/warehouse/some_db.db/foo'; ``` Even if the user has permissions to create table on that database, Sentry will not allow it. Works fine if you remove the explicit location. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/21398 @vanzin should we merge this or add a comment in the docs? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/21398 thank you @vanzin. Tricky thing to debug btw :) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21398 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90994/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21398 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21398 **[Test build #90994 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90994/testReport)** for PR 21398 at commit [`5f64a95`](https://github.com/apache/spark/commit/5f64a95e9557feb3366cf2ac986de6da8ce165a4). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21398 **[Test build #90994 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90994/testReport)** for PR 21398 at commit [`5f64a95`](https://github.com/apache/spark/commit/5f64a95e9557feb3366cf2ac986de6da8ce165a4). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21398 **[Test build #90992 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90992/testReport)** for PR 21398 at commit [`0f4be31`](https://github.com/apache/spark/commit/0f4be3110df5f7a92313260872994bc18685332e). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21398 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21398 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90992/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21398 **[Test build #90992 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90992/testReport)** for PR 21398 at commit [`0f4be31`](https://github.com/apache/spark/commit/0f4be3110df5f7a92313260872994bc18685332e). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21398 (Although I'm pretty sure it will fail style checks.) Also, @cloud-fan since you wrote the original change. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21398 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user yuchaoran2011 commented on the issue: https://github.com/apache/spark/pull/21398 Thanks @vanzin. I've verified this commit has all the changes you mentioned. I've also removed references to CDH in the code comments. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21398 Oh man, this thing. I didn't post this upstream because I think this is kinda hacky, and I also consider this primarily a Sentry bug. But if others are ok with it... Just make sure you have all the necessary changes. My initial patch caused a couple of breakages. https://github.com/cloudera/spark/commit/4c8289d https://github.com/cloudera/spark/commit/629b1c3 https://github.com/cloudera/spark/commit/41fc8a1 Also you probably shouldn't reference CDH bugs in the code here... --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/21398 @vanzin Could you pls review it? I know the fix exists on Cloudera's fork but right now since this is missing from the upstream project, it will affect all sentry users who dont use Cloudera's distro. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21398 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21398: [SPARK-24338][SQL] Fixed Hive CREATETABLE error in Sentr...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21398 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
