[GitHub] spark pull request: [Spark-5111][SQL]HiveContext and Thriftserver ...
Github user bolkedebruin commented on the pull request: https://github.com/apache/spark/pull/4064#issuecomment-113956052 To make it a bit more specific: the latest spark does not support secure hive 0.14 yet. I tried the patch but this fails when using a yarn cluster (spark-shell --master yarn-client) in SparkContext. I move the patch to be able to be used in SparkContext and that got me a bit further. However, I couldn't set it in HiveContext anymore and this seems to be required to function, ie. if I remove the patch from HiveContext but leave it in SparkContext it fails. So it seems to be required twice (maybe due to isolated loading? just a guess), but the patch fails in the second instance. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-5111][SQL]HiveContext and Thriftserver ...
Github user bolkedebruin commented on the pull request: https://github.com/apache/spark/pull/4064#issuecomment-113505275 @zhzhan @WangTaoTheTonic I just cloned the latest and this issue still seems to exist. This is on Hadoop 2.6 from HDP 2.2 and Hive 0.14 + Kerberos --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-5111][SQL]HiveContext and Thriftserver ...
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/4064#issuecomment-112679214 @zhzhan Hey Could you describe the error and your configurations in detail please? As we now use Hive 13 + Hadoop 2.7 in our product and never ran into this. And now Spark could support working with Hive 14 now, per https://github.com/apache/spark/commit/4eb48ed1dadee80d78ada5d15884dd348c46ad27. Is there still this issue? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-5111][SQL]HiveContext and Thriftserver ...
Github user zhzhan commented on the pull request: https://github.com/apache/spark/pull/4064#issuecomment-112875139 @WangTaoTheTonic The problem happens with spark-1.3 and hadoop-2.6 in kerberos cluster. With hive-0.14 support, I suppose the problem may be gone, but I didn't verify it yet. I will close it since hive-0.14 is supported. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-5111][SQL]HiveContext and Thriftserver ...
Github user zhzhan closed the pull request at: https://github.com/apache/spark/pull/4064 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-5111][SQL]HiveContext and Thriftserver ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4064#issuecomment-87817047 [Test build #29420 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29420/consoleFull) for PR 4064 at commit [`77871ca`](https://github.com/apache/spark/commit/77871cab8279d85da8b38f86d6c7d16e77053c05). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-5111][SQL]HiveContext and Thriftserver ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4064#issuecomment-87847743 [Test build #29420 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29420/consoleFull) for PR 4064 at commit [`77871ca`](https://github.com/apache/spark/commit/77871cab8279d85da8b38f86d6c7d16e77053c05). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class HadoopThriftAuthBridge23 extends HadoopThriftAuthBridge20S ` * This patch does not change any dependencies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-5111][SQL]HiveContext and Thriftserver ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4064#issuecomment-8784 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29420/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-5111][SQL]HiveContext and Thriftserver ...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/4064#issuecomment-86889339 @zhzhan Would you mind to give a summary of the code added in this PR? I feel kinda hesitant to have this... Mostly because Spark doesn't explicitly support Hadoop 2.5.0 and above versions. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-5111][SQL]HiveContext and Thriftserver ...
Github user zhzhan commented on the pull request: https://github.com/apache/spark/pull/4064#issuecomment-87139435 @liancheng This is to port HIVE-7620 into spark to solve hadoop incompatibility issue. It will replace the getHadoopSaslProperties in org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S if necessary by checking the hadoop version. https://issues.apache.org/jira/browse/HIVE-7620 I agree the code may not be fitting into the overall code base. But it is broken in hive-0.13 connecting to secure cluster hadoop-2.5 and beyond due to this issue. If hive-0.14 will be supported soon, this patch may not be critical. By the way, hadoop-2.6 and beyond is becoming main stream in my point of view. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-5111][SQL]HiveContext and Thriftserver ...
Github user zhzhan commented on the pull request:
https://github.com/apache/spark/pull/4064#issuecomment-87141038
@liancheng By the way, if you think it can be merged to upstream, please
let me know. Because SparkSQLCLIDriver also need to replace the method to make
it work.
def main(args: Array[String]) {
+HiveInstrumentationAgent.instrument
val oproc = new OptionsProcessor()
if (!oproc.process_stage1(args)) {
System.exit(1)
}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-5111][SQL]HiveContext and Thriftserver ...
Github user zhzhan commented on the pull request: https://github.com/apache/spark/pull/4064#issuecomment-70897525 @marmbrus Can you take a look at this and let you know if you have any concern? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-5111][SQL]HiveContext and Thriftserver ...
GitHub user zhzhan opened a pull request: https://github.com/apache/spark/pull/4064 [Spark-5111][SQL]HiveContext and Thriftserver cannot work in secure cluster beyond hadoop2.5 Hive0.13 cannot work with secure cluster in hadoop-2.5 and beyound. Due to java.lang.NoSuchFieldError: SASL_PROPS error. Need to backport some hive-0.14 fix into spark, since there is no effort to upgrade hive to 0.14 support in spark. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zhzhan/spark spark5111 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4064.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4064 commit 3bf966c2f1bb913149a34176598a69041487cb88 Author: Zhan Zhang zhaz...@gmail.com Date: 2014-08-08T17:47:18Z test commit fc56b25ff62964f59b96d2db13b5c357ae1c2f2b Author: Zhan Zhang zhaz...@gmail.com Date: 2015-01-07T21:01:45Z squash all commits commit c6b57402d19557105bc2bb95978b5815d7e95907 Author: Zhan Zhang zhaz...@gmail.com Date: 2015-01-09T17:48:45Z hive secure cluster fix commit 456232c1ce29a7bff7f7d606764d5da00a478695 Author: Zhan Zhang zhaz...@gmail.com Date: 2015-01-09T21:57:54Z hive on secure cluster fix commit 6532a342ba85be0300c169ce81f671da7ea5dcb1 Author: Zhan Zhang zhaz...@gmail.com Date: 2015-01-15T19:53:36Z rebase --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
