Re: Review Request 58865: HIVE-16552: Limit the number of tasks a Spark job may contain

[Lefty Leverenz]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58865/#review173436
---




common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 3340-3341 (patched)


"... more tasks than the maximu, ..." -> maximum

Also, please add one or more newlines (\n) so the generated file 
hive-default.xml.template won't put the entire parameter description on a 
single line.


- Lefty Leverenz


On April 28, 2017, 11:34 p.m., Xuefu Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58865/
> ---
> 
> (Updated April 28, 2017, 11:34 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-16552
> https://issues.apache.org/jira/browse/HIVE-16552
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See JIRA description
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d3ea824 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java 32a7730 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/RemoteSparkJobMonitor.java
>  dd73f3e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java 
> 0b224f2 
> 
> 
> Diff: https://reviews.apache.org/r/58865/diff/1/
> 
> 
> Testing
> ---
> 
> Test locally
> 
> 
> Thanks,
> 
> Xuefu Zhang
> 
>

Fwd: Exception while renewing a Delegation Token

[Manikumar]

+dev

-- Forwarded message --
From: Manikumar 
Date: Thu, Apr 27, 2017 at 6:19 PM
Subject: Exception while renewing a Delegation Token
To: u...@hive.apache.org


Hi,

I am adding support to Storm Hive Bolt to access Hive meta-store using
delegation
tokens. I am able to create and cancel delegation tokens using HCatClient.
I am getting
below exception while renewing a delegation token.

Exception:
org.apache.hive.hcatalog.common.HCatException:
org.apache.hive.hcatalog.common.HCatException
: 9001 : Exception occurred while processing HCat request : MetaException
while renewing delegation token.. Cause : MetaException(message:hive/tes
t.cluster@example.com tries to renew a token with renewer hive)
at 
org.apache.hive.hcatalog.api.HCatClientHMSImpl.renewDelegationToken(HCatClientHMSImpl.java:643)
~[hive-webhcat-java-client-0.14.0.jar:0.14.0]

Hadoop's AbstractDelegationTokenIdentifier sets the renewer to Kerberos
shortname
but HCatCleint.renewDelegationToken uses the fullName. This causes the
renewal to fail.

Relavent parts of code : https://github.com/apache/
hive/blob/master/shims/common/src/main/java/org/apache/hadoop/hive/thrift/
DelegationTokenSecretManager.java#L96

https://github.com/apache/hadoop/blob/trunk/hadoop-
common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/
delegation/AbstractDelegationTokenIdentifier.java#L105


Let me know If I am missing something. Is there any existing JIRA for this?


Thanks,
Manikumar

[jira] [Created] (HIVE-16561) Flaky test :TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery

[Peter Vary (JIRA)]

Peter Vary created HIVE-16561:
-

 Summary: Flaky test 
:TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery
 Key: HIVE-16561
 URL: https://issues.apache.org/jira/browse/HIVE-16561
 Project: Hive
  Issue Type: Sub-task
Reporter: Peter Vary


This test sometimes failing. See: 
https://builds.apache.org/job/PreCommit-HIVE-Build/4933/testReport/org.apache.hive.jdbc/TestMultiSessionsHS2WithLocalClusterSpark/testSparkQuery/

{code}
java.util.concurrent.ExecutionException: java.sql.SQLException: 
org.apache.hive.service.cli.HiveSQLException: Error while processing statement: 
FAILED: Execution Error, return code 3 from 
org.apache.hadoop.hive.ql.exec.spark.SparkTask. java.io.IOException: Failed to 
create local dir in /tmp/blockmgr-18ab0417-d5c5-43f1-9a1f-0aa8c93f6e48/0e.
at 
org.apache.spark.storage.DiskBlockManager.getFile(DiskBlockManager.scala:70)
at org.apache.spark.storage.DiskStore.contains(DiskStore.scala:124)
at 
org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$getCurrentBlockStatus(BlockManager.scala:379)
at 
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:959)
at 
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:910)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:866)
at 
org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:910)
at 
org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:700)
at 
org.apache.spark.storage.BlockManager.putSingle(BlockManager.scala:1213)
at 
org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:103)
at 
org.apache.spark.broadcast.TorrentBroadcast.(TorrentBroadcast.scala:86)
at 
org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
at 
org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:56)
at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1370)
at org.apache.spark.rdd.HadoopRDD.(HadoopRDD.scala:125)
at 
org.apache.spark.SparkContext$$anonfun$hadoopRDD$1.apply(SparkContext.scala:965)
at 
org.apache.spark.SparkContext$$anonfun$hadoopRDD$1.apply(SparkContext.scala:961)
at 
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at 
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.SparkContext.withScope(SparkContext.scala:682)
at org.apache.spark.SparkContext.hadoopRDD(SparkContext.scala:961)
at 
org.apache.spark.api.java.JavaSparkContext.hadoopRDD(JavaSparkContext.scala:412)
at 
org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generateMapInput(SparkPlanGenerator.java:198)
at 
org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generateParentTran(SparkPlanGenerator.java:138)
at 
org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generate(SparkPlanGenerator.java:110)
at 
org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient$JobStatusJob.call(RemoteHiveSparkClient.java:346)
at 
org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:358)
at 
org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:323)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

at 
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315)
at 
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:258)
at 
org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:92)
at 
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:347)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at 
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:360)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.spark.SparkException: java.io.IOException: