[jira] [Updated] (HIVE-17721) with Postgres rdbms for metastore and dbnotification enabled, hive DDL SQL query fails
[ https://issues.apache.org/jira/browse/HIVE-17721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anishek updated HIVE-17721: --- Resolution: Fixed Status: Resolved (was: Patch Available) > with Postgres rdbms for metastore and dbnotification enabled, hive DDL SQL > query fails > --- > > Key: HIVE-17721 > URL: https://issues.apache.org/jira/browse/HIVE-17721 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: anishek > Fix For: 3.0.0 > > Attachments: HIVE-17721.0.patch > > > with postgres rdbms for hive-metastore any DDL fails when dbnotification is > enabled, the reason being a lock on the notification sequence is required, > which for Postgres requires the column-names and table names to enclosed in > "(double quotes) as we are using direct SQL and not going through datanucleus > and postgres is case sensitive. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17721) with Postgres rdbms for metastore and dbnotification enabled, hive DDL SQL query fails
[ https://issues.apache.org/jira/browse/HIVE-17721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196514#comment-16196514 ] anishek commented on HIVE-17721: test failures are not related to the change, committed to master. Thanks [~thejas] for the review. > with Postgres rdbms for metastore and dbnotification enabled, hive DDL SQL > query fails > --- > > Key: HIVE-17721 > URL: https://issues.apache.org/jira/browse/HIVE-17721 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: anishek > Fix For: 3.0.0 > > Attachments: HIVE-17721.0.patch > > > with postgres rdbms for hive-metastore any DDL fails when dbnotification is > enabled, the reason being a lock on the notification sequence is required, > which for Postgres requires the column-names and table names to enclosed in > "(double quotes) as we are using direct SQL and not going through datanucleus > and postgres is case sensitive. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17743) Add InterfaceAudience and InterfaceStability annotations for Thrift generated APIs
[
https://issues.apache.org/jira/browse/HIVE-17743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196512#comment-16196512
]
Hive QA commented on HIVE-17743:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12891000/HIVE-17743.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 11190 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
(batchId=231)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
(batchId=231)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=171)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14]
(batchId=239)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7189/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7189/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7189/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12891000 - PreCommit-HIVE-Build
> Add InterfaceAudience and InterfaceStability annotations for Thrift generated
> APIs
> --
>
> Key: HIVE-17743
> URL: https://issues.apache.org/jira/browse/HIVE-17743
> Project: Hive
> Issue Type: Sub-task
> Components: Thrift API
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-17743.1.patch
>
>
> The Thrift generated files don't have {{InterfaceAudience}} or
> {{InterfaceStability}} annotations on them, mainly because all the files are
> auto-generated.
> We should add some code that auto-tags all the Java Thrift generated files
> with these annotations. This way even when they are re-generated, they still
> contain the annotations.
> We should be able to do this using the
> {{com.google.code.maven-replacer-plugin}} similar to what we do in
> {{standalone-metastore/pom.xml}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[
https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196484#comment-16196484
]
Hive QA commented on HIVE-17139:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12890998/HIVE-17139.20.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11190 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
(batchId=231)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
(batchId=231)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=171)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query23]
(batchId=239)
org.apache.hive.spark.client.rpc.TestRpc.testServerPort (batchId=287)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7188/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7188/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7188/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12890998 - PreCommit-HIVE-Build
> Conditional expressions optimization: skip the expression evaluation if the
> condition is not satisfied for vectorization engine.
>
>
> Key: HIVE-17139
> URL: https://issues.apache.org/jira/browse/HIVE-17139
> Project: Hive
> Issue Type: Improvement
>Reporter: Ke Jia
>Assignee: Ke Jia
> Attachments: HIVE-17139.1.patch, HIVE-17139.10.patch,
> HIVE-17139.11.patch, HIVE-17139.12.patch, HIVE-17139.13.patch,
> HIVE-17139.13.patch, HIVE-17139.14.patch, HIVE-17139.15.patch,
> HIVE-17139.16.patch, HIVE-17139.17.patch, HIVE-17139.18.patch,
> HIVE-17139.18.patch, HIVE-17139.19.patch, HIVE-17139.2.patch,
> HIVE-17139.20.patch, HIVE-17139.3.patch, HIVE-17139.4.patch,
> HIVE-17139.5.patch, HIVE-17139.6.patch, HIVE-17139.7.patch,
> HIVE-17139.8.patch, HIVE-17139.9.patch
>
>
> The case when and if statement execution for Hive vectorization is not
> optimal, which all the conditional and else expressions are evaluated for
> current implementation. The optimized approach is to update the selected
> array of batch parameter after the conditional expression is executed. Then
> the else expression will only do the selected rows instead of all.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17718) spark-client and job monitor logging improvements
[
https://issues.apache.org/jira/browse/HIVE-17718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196457#comment-16196457
]
Hive QA commented on HIVE-17718:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12890995/HIVE-17718.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 11190 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
(batchId=231)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
(batchId=231)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=101)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14]
(batchId=239)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7187/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7187/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7187/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12890995 - PreCommit-HIVE-Build
> spark-client and job monitor logging improvements
> -
>
> Key: HIVE-17718
> URL: https://issues.apache.org/jira/browse/HIVE-17718
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-17718.1.patch, HIVE-17718.2.patch,
> HIVE-17718.3.patch, HIVE-17718.4.patch, HIVE-17718.5.patch
>
>
> Example:
> {code}
> 2017-10-05 17:47:11,881 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2017-10-05 17:47:11,881 WARN org.apache.hadoop.hive.ql.Driver:
> [HiveServer2-Handler-Pool: Thread-105]: Shutting down task : Stage-2:MAPRED
> 2017-10-05 17:47:11,882 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
>
[jira] [Resolved] (HIVE-17200) TestMiniSparkOnYarnCliDriver: [spark_vectorized_dynamic_partition_pruning]
[ https://issues.apache.org/jira/browse/HIVE-17200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar resolved HIVE-17200. - Resolution: Fixed Fixed in HIVE-17122 > TestMiniSparkOnYarnCliDriver: [spark_vectorized_dynamic_partition_pruning] > -- > > Key: HIVE-17200 > URL: https://issues.apache.org/jira/browse/HIVE-17200 > Project: Hive > Issue Type: Sub-task >Reporter: Janaki Lahorani > > Error: > Client execution failed with error code = 3 running " > select count(*) from srcpart join (select ds as ds, ds as `date` from srcpart > group by ds) s on (srcpart.ds = s.ds) where s.`date` = '2008-04-08'" > fname=spark_vectorized_dynamic_partition_pruning.q -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HIVE-17122) spark_vectorized_dynamic_partition_pruning.q is continuously failing
[
https://issues.apache.org/jira/browse/HIVE-17122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar resolved HIVE-17122.
-
Resolution: Fixed
Test has been working successfully for a while now.
> spark_vectorized_dynamic_partition_pruning.q is continuously failing
>
>
> Key: HIVE-17122
> URL: https://issues.apache.org/jira/browse/HIVE-17122
> Project: Hive
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>
> {code}
> Driver stacktrace:
> at
> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1450)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1438)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1437)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1437)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
> at scala.Option.foreach(Option.scala:257)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:811)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1659)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1618)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1607)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> Caused by: java.lang.RuntimeException: Hive Runtime Error while closing
> operators: 1
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.close(SparkReduceRecordHandler.java:616)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunctionResultList.closeRecordProcessor(HiveReduceFunctionResultList.java:67)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList.hasNext(HiveBaseFunctionResultList.java:96)
> at
> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42)
> at
> org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
> at org.apache.spark.scheduler.Task.run(Task.scala:85)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupKeyHelper.copyGroupKey(VectorGroupKeyHelper.java:107)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeReduceMergePartial.doProcessBatch(VectorGroupByOperator.java:832)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processBatch(VectorGroupByOperator.java:179)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.process(VectorGroupByOperator.java:1037)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.forwardBatch(SparkReduceRecordHandler.java:542)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.close(SparkReduceRecordHandler.java:584)
> ... 11 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-16521) HoS user level explain plan possibly incorrect for UNION clause
[
https://issues.apache.org/jira/browse/HIVE-16521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-16521:
Status: Open (was: Patch Available)
> HoS user level explain plan possibly incorrect for UNION clause
> ---
>
> Key: HIVE-16521
> URL: https://issues.apache.org/jira/browse/HIVE-16521
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 3.0.0
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16521.1.patch
>
>
> The user-level explain plan for queries with a UNION operator look very
> different for HoS vs. Hive-on-Tez. Furthermore, the HoS plan looks incomplete:
> Query: {{EXPLAIN select count(*) from srcpart where srcpart.ds in (select
> max(srcpart.ds) from srcpart union all select min(srcpart.ds) from srcpart)}}
> Hive-on-Tez:
> {code}
> Plan optimized by CBO.
> Vertex dependency in root stage
> Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
> Reducer 3 <- Reducer 2 (CUSTOM_SIMPLE_EDGE)
> Reducer 5 <- Map 4 (CUSTOM_SIMPLE_EDGE), Union 6 (CONTAINS)
> Reducer 7 <- Union 6 (SIMPLE_EDGE)
> Reducer 9 <- Map 8 (CUSTOM_SIMPLE_EDGE), Union 6 (CONTAINS)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Reducer 3
> File Output Operator [FS_34]
> Group By Operator [GBY_32] (rows=1 width=8)
> Output:["_col0"],aggregations:["count(VALUE._col0)"]
> <-Reducer 2 [CUSTOM_SIMPLE_EDGE]
> PARTITION_ONLY_SHUFFLE [RS_31]
> Group By Operator [GBY_30] (rows=1 width=8)
> Output:["_col0"],aggregations:["count()"]
> Merge Join Operator [MERGEJOIN_44] (rows=1000 width=8)
> Conds:RS_26._col0=RS_27._col0(Inner)
> <-Map 1 [SIMPLE_EDGE]
> SHUFFLE [RS_26]
> PartitionCols:_col0
> Select Operator [SEL_2] (rows=2000 width=184)
> Output:["_col0"]
> TableScan [TS_0] (rows=2000 width=194)
> default@srcpart,srcpart,Tbl:COMPLETE,Col:COMPLETE
> <-Reducer 7 [SIMPLE_EDGE]
> SHUFFLE [RS_27]
> PartitionCols:_col0
> Group By Operator [GBY_24] (rows=1 width=184)
> Output:["_col0"],keys:KEY._col0
> <-Union 6 [SIMPLE_EDGE]
> <-Reducer 5 [CONTAINS]
> Reduce Output Operator [RS_23]
> PartitionCols:_col0
> Group By Operator [GBY_22] (rows=1 width=184)
> Output:["_col0"],keys:_col0
> Filter Operator [FIL_9] (rows=1 width=184)
> predicate:_col0 is not null
> Group By Operator [GBY_7] (rows=1 width=184)
>
> Output:["_col0"],aggregations:["max(VALUE._col0)"]
> <-Map 4 [CUSTOM_SIMPLE_EDGE]
> PARTITION_ONLY_SHUFFLE [RS_6]
> Group By Operator [GBY_5] (rows=1 width=184)
> Output:["_col0"],aggregations:["max(ds)"]
> Select Operator [SEL_4] (rows=2000
> width=194)
> Output:["ds"]
> TableScan [TS_3] (rows=2000 width=194)
>
> default@srcpart,srcpart,Tbl:COMPLETE,Col:COMPLETE
> <-Reducer 9 [CONTAINS]
> Reduce Output Operator [RS_23]
> PartitionCols:_col0
> Group By Operator [GBY_22] (rows=1 width=184)
> Output:["_col0"],keys:_col0
> Filter Operator [FIL_17] (rows=1 width=184)
> predicate:_col0 is not null
> Group By Operator [GBY_15] (rows=1 width=184)
>
> Output:["_col0"],aggregations:["min(VALUE._col0)"]
> <-Map 8 [CUSTOM_SIMPLE_EDGE]
> PARTITION_ONLY_SHUFFLE [RS_14]
> Group By Operator [GBY_13] (rows=1 width=184)
> Output:["_col0"],aggregations:["min(ds)"]
> Select Operator [SEL_12] (rows=2000
> width=194)
> Output:["ds"]
> TableScan [TS_11] (rows=2000 width=194)
>
> default@srcpart,srcpart,Tbl:COMPLETE,Col:COMPLETE
> Dynamic Partitioning Event Operator [EVENT_43] (rows=1
> width=184)
> Group By Operator
[jira] [Updated] (HIVE-17743) Add InterfaceAudience and InterfaceStability annotations for Thrift generated APIs
[
https://issues.apache.org/jira/browse/HIVE-17743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17743:
Status: Patch Available (was: Open)
> Add InterfaceAudience and InterfaceStability annotations for Thrift generated
> APIs
> --
>
> Key: HIVE-17743
> URL: https://issues.apache.org/jira/browse/HIVE-17743
> Project: Hive
> Issue Type: Sub-task
> Components: Thrift API
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-17743.1.patch
>
>
> The Thrift generated files don't have {{InterfaceAudience}} or
> {{InterfaceStability}} annotations on them, mainly because all the files are
> auto-generated.
> We should add some code that auto-tags all the Java Thrift generated files
> with these annotations. This way even when they are re-generated, they still
> contain the annotations.
> We should be able to do this using the
> {{com.google.code.maven-replacer-plugin}} similar to what we do in
> {{standalone-metastore/pom.xml}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17743) Add InterfaceAudience and InterfaceStability annotations for Thrift generated APIs
[
https://issues.apache.org/jira/browse/HIVE-17743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17743:
Attachment: HIVE-17743.1.patch
> Add InterfaceAudience and InterfaceStability annotations for Thrift generated
> APIs
> --
>
> Key: HIVE-17743
> URL: https://issues.apache.org/jira/browse/HIVE-17743
> Project: Hive
> Issue Type: Sub-task
> Components: Thrift API
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-17743.1.patch
>
>
> The Thrift generated files don't have {{InterfaceAudience}} or
> {{InterfaceStability}} annotations on them, mainly because all the files are
> auto-generated.
> We should add some code that auto-tags all the Java Thrift generated files
> with these annotations. This way even when they are re-generated, they still
> contain the annotations.
> We should be able to do this using the
> {{com.google.code.maven-replacer-plugin}} similar to what we do in
> {{standalone-metastore/pom.xml}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated HIVE-17139: -- Attachment: HIVE-17139.20.patch > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.1.patch, HIVE-17139.10.patch, > HIVE-17139.11.patch, HIVE-17139.12.patch, HIVE-17139.13.patch, > HIVE-17139.13.patch, HIVE-17139.14.patch, HIVE-17139.15.patch, > HIVE-17139.16.patch, HIVE-17139.17.patch, HIVE-17139.18.patch, > HIVE-17139.18.patch, HIVE-17139.19.patch, HIVE-17139.2.patch, > HIVE-17139.20.patch, HIVE-17139.3.patch, HIVE-17139.4.patch, > HIVE-17139.5.patch, HIVE-17139.6.patch, HIVE-17139.7.patch, > HIVE-17139.8.patch, HIVE-17139.9.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17502) Reuse of default session should not throw an exception in LLAP w/ Tez
[
https://issues.apache.org/jira/browse/HIVE-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196434#comment-16196434
]
Hive QA commented on HIVE-17502:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12890993/HIVE-17502.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 11191 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
(batchId=231)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
(batchId=231)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=171)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14]
(batchId=239)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7186/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7186/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7186/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12890993 - PreCommit-HIVE-Build
> Reuse of default session should not throw an exception in LLAP w/ Tez
> -
>
> Key: HIVE-17502
> URL: https://issues.apache.org/jira/browse/HIVE-17502
> Project: Hive
> Issue Type: Bug
> Components: llap, Tez
>Affects Versions: 2.1.1, 2.2.0
> Environment: HDP 2.6.1.0-129, Hue 4
>Reporter: Thai Bui
>Assignee: Thai Bui
> Fix For: 3.0.0
>
> Attachments: HIVE-17502.patch
>
>
> Hive2 w/ LLAP on Tez doesn't allow a currently used, default session to be
> skipped mostly because of this line
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java#L365.
> However, some clients such as Hue 4, allow multiple sessions to be used per
> user. Under this configuration, a Thrift client will send a request to either
> reuse or open a new session. The reuse request could include the session id
> of a currently used snippet being executed in Hue, this causes HS2 to throw
> an exception:
> {noformat}
> 2017-09-10T17:51:36,548 INFO [Thread-89]: tez.TezSessionPoolManager
> (TezSessionPoolManager.java:canWorkWithSameSession(512)) - The current user:
> hive, session user: hive
> 2017-09-10T17:51:36,549 ERROR [Thread-89]: exec.Task
> (TezTask.java:execute(232)) - Failed to execute tez graph.
> org.apache.hadoop.hive.ql.metadata.HiveException: The pool session
> sessionId=5b61a578-6336-41c5-860d-9838166f97fe, queueName=llap, user=hive,
> doAs=false, isOpen=true, isDefault=true, expires in 591015330ms should have
> been returned to the pool
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:534)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:544)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:147)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:79)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> {noformat}
> Note that every query is issued as a single 'hive' user to share the LLAP
> daemon pool, a set of pre-determined number of AMs is initialized at setup
> time. Thus, HS2 should allow new sessions from a Thrift client to be used out
> of the pool, or an existing session to be skipped and an unused session from
> the pool to be returned. The logic to throw an exception in the
> `canWorkWithSameSession` doesn't make sense to me.
> I have a solution to fix this issue in my local branch at
> https://github.com/thaibui/hive/commit/078a521b9d0906fe6c0323b63e567f6eee2f3a70.
> When applied, the log will become like so
> {noformat}
> 2017-09-10T09:15:33,578 INFO [Thread-239]:
[jira] [Updated] (HIVE-17718) spark-client and job monitor logging improvements
[
https://issues.apache.org/jira/browse/HIVE-17718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17718:
Attachment: HIVE-17718.5.patch
> spark-client and job monitor logging improvements
> -
>
> Key: HIVE-17718
> URL: https://issues.apache.org/jira/browse/HIVE-17718
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-17718.1.patch, HIVE-17718.2.patch,
> HIVE-17718.3.patch, HIVE-17718.4.patch, HIVE-17718.5.patch
>
>
> Example:
> {code}
> 2017-10-05 17:47:11,881 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2017-10-05 17:47:11,881 WARN org.apache.hadoop.hive.ql.Driver:
> [HiveServer2-Handler-Pool: Thread-105]: Shutting down task : Stage-2:MAPRED
> 2017-10-05 17:47:11,882 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at
[jira] [Commented] (HIVE-17139) Conditional expressions optimization: skip the expression evaluation if the condition is not satisfied for vectorization engine.
[ https://issues.apache.org/jira/browse/HIVE-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196418#comment-16196418 ] Ferdinand Xu commented on HIVE-17139: - LGTM +1 > Conditional expressions optimization: skip the expression evaluation if the > condition is not satisfied for vectorization engine. > > > Key: HIVE-17139 > URL: https://issues.apache.org/jira/browse/HIVE-17139 > Project: Hive > Issue Type: Improvement >Reporter: Ke Jia >Assignee: Ke Jia > Attachments: HIVE-17139.1.patch, HIVE-17139.10.patch, > HIVE-17139.11.patch, HIVE-17139.12.patch, HIVE-17139.13.patch, > HIVE-17139.13.patch, HIVE-17139.14.patch, HIVE-17139.15.patch, > HIVE-17139.16.patch, HIVE-17139.17.patch, HIVE-17139.18.patch, > HIVE-17139.18.patch, HIVE-17139.19.patch, HIVE-17139.2.patch, > HIVE-17139.3.patch, HIVE-17139.4.patch, HIVE-17139.5.patch, > HIVE-17139.6.patch, HIVE-17139.7.patch, HIVE-17139.8.patch, HIVE-17139.9.patch > > > The case when and if statement execution for Hive vectorization is not > optimal, which all the conditional and else expressions are evaluated for > current implementation. The optimized approach is to update the selected > array of batch parameter after the conditional expression is executed. Then > the else expression will only do the selected rows instead of all. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17502) Reuse of default session should not throw an exception in LLAP w/ Tez
[
https://issues.apache.org/jira/browse/HIVE-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196398#comment-16196398
]
Thai Bui commented on HIVE-17502:
-
[~sershe] Based on our previous discussions, I've made a patch to add an option
to allow default sessions reuse. Please review when you have time and let me
know if there's anything I need to change. Thanks!
> Reuse of default session should not throw an exception in LLAP w/ Tez
> -
>
> Key: HIVE-17502
> URL: https://issues.apache.org/jira/browse/HIVE-17502
> Project: Hive
> Issue Type: Bug
> Components: llap, Tez
>Affects Versions: 2.1.1, 2.2.0
> Environment: HDP 2.6.1.0-129, Hue 4
>Reporter: Thai Bui
>Assignee: Thai Bui
> Fix For: 3.0.0
>
> Attachments: HIVE-17502.patch
>
>
> Hive2 w/ LLAP on Tez doesn't allow a currently used, default session to be
> skipped mostly because of this line
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java#L365.
> However, some clients such as Hue 4, allow multiple sessions to be used per
> user. Under this configuration, a Thrift client will send a request to either
> reuse or open a new session. The reuse request could include the session id
> of a currently used snippet being executed in Hue, this causes HS2 to throw
> an exception:
> {noformat}
> 2017-09-10T17:51:36,548 INFO [Thread-89]: tez.TezSessionPoolManager
> (TezSessionPoolManager.java:canWorkWithSameSession(512)) - The current user:
> hive, session user: hive
> 2017-09-10T17:51:36,549 ERROR [Thread-89]: exec.Task
> (TezTask.java:execute(232)) - Failed to execute tez graph.
> org.apache.hadoop.hive.ql.metadata.HiveException: The pool session
> sessionId=5b61a578-6336-41c5-860d-9838166f97fe, queueName=llap, user=hive,
> doAs=false, isOpen=true, isDefault=true, expires in 591015330ms should have
> been returned to the pool
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:534)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:544)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:147)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:79)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> {noformat}
> Note that every query is issued as a single 'hive' user to share the LLAP
> daemon pool, a set of pre-determined number of AMs is initialized at setup
> time. Thus, HS2 should allow new sessions from a Thrift client to be used out
> of the pool, or an existing session to be skipped and an unused session from
> the pool to be returned. The logic to throw an exception in the
> `canWorkWithSameSession` doesn't make sense to me.
> I have a solution to fix this issue in my local branch at
> https://github.com/thaibui/hive/commit/078a521b9d0906fe6c0323b63e567f6eee2f3a70.
> When applied, the log will become like so
> {noformat}
> 2017-09-10T09:15:33,578 INFO [Thread-239]: tez.TezSessionPoolManager
> (TezSessionPoolManager.java:canWorkWithSameSession(533)) - Skipping default
> session sessionId=6638b1da-0f8a-405e-85f0-9586f484e6de, queueName=llap,
> user=hive, doAs=false, isOpen=true, isDefault=true, expires in 591868732ms
> since it is being used.
> {noformat}
> A test case is provided in my branch to demonstrate how it works. If possible
> I would like this patch to be applied to version 2.1, 2.2 and master. Since
> we are using 2.1 LLAP in production with Hue 4, this patch is critical to our
> success.
> Alternatively, if this patch is too broad in scope, I propose adding an
> option to allow "skipping of currently used default sessions". With this new
> option default to "false", existing behavior won't change unless the option
> is turned on.
> I will prepare an official path if this change to master &/ the other
> branches is acceptable. I'm not an contributor &/ committer, this will be my
> first time contributing to Hive and the Apache foundation. Any early review
> is greatly appreciated, thanks!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17502) Reuse of default session should not throw an exception in LLAP w/ Tez
[
https://issues.apache.org/jira/browse/HIVE-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thai Bui updated HIVE-17502:
Fix Version/s: 3.0.0
Release Note: Add an option to allow parallel default sessions reuse in
HiveServer2 with LLAP/Tez
Status: Patch Available (was: In Progress)
> Reuse of default session should not throw an exception in LLAP w/ Tez
> -
>
> Key: HIVE-17502
> URL: https://issues.apache.org/jira/browse/HIVE-17502
> Project: Hive
> Issue Type: Bug
> Components: llap, Tez
>Affects Versions: 2.2.0, 2.1.1
> Environment: HDP 2.6.1.0-129, Hue 4
>Reporter: Thai Bui
>Assignee: Thai Bui
> Fix For: 3.0.0
>
> Attachments: HIVE-17502.patch
>
>
> Hive2 w/ LLAP on Tez doesn't allow a currently used, default session to be
> skipped mostly because of this line
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java#L365.
> However, some clients such as Hue 4, allow multiple sessions to be used per
> user. Under this configuration, a Thrift client will send a request to either
> reuse or open a new session. The reuse request could include the session id
> of a currently used snippet being executed in Hue, this causes HS2 to throw
> an exception:
> {noformat}
> 2017-09-10T17:51:36,548 INFO [Thread-89]: tez.TezSessionPoolManager
> (TezSessionPoolManager.java:canWorkWithSameSession(512)) - The current user:
> hive, session user: hive
> 2017-09-10T17:51:36,549 ERROR [Thread-89]: exec.Task
> (TezTask.java:execute(232)) - Failed to execute tez graph.
> org.apache.hadoop.hive.ql.metadata.HiveException: The pool session
> sessionId=5b61a578-6336-41c5-860d-9838166f97fe, queueName=llap, user=hive,
> doAs=false, isOpen=true, isDefault=true, expires in 591015330ms should have
> been returned to the pool
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:534)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:544)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:147)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:79)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> {noformat}
> Note that every query is issued as a single 'hive' user to share the LLAP
> daemon pool, a set of pre-determined number of AMs is initialized at setup
> time. Thus, HS2 should allow new sessions from a Thrift client to be used out
> of the pool, or an existing session to be skipped and an unused session from
> the pool to be returned. The logic to throw an exception in the
> `canWorkWithSameSession` doesn't make sense to me.
> I have a solution to fix this issue in my local branch at
> https://github.com/thaibui/hive/commit/078a521b9d0906fe6c0323b63e567f6eee2f3a70.
> When applied, the log will become like so
> {noformat}
> 2017-09-10T09:15:33,578 INFO [Thread-239]: tez.TezSessionPoolManager
> (TezSessionPoolManager.java:canWorkWithSameSession(533)) - Skipping default
> session sessionId=6638b1da-0f8a-405e-85f0-9586f484e6de, queueName=llap,
> user=hive, doAs=false, isOpen=true, isDefault=true, expires in 591868732ms
> since it is being used.
> {noformat}
> A test case is provided in my branch to demonstrate how it works. If possible
> I would like this patch to be applied to version 2.1, 2.2 and master. Since
> we are using 2.1 LLAP in production with Hue 4, this patch is critical to our
> success.
> Alternatively, if this patch is too broad in scope, I propose adding an
> option to allow "skipping of currently used default sessions". With this new
> option default to "false", existing behavior won't change unless the option
> is turned on.
> I will prepare an official path if this change to master &/ the other
> branches is acceptable. I'm not an contributor &/ committer, this will be my
> first time contributing to Hive and the Apache foundation. Any early review
> is greatly appreciated, thanks!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17502) Reuse of default session should not throw an exception in LLAP w/ Tez
[
https://issues.apache.org/jira/browse/HIVE-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thai Bui updated HIVE-17502:
Attachment: HIVE-17502.patch
> Reuse of default session should not throw an exception in LLAP w/ Tez
> -
>
> Key: HIVE-17502
> URL: https://issues.apache.org/jira/browse/HIVE-17502
> Project: Hive
> Issue Type: Bug
> Components: llap, Tez
>Affects Versions: 2.1.1, 2.2.0
> Environment: HDP 2.6.1.0-129, Hue 4
>Reporter: Thai Bui
>Assignee: Thai Bui
> Attachments: HIVE-17502.patch
>
>
> Hive2 w/ LLAP on Tez doesn't allow a currently used, default session to be
> skipped mostly because of this line
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java#L365.
> However, some clients such as Hue 4, allow multiple sessions to be used per
> user. Under this configuration, a Thrift client will send a request to either
> reuse or open a new session. The reuse request could include the session id
> of a currently used snippet being executed in Hue, this causes HS2 to throw
> an exception:
> {noformat}
> 2017-09-10T17:51:36,548 INFO [Thread-89]: tez.TezSessionPoolManager
> (TezSessionPoolManager.java:canWorkWithSameSession(512)) - The current user:
> hive, session user: hive
> 2017-09-10T17:51:36,549 ERROR [Thread-89]: exec.Task
> (TezTask.java:execute(232)) - Failed to execute tez graph.
> org.apache.hadoop.hive.ql.metadata.HiveException: The pool session
> sessionId=5b61a578-6336-41c5-860d-9838166f97fe, queueName=llap, user=hive,
> doAs=false, isOpen=true, isDefault=true, expires in 591015330ms should have
> been returned to the pool
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:534)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:544)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:147)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:79)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> {noformat}
> Note that every query is issued as a single 'hive' user to share the LLAP
> daemon pool, a set of pre-determined number of AMs is initialized at setup
> time. Thus, HS2 should allow new sessions from a Thrift client to be used out
> of the pool, or an existing session to be skipped and an unused session from
> the pool to be returned. The logic to throw an exception in the
> `canWorkWithSameSession` doesn't make sense to me.
> I have a solution to fix this issue in my local branch at
> https://github.com/thaibui/hive/commit/078a521b9d0906fe6c0323b63e567f6eee2f3a70.
> When applied, the log will become like so
> {noformat}
> 2017-09-10T09:15:33,578 INFO [Thread-239]: tez.TezSessionPoolManager
> (TezSessionPoolManager.java:canWorkWithSameSession(533)) - Skipping default
> session sessionId=6638b1da-0f8a-405e-85f0-9586f484e6de, queueName=llap,
> user=hive, doAs=false, isOpen=true, isDefault=true, expires in 591868732ms
> since it is being used.
> {noformat}
> A test case is provided in my branch to demonstrate how it works. If possible
> I would like this patch to be applied to version 2.1, 2.2 and master. Since
> we are using 2.1 LLAP in production with Hue 4, this patch is critical to our
> success.
> Alternatively, if this patch is too broad in scope, I propose adding an
> option to allow "skipping of currently used default sessions". With this new
> option default to "false", existing behavior won't change unless the option
> is turned on.
> I will prepare an official path if this change to master &/ the other
> branches is acceptable. I'm not an contributor &/ committer, this will be my
> first time contributing to Hive and the Apache foundation. Any early review
> is greatly appreciated, thanks!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Work started] (HIVE-17502) Reuse of default session should not throw an exception in LLAP w/ Tez
[
https://issues.apache.org/jira/browse/HIVE-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-17502 started by Thai Bui.
---
> Reuse of default session should not throw an exception in LLAP w/ Tez
> -
>
> Key: HIVE-17502
> URL: https://issues.apache.org/jira/browse/HIVE-17502
> Project: Hive
> Issue Type: Bug
> Components: llap, Tez
>Affects Versions: 2.1.1, 2.2.0
> Environment: HDP 2.6.1.0-129, Hue 4
>Reporter: Thai Bui
>Assignee: Thai Bui
> Attachments: HIVE-17502.patch
>
>
> Hive2 w/ LLAP on Tez doesn't allow a currently used, default session to be
> skipped mostly because of this line
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java#L365.
> However, some clients such as Hue 4, allow multiple sessions to be used per
> user. Under this configuration, a Thrift client will send a request to either
> reuse or open a new session. The reuse request could include the session id
> of a currently used snippet being executed in Hue, this causes HS2 to throw
> an exception:
> {noformat}
> 2017-09-10T17:51:36,548 INFO [Thread-89]: tez.TezSessionPoolManager
> (TezSessionPoolManager.java:canWorkWithSameSession(512)) - The current user:
> hive, session user: hive
> 2017-09-10T17:51:36,549 ERROR [Thread-89]: exec.Task
> (TezTask.java:execute(232)) - Failed to execute tez graph.
> org.apache.hadoop.hive.ql.metadata.HiveException: The pool session
> sessionId=5b61a578-6336-41c5-860d-9838166f97fe, queueName=llap, user=hive,
> doAs=false, isOpen=true, isDefault=true, expires in 591015330ms should have
> been returned to the pool
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:534)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:544)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:147)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:79)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> {noformat}
> Note that every query is issued as a single 'hive' user to share the LLAP
> daemon pool, a set of pre-determined number of AMs is initialized at setup
> time. Thus, HS2 should allow new sessions from a Thrift client to be used out
> of the pool, or an existing session to be skipped and an unused session from
> the pool to be returned. The logic to throw an exception in the
> `canWorkWithSameSession` doesn't make sense to me.
> I have a solution to fix this issue in my local branch at
> https://github.com/thaibui/hive/commit/078a521b9d0906fe6c0323b63e567f6eee2f3a70.
> When applied, the log will become like so
> {noformat}
> 2017-09-10T09:15:33,578 INFO [Thread-239]: tez.TezSessionPoolManager
> (TezSessionPoolManager.java:canWorkWithSameSession(533)) - Skipping default
> session sessionId=6638b1da-0f8a-405e-85f0-9586f484e6de, queueName=llap,
> user=hive, doAs=false, isOpen=true, isDefault=true, expires in 591868732ms
> since it is being used.
> {noformat}
> A test case is provided in my branch to demonstrate how it works. If possible
> I would like this patch to be applied to version 2.1, 2.2 and master. Since
> we are using 2.1 LLAP in production with Hue 4, this patch is critical to our
> success.
> Alternatively, if this patch is too broad in scope, I propose adding an
> option to allow "skipping of currently used default sessions". With this new
> option default to "false", existing behavior won't change unless the option
> is turned on.
> I will prepare an official path if this change to master &/ the other
> branches is acceptable. I'm not an contributor &/ committer, this will be my
> first time contributing to Hive and the Apache foundation. Any early review
> is greatly appreciated, thanks!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17718) spark-client and job monitor logging improvements
[
https://issues.apache.org/jira/browse/HIVE-17718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196387#comment-16196387
]
Hive QA commented on HIVE-17718:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12890991/HIVE-17718.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7185/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7185/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7185/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-10-09 00:23:59.321
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-7185/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-10-09 00:23:59.324
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at caf3330 HIVE-17728: TestHCatClient should use
hive.metastore.transactional.event.listeners as per recommendation (Sankar
Hariappan, reviewed by Thejas Nair)
+ git clean -f -d
Removing standalone-metastore/src/gen/org/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at caf3330 HIVE-17728: TestHCatClient should use
hive.metastore.transactional.event.listeners as per recommendation (Sankar
Hariappan, reviewed by Thejas Nair)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-10-09 00:24:00.254
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: patch -p1
patching file ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/RemoteSparkJobMonitor.java
patching file
spark-client/src/main/java/org/apache/hive/spark/client/BaseProtocol.java
patching file
spark-client/src/main/java/org/apache/hive/spark/client/Message.java
patching file
spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java
patching file
spark-client/src/main/java/org/apache/hive/spark/client/rpc/Rpc.java
patching file
spark-client/src/main/java/org/apache/hive/spark/client/rpc/RpcServer.java
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: protoc version: 250, detected platform: linux/amd64
protoc-jar: executing: [/tmp/protoc6365283986091419098.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
[ERROR] COMPILATION ERROR :
[ERROR]
/data/hiveptest/working/apache-github-source-source/spark-client/src/test/java/org/apache/hive/spark/client/rpc/TestRpc.java:[89,41]
method call in class org.apache.hive.spark.client.rpc.Rpc cannot be applied to
given types;
required: org.apache.hive.spark.client.Message,java.lang.Class
found:
org.apache.hive.spark.client.rpc.TestRpc.TestMessage,java.lang.Class
reason: cannot infer type-variable(s) T
(argument mismatch; org.apache.hive.spark.client.rpc.TestRpc.TestMessage
cannot be converted to org.apache.hive.spark.client.Message)
[ERROR]
/data/hiveptest/working/apache-github-source-source/spark-client/src/test/java/org/apache/hive/spark/client/rpc/TestRpc.java:[106,38]
[jira] [Updated] (HIVE-17718) spark-client and job monitor logging improvements
[
https://issues.apache.org/jira/browse/HIVE-17718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17718:
Attachment: HIVE-17718.4.patch
> spark-client and job monitor logging improvements
> -
>
> Key: HIVE-17718
> URL: https://issues.apache.org/jira/browse/HIVE-17718
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-17718.1.patch, HIVE-17718.2.patch,
> HIVE-17718.3.patch, HIVE-17718.4.patch
>
>
> Example:
> {code}
> 2017-10-05 17:47:11,881 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2017-10-05 17:47:11,881 WARN org.apache.hadoop.hive.ql.Driver:
> [HiveServer2-Handler-Pool: Thread-105]: Shutting down task : Stage-2:MAPRED
> 2017-10-05 17:47:11,882 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at
[jira] [Updated] (HIVE-17701) Added restriction to historic queries on web UI
[ https://issues.apache.org/jira/browse/HIVE-17701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-17701: - Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) > Added restriction to historic queries on web UI > --- > > Key: HIVE-17701 > URL: https://issues.apache.org/jira/browse/HIVE-17701 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Thejas M Nair >Assignee: Tao Li > Fix For: 3.0.0 > > Attachments: HIVE-17701.1.patch, HIVE-17701.2.patch, > HIVE-17701.3.patch, HIVE-17701.4.patch, HIVE-17701.5.patch, HIVE-17701.6.patch > > > The HiveServer2 Web UI (HIVE-12550) shows recently completed queries. > However, a user can see the queries run by other users as well, and that is a > security/privacy concern. > Only admin users should be allowed to see queries from other users (similar > to behavior of display for configs, stack trace etc). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17744) Acid LockManager optimization
[ https://issues.apache.org/jira/browse/HIVE-17744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-17744: - > Acid LockManager optimization > - > > Key: HIVE-17744 > URL: https://issues.apache.org/jira/browse/HIVE-17744 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > > does it make sense to periodically compute and store min(lock_id) of a > Write/semi shared lock to know that all earlier locks are Read locks and thus > don't need to be even retrieved from storage to check if a new Read/semi > shared lock can be granted? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17743) Add InterfaceAudience and InterfaceStability annotations for Thrift generated APIs
[
https://issues.apache.org/jira/browse/HIVE-17743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar reassigned HIVE-17743:
---
> Add InterfaceAudience and InterfaceStability annotations for Thrift generated
> APIs
> --
>
> Key: HIVE-17743
> URL: https://issues.apache.org/jira/browse/HIVE-17743
> Project: Hive
> Issue Type: Sub-task
> Components: Thrift API
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>
> The Thrift generated files don't have {{InterfaceAudience}} or
> {{InterfaceStability}} annotations on them, mainly because all the files are
> auto-generated.
> We should add some code that auto-tags all the Java Thrift generated files
> with these annotations. This way even when they are re-generated, they still
> contain the annotations.
> We should be able to do this using the
> {{com.google.code.maven-replacer-plugin}} similar to what we do in
> {{standalone-metastore/pom.xml}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Reopened] (HIVE-17129) Increase usage of InterfaceAudience and InterfaceStability annotations
[
https://issues.apache.org/jira/browse/HIVE-17129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar reopened HIVE-17129:
-
> Increase usage of InterfaceAudience and InterfaceStability annotations
> ---
>
> Key: HIVE-17129
> URL: https://issues.apache.org/jira/browse/HIVE-17129
> Project: Hive
> Issue Type: Improvement
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>
> The {{InterfaceAudience}} and {{InterfaceStability}} annotations were added a
> while ago to mark certain classes as available for public use. However, they
> were only added to a few classes. The annotations are largely missing for
> major APIs such as the SerDe and UDF APIs. We should update these interfaces
> to use these annotations.
> When done in conjunction with HIVE-17130, we should have an automated way to
> prevent backwards incompatible changes to Hive APIs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17718) spark-client and job monitor logging improvements
[
https://issues.apache.org/jira/browse/HIVE-17718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17718:
Summary: spark-client and job monitor logging improvements (was: HS2 Logs
print unnecessary stack trace when HoS query is cancelled)
> spark-client and job monitor logging improvements
> -
>
> Key: HIVE-17718
> URL: https://issues.apache.org/jira/browse/HIVE-17718
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-17718.1.patch, HIVE-17718.2.patch,
> HIVE-17718.3.patch
>
>
> Example:
> {code}
> 2017-10-05 17:47:11,881 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2017-10-05 17:47:11,881 WARN org.apache.hadoop.hive.ql.Driver:
> [HiveServer2-Handler-Pool: Thread-105]: Shutting down task : Stage-2:MAPRED
> 2017-10-05 17:47:11,882 ERROR
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor:
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
>
[jira] [Commented] (HIVE-17742) AccumuloIndexedOutputFormat Use SLF4J
[
https://issues.apache.org/jira/browse/HIVE-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196260#comment-16196260
]
Hive QA commented on HIVE-17742:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12890975/HIVE-17742.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 11190 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
(batchId=231)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
(batchId=231)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=171)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14]
(batchId=239)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7184/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7184/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7184/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12890975 - PreCommit-HIVE-Build
> AccumuloIndexedOutputFormat Use SLF4J
> -
>
> Key: HIVE-17742
> URL: https://issues.apache.org/jira/browse/HIVE-17742
> Project: Hive
> Issue Type: Improvement
> Components: Accumulo Storage Handler
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-17742.1.patch
>
>
> {{org.apache.hadoop.hive.accumulo.mr.AccumuloIndexedOutputFormat}}
> # Change to use SL4J instead of core Log4J classes
> # Use SL4J parameterization
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17740) HiveConf - Use SLF4J Parameterization
[
https://issues.apache.org/jira/browse/HIVE-17740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196205#comment-16196205
]
BELUGA BEHR commented on HIVE-17740:
Failures don't appear to be related
> HiveConf - Use SLF4J Parameterization
> ---
>
> Key: HIVE-17740
> URL: https://issues.apache.org/jira/browse/HIVE-17740
> Project: Hive
> Issue Type: Improvement
> Components: Configuration, Hive
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-17740.patch
>
>
> {{org.apache.hadoop.hive.conf.HiveConf}}
> # Parameterize the SLF4J logging
> # Refactor log variable name to align with rest of code base
> # Couple of small nit-picks
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17732) Minor Improvements - org.apache.hive.hcatalog.data.JsonSerDe.java
[ https://issues.apache.org/jira/browse/HIVE-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196204#comment-16196204 ] BELUGA BEHR commented on HIVE-17732: Failures don't appear to be related > Minor Improvements - org.apache.hive.hcatalog.data.JsonSerDe.java > - > > Key: HIVE-17732 > URL: https://issues.apache.org/jira/browse/HIVE-17732 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 3.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Attachments: HIVE-17732.patch > > > Some simple improvements for org.apache.hive.hcatalog.data.JsonSerDe > Remove superfluous logging, cut down on object instantiation -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17742) AccumuloIndexedOutputFormat Use SLF4J
[
https://issues.apache.org/jira/browse/HIVE-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-17742:
---
Attachment: HIVE-17742.1.patch
> AccumuloIndexedOutputFormat Use SLF4J
> -
>
> Key: HIVE-17742
> URL: https://issues.apache.org/jira/browse/HIVE-17742
> Project: Hive
> Issue Type: Improvement
> Components: Accumulo Storage Handler
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-17742.1.patch
>
>
> {{org.apache.hadoop.hive.accumulo.mr.AccumuloIndexedOutputFormat}}
> # Change to use SL4J instead of core Log4J classes
> # Use SL4J parameterization
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17742) AccumuloIndexedOutputFormat Use SLF4J
[
https://issues.apache.org/jira/browse/HIVE-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-17742:
---
Status: Patch Available (was: Open)
> AccumuloIndexedOutputFormat Use SLF4J
> -
>
> Key: HIVE-17742
> URL: https://issues.apache.org/jira/browse/HIVE-17742
> Project: Hive
> Issue Type: Improvement
> Components: Accumulo Storage Handler
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-17742.1.patch
>
>
> {{org.apache.hadoop.hive.accumulo.mr.AccumuloIndexedOutputFormat}}
> # Change to use SL4J instead of core Log4J classes
> # Use SL4J parameterization
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17742) AccumuloIndexedOutputFormat Use SLF4J
[
https://issues.apache.org/jira/browse/HIVE-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR reassigned HIVE-17742:
--
Assignee: BELUGA BEHR
> AccumuloIndexedOutputFormat Use SLF4J
> -
>
> Key: HIVE-17742
> URL: https://issues.apache.org/jira/browse/HIVE-17742
> Project: Hive
> Issue Type: Improvement
> Components: Accumulo Storage Handler
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
>
> {{org.apache.hadoop.hive.accumulo.mr.AccumuloIndexedOutputFormat}}
> # Change to use SL4J instead of core Log4J classes
> # Use SL4J parameterization
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17740) HiveConf - Use SLF4J Parameterization
[
https://issues.apache.org/jira/browse/HIVE-17740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-17740:
---
Description:
{{org.apache.hadoop.hive.conf.HiveConf}}
# Parameterize the SLF4J logging
# Refactor log variable name to align with rest of code base
# Couple of small nit-picks
was:
{{org.apache.hadoop.hive.conf.HiveConf}}
# Parameterize the SLF4J logging and refactor log variable name to align with
rest of code base
# Couple of small nit-picks
> HiveConf - Use SLF4J Parameterization
> ---
>
> Key: HIVE-17740
> URL: https://issues.apache.org/jira/browse/HIVE-17740
> Project: Hive
> Issue Type: Improvement
> Components: Configuration, Hive
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-17740.patch
>
>
> {{org.apache.hadoop.hive.conf.HiveConf}}
> # Parameterize the SLF4J logging
> # Refactor log variable name to align with rest of code base
> # Couple of small nit-picks
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17740) HiveConf - Use SLF4J Parameterization
[
https://issues.apache.org/jira/browse/HIVE-17740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-17740:
---
Summary: HiveConf - Use SLF4J Parameterization(was: HiveConf 0 Use
SLF4J Parameterization )
> HiveConf - Use SLF4J Parameterization
> ---
>
> Key: HIVE-17740
> URL: https://issues.apache.org/jira/browse/HIVE-17740
> Project: Hive
> Issue Type: Improvement
> Components: Configuration, Hive
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-17740.patch
>
>
> {{org.apache.hadoop.hive.conf.HiveConf}}
> # Parameterize the SLF4J logging and refactor log variable name to align with
> rest of code base
> # Couple of small nit-picks
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17741) hive server2 ha register 0.0.0.0 to zookeeper
[ https://issues.apache.org/jira/browse/HIVE-17741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nie Gus reassigned HIVE-17741: -- > hive server2 ha register 0.0.0.0 to zookeeper > - > > Key: HIVE-17741 > URL: https://issues.apache.org/jira/browse/HIVE-17741 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Nie Gus >Assignee: Nie Gus >Priority: Blocker > > Hive Server2 HA based on zookeeper. > when set hive.server2.thrift.bind.host to "0.0.0.0" > hive server2 will register this "0.0.0.0" to zookeeper, and when client > request the hive server address, it will return the wrong address "0.0.0.0". > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
