date:20220217

[jira] [Commented] (HIVE-13632) Hive failing on insert empty array into parquet table

2022-02-17 Thread Jira



[ 
https://issues.apache.org/jira/browse/HIVE-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17494340#comment-17494340
 ] 

Mr.黄 commented on HIVE-13632:
-

[~hiveqa] Hello, I did not have this problem in Hive 2.1.1 version, but this 
problem reappeared in Hive 3.1.2 version, I do not understand why the lower 
version succeeded and the higher version failed, may I ask whether this bug 
will be fixed later? The following is my version information and error message:

 
{code:java}
spark version: 2.4.0-cdh6.3.2
hive version: 2.1.1-cdh.6.3.2
scala> spark.sql("create table test STORED AS PARQUET as select map() as a")
scala> sql("select * from test").show
+---+                                                                           
|  a|
+---+
| []|
+---+

-
spark version: 2.4.3
hive version: 3.1.2
scala> spark.sql("create table test STORED AS PARQUET as select map() as a")

Caused by: org.apache.spark.SparkException: Task failed while writing rows.
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:257)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  at org.apache.spark.scheduler.Task.run(Task.scala:121)
  at 
org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
  at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Parquet record is malformed: empty 
fields are illegal, the field should be ommited completely instead
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:64)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:59)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:31)
  at 
parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:121)
  at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:123)
  at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:42)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:111)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:124)
  at 
org.apache.spark.sql.hive.execution.HiveOutputWriter.write(HiveFileFormat.scala:149)
  at 
org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.write(FileFormatDataWriter.scala:137)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:245)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:242)
  at 
org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:248)
  ... 10 more
Caused by: parquet.io.ParquetEncodingException: empty fields are illegal, the 
field should be ommited completely instead
  at 
parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.endField(MessageColumnIO.java:244)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeMap(DataWritableWriter.java:241)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeValue(DataWritableWriter.java:116)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeGroupFields(DataWritableWriter.java:89)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:60)
  ... 23 more {code}

> Hive failing on insert empty array into parquet table
> -
>
> Key: HIVE-13632
> URL: https://issues.apache.org/jira/browse/HIVE-13632
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>

[jira] (HIVE-13632) Hive failing on insert empty array into parquet table

2022-02-17 Thread Jira



[ https://issues.apache.org/jira/browse/HIVE-13632 ]


Mr.黄 deleted comment on HIVE-13632:
-

was (Author: JIRAUSER281773):
Hello, I did not have this problem in Hive 2.1.1 version, but this problem 
reappeared in Hive 3.1.2 version, I do not understand why the lower version 
succeeded and the higher version failed, may I ask whether this bug will be 
fixed later? The following is my version information and error message:
spark version: 2.4.0-cdh6.3.2
hive version: 2.1.1-cdh.6.3.2
scala> spark.sql("create table test STORED AS PARQUET as select map() as a")
scala> sql("select * from test").show
+---+                                                                           
|  a|
+---+
| []|
+---+

-
spark version: 2.4.3
hive version: 3.1.2
scala> spark.sql("create table test STORED AS PARQUET as select map() as a")

Caused by: org.apache.spark.SparkException: Task failed while writing rows.
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:257)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  at org.apache.spark.scheduler.Task.run(Task.scala:121)
  at 
org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
  at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Parquet record is malformed: empty 
fields are illegal, the field should be ommited completely instead
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:64)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:59)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:31)
  at 
parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:121)
  at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:123)
  at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:42)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:111)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:124)
  at 
org.apache.spark.sql.hive.execution.HiveOutputWriter.write(HiveFileFormat.scala:149)
  at 
org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.write(FileFormatDataWriter.scala:137)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:245)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:242)
  at 
org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:248)
  ... 10 more
Caused by: parquet.io.ParquetEncodingException: empty fields are illegal, the 
field should be ommited completely instead
  at 
parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.endField(MessageColumnIO.java:244)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeMap(DataWritableWriter.java:241)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeValue(DataWritableWriter.java:116)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeGroupFields(DataWritableWriter.java:89)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:60)
  ... 23 more

> Hive failing on insert empty array into parquet table
> -
>
> Key: HIVE-13632
> URL: https://issues.apache.org/jira/browse/HIVE-13632
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
>Priority: Major
> Fix For: 1.3.0, 2.1.0
>
>

[jira] [Commented] (HIVE-13632) Hive failing on insert empty array into parquet table

2022-02-17 Thread Jira



[ 
https://issues.apache.org/jira/browse/HIVE-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17494338#comment-17494338
 ] 

Mr.黄 commented on HIVE-13632:
-

Hello, I did not have this problem in Hive 2.1.1 version, but this problem 
reappeared in Hive 3.1.2 version, I do not understand why the lower version 
succeeded and the higher version failed, may I ask whether this bug will be 
fixed later? The following is my version information and error message:
spark version: 2.4.0-cdh6.3.2
hive version: 2.1.1-cdh.6.3.2
scala> spark.sql("create table test STORED AS PARQUET as select map() as a")
scala> sql("select * from test").show
+---+                                                                           
|  a|
+---+
| []|
+---+

-
spark version: 2.4.3
hive version: 3.1.2
scala> spark.sql("create table test STORED AS PARQUET as select map() as a")

Caused by: org.apache.spark.SparkException: Task failed while writing rows.
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:257)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  at org.apache.spark.scheduler.Task.run(Task.scala:121)
  at 
org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
  at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Parquet record is malformed: empty 
fields are illegal, the field should be ommited completely instead
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:64)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:59)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:31)
  at 
parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:121)
  at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:123)
  at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:42)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:111)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:124)
  at 
org.apache.spark.sql.hive.execution.HiveOutputWriter.write(HiveFileFormat.scala:149)
  at 
org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.write(FileFormatDataWriter.scala:137)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:245)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:242)
  at 
org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:248)
  ... 10 more
Caused by: parquet.io.ParquetEncodingException: empty fields are illegal, the 
field should be ommited completely instead
  at 
parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.endField(MessageColumnIO.java:244)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeMap(DataWritableWriter.java:241)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeValue(DataWritableWriter.java:116)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeGroupFields(DataWritableWriter.java:89)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:60)
  ... 23 more

> Hive failing on insert empty array into parquet table
> -
>
> Key: HIVE-13632
> URL: https://issues.apache.org/jira/browse/HIVE-13632
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
>

[jira] (HIVE-13632) Hive failing on insert empty array into parquet table

2022-02-17 Thread Jira



[ https://issues.apache.org/jira/browse/HIVE-13632 ]


Mr.黄 deleted comment on HIVE-13632:
-

was (Author: JIRAUSER281773):
spark version: 2.4.0-cdh6.3.2
hive version: 2.1.1-cdh.6.3.2
scala> spark.sql("create table test STORED AS PARQUET as select map() as a")
scala> sql("select * from test").show
+---+                                                                           
|  a|
+---+
| []|
+---+

-
spark version: 2.4.3
hive version: 3.1.2
scala> spark.sql("create table test STORED AS PARQUET as select map() as a")

Caused by: org.apache.spark.SparkException: Task failed while writing rows.
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:257)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  at org.apache.spark.scheduler.Task.run(Task.scala:121)
  at 
org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
  at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Parquet record is malformed: empty 
fields are illegal, the field should be ommited completely instead
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:64)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:59)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:31)
  at 
parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:121)
  at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:123)
  at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:42)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:111)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:124)
  at 
org.apache.spark.sql.hive.execution.HiveOutputWriter.write(HiveFileFormat.scala:149)
  at 
org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.write(FileFormatDataWriter.scala:137)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:245)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:242)
  at 
org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:248)
  ... 10 more
Caused by: parquet.io.ParquetEncodingException: empty fields are illegal, the 
field should be ommited completely instead
  at 
parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.endField(MessageColumnIO.java:244)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeMap(DataWritableWriter.java:241)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeValue(DataWritableWriter.java:116)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeGroupFields(DataWritableWriter.java:89)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:60)
  ... 23 more

> Hive failing on insert empty array into parquet table
> -
>
> Key: HIVE-13632
> URL: https://issues.apache.org/jira/browse/HIVE-13632
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
>Priority: Major
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-13632.1.patch, HIVE-13632.2.patch, 
> HIVE-13632.3.patch
>
>
> The insert will fail with following stack:
> {noformat}
> by: parquet.io.ParquetEncodingException: empty fields are illegal, the field 
> should be ommited completely instead
>   at 
>

[jira] [Commented] (HIVE-13632) Hive failing on insert empty array into parquet table

2022-02-17 Thread Jira



[ 
https://issues.apache.org/jira/browse/HIVE-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17494337#comment-17494337
 ] 

Mr.黄 commented on HIVE-13632:
-

spark version: 2.4.0-cdh6.3.2
hive version: 2.1.1-cdh.6.3.2
scala> spark.sql("create table test STORED AS PARQUET as select map() as a")
scala> sql("select * from test").show
+---+                                                                           
|  a|
+---+
| []|
+---+

-
spark version: 2.4.3
hive version: 3.1.2
scala> spark.sql("create table test STORED AS PARQUET as select map() as a")

Caused by: org.apache.spark.SparkException: Task failed while writing rows.
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:257)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  at org.apache.spark.scheduler.Task.run(Task.scala:121)
  at 
org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
  at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Parquet record is malformed: empty 
fields are illegal, the field should be ommited completely instead
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:64)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:59)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:31)
  at 
parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:121)
  at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:123)
  at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:42)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:111)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:124)
  at 
org.apache.spark.sql.hive.execution.HiveOutputWriter.write(HiveFileFormat.scala:149)
  at 
org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.write(FileFormatDataWriter.scala:137)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:245)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:242)
  at 
org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:248)
  ... 10 more
Caused by: parquet.io.ParquetEncodingException: empty fields are illegal, the 
field should be ommited completely instead
  at 
parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.endField(MessageColumnIO.java:244)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeMap(DataWritableWriter.java:241)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeValue(DataWritableWriter.java:116)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeGroupFields(DataWritableWriter.java:89)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:60)
  ... 23 more

> Hive failing on insert empty array into parquet table
> -
>
> Key: HIVE-13632
> URL: https://issues.apache.org/jira/browse/HIVE-13632
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
>Priority: Major
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-13632.1.patch, HIVE-13632.2.patch, 
> HIVE-13632.3.patch
>
>
> The insert will fail with following stack:
> {noformat}
> by: parquet.io.ParquetEncodingException: empty fields are illegal, the field 
> should be ommited

[jira] [Commented] (HIVE-11625) Map instances with null keys are not properly handled for Parquet tables

2022-02-17 Thread Jira



[ 
https://issues.apache.org/jira/browse/HIVE-11625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17494334#comment-17494334
 ] 

Mr.黄 commented on HIVE-11625:
-

Hello, I did not have this problem in Hive 2.1.1 version, but this problem 
reappeared in Hive 3.1.2 version, I do not understand why the lower version 
succeeded and the higher version failed, may I ask whether this bug will be 
fixed later? The following is my version information and error message:
{code:java}
spark version: 2.4.0-cdh6.3.2
hive version: 2.1.1-cdh.6.3.2
scala> spark.sql("create table test STORED AS PARQUET as select map() as a")
scala> sql("select * from test").show
+---+                                                                           
|  a|
+---+
| []|
+---+

-
spark version: 2.4.3
hive version: 3.1.2
scala> spark.sql("create table test STORED AS PARQUET as select map() as a")

Caused by: org.apache.spark.SparkException: Task failed while writing rows.
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:257)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  at org.apache.spark.scheduler.Task.run(Task.scala:121)
  at 
org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
  at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Parquet record is malformed: empty 
fields are illegal, the field should be ommited completely instead
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:64)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:59)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:31)
  at 
parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:121)
  at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:123)
  at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:42)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:111)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:124)
  at 
org.apache.spark.sql.hive.execution.HiveOutputWriter.write(HiveFileFormat.scala:149)
  at 
org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.write(FileFormatDataWriter.scala:137)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:245)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:242)
  at 
org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)
  at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:248)
  ... 10 more
Caused by: parquet.io.ParquetEncodingException: empty fields are illegal, the 
field should be ommited completely instead
  at 
parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.endField(MessageColumnIO.java:244)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeMap(DataWritableWriter.java:241)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeValue(DataWritableWriter.java:116)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeGroupFields(DataWritableWriter.java:89)
  at 
org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:60)
  ... 23 more {code}

> Map instances with null keys are not properly handled for Parquet tables
> 
>
> Key: HIVE-11625
> URL: https://issues.apache.org/jira/browse/HIVE-11625
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 0.13.1, 0.14.0, 1.0.1, 1.1.1, 1.2.1
>Reporter: Cheng Lian
>

[jira] [Work logged] (HIVE-25750) Beeline: Creating a standalone tarball by isolating dependencies

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25750?focusedWorklogId=729359=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-729359
 ]

ASF GitHub Bot logged work on HIVE-25750:
-

Author: ASF GitHub Bot
Created on: 18/Feb/22 00:55
Start Date: 18/Feb/22 00:55
Worklog Time Spent: 10m 
  Work Description: achennagiri commented on a change in pull request #2824:
URL: https://github.com/apache/hive/pull/2824#discussion_r809592831



##
File path: beeline/pom.xml
##
@@ -233,6 +233,53 @@
   
 
   
+  
+org.apache.maven.plugins
+maven-shade-plugin
+
+  
+package
+
+  shade
+
+
+  
+jar-with-dependencies
+  
+  jar-with-dependencies
+  
+
+
+  org.openjdk.jmh.Main
+
+  
+  
+
+  
+  *:*
+  
+META-INF/*.SF
+META-INF/*.DSA
+META-INF/*.RSA
+  
+
+  
+
+  
+
+
+  
+com.github.edwgiz
+
maven-shade-plugin.log4j2-cachefile-transformer
+2.1

Review comment:
   Umm, this is the version of the maven shade plugin and not the log4j 
version.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 729359)
Time Spent: 2h 20m  (was: 2h 10m)

> Beeline: Creating a standalone tarball by isolating dependencies
> 
>
> Key: HIVE-25750
> URL: https://issues.apache.org/jira/browse/HIVE-25750
> Project: Hive
>  Issue Type: Bug
>Reporter: Abhay
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> The code to create a standalone beeline tarball was created as part of this 
> ticket https://issues.apache.org/jira/browse/HIVE-24348. However, a bug was 
> reported in the case when the beeline is tried to install without the hadoop 
> installed. 
> The beeline script complains of missing dependencies when it is run.
> The ask as part of this ticket is to fix that bug. 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25750) Beeline: Creating a standalone tarball by isolating dependencies

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25750?focusedWorklogId=729358=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-729358
 ]

ASF GitHub Bot logged work on HIVE-25750:
-

Author: ASF GitHub Bot
Created on: 18/Feb/22 00:55
Start Date: 18/Feb/22 00:55
Worklog Time Spent: 10m 
  Work Description: achennagiri commented on a change in pull request #2824:
URL: https://github.com/apache/hive/pull/2824#discussion_r809592526



##
File path: beeline/pom.xml
##
@@ -233,6 +233,53 @@
   
 
   
+  
+org.apache.maven.plugins
+maven-shade-plugin
+
+  
+package
+
+  shade
+
+
+  
+jar-with-dependencies
+  
+  jar-with-dependencies

Review comment:
   If the above comment didn't make sense, let me know :)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 729358)
Time Spent: 2h 10m  (was: 2h)

> Beeline: Creating a standalone tarball by isolating dependencies
> 
>
> Key: HIVE-25750
> URL: https://issues.apache.org/jira/browse/HIVE-25750
> Project: Hive
>  Issue Type: Bug
>Reporter: Abhay
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> The code to create a standalone beeline tarball was created as part of this 
> ticket https://issues.apache.org/jira/browse/HIVE-24348. However, a bug was 
> reported in the case when the beeline is tried to install without the hadoop 
> installed. 
> The beeline script complains of missing dependencies when it is run.
> The ask as part of this ticket is to fix that bug. 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25750) Beeline: Creating a standalone tarball by isolating dependencies

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25750?focusedWorklogId=729357=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-729357
 ]

ASF GitHub Bot logged work on HIVE-25750:
-

Author: ASF GitHub Bot
Created on: 18/Feb/22 00:54
Start Date: 18/Feb/22 00:54
Worklog Time Spent: 10m 
  Work Description: achennagiri commented on a change in pull request #2824:
URL: https://github.com/apache/hive/pull/2824#discussion_r809592375



##
File path: beeline/pom.xml
##
@@ -233,6 +233,53 @@
   
 
   
+  
+org.apache.maven.plugins
+maven-shade-plugin
+
+  
+package
+
+  shade
+
+
+  
+jar-with-dependencies
+  
+  jar-with-dependencies

Review comment:
   **_apache-hive-beeline-4.0.0-SNAPSHOT.tar.gz_** is what the final tar gz 
output is. To give you a better context, I added this file 
https://github.com/achennagiri/hive/blob/a5dab6e8d1b41dd0bbbd0f876be59e24521795d4/packaging/src/main/assembly/beeline.xml
 as part of this PR https://github.com/apache/hive/pull/1906/files. As part of 
that, I forgot to add the this piece of code which builds the beeline jar with 
its dependencies. So, this uber/shaded jar had some problems with the log4j 
versioning and hence the use of the transformer code




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 729357)
Time Spent: 2h  (was: 1h 50m)

> Beeline: Creating a standalone tarball by isolating dependencies
> 
>
> Key: HIVE-25750
> URL: https://issues.apache.org/jira/browse/HIVE-25750
> Project: Hive
>  Issue Type: Bug
>Reporter: Abhay
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> The code to create a standalone beeline tarball was created as part of this 
> ticket https://issues.apache.org/jira/browse/HIVE-24348. However, a bug was 
> reported in the case when the beeline is tried to install without the hadoop 
> installed. 
> The beeline script complains of missing dependencies when it is run.
> The ask as part of this ticket is to fix that bug. 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25750) Beeline: Creating a standalone tarball by isolating dependencies

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25750?focusedWorklogId=729352=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-729352
 ]

ASF GitHub Bot logged work on HIVE-25750:
-

Author: ASF GitHub Bot
Created on: 18/Feb/22 00:37
Start Date: 18/Feb/22 00:37
Worklog Time Spent: 10m 
  Work Description: achennagiri commented on pull request #2824:
URL: https://github.com/apache/hive/pull/2824#issuecomment-1043677756


   > 
   
   So,  Apache Log4j is a logging library right. Almost every other jar has a 
dependency on this. We need this to make all the jars that are included as part 
of the big uber jar work properly and the beeline java utility runs without any 
hiccup.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 729352)
Time Spent: 1h 50m  (was: 1h 40m)

> Beeline: Creating a standalone tarball by isolating dependencies
> 
>
> Key: HIVE-25750
> URL: https://issues.apache.org/jira/browse/HIVE-25750
> Project: Hive
>  Issue Type: Bug
>Reporter: Abhay
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> The code to create a standalone beeline tarball was created as part of this 
> ticket https://issues.apache.org/jira/browse/HIVE-24348. However, a bug was 
> reported in the case when the beeline is tried to install without the hadoop 
> installed. 
> The beeline script complains of missing dependencies when it is run.
> The ask as part of this ticket is to fix that bug. 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25750) Beeline: Creating a standalone tarball by isolating dependencies

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25750?focusedWorklogId=729351=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-729351
 ]

ASF GitHub Bot logged work on HIVE-25750:
-

Author: ASF GitHub Bot
Created on: 18/Feb/22 00:32
Start Date: 18/Feb/22 00:32
Worklog Time Spent: 10m 
  Work Description: achennagiri opened a new pull request #3043:
URL: https://github.com/apache/hive/pull/3043


   The code to create a standalone beeline tarball was created as part of this 
ticket https://issues.apache.org/jira/browse/HIVE-24348. However, a bug was 
reported in the case when the beeline is tried to install without the hadoop 
installed. 
   The beeline script complains of missing dependencies when it is run. 
   
   Update:
   Was running in to the below error with the file mode on in Beeline
   
   ```
   Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.JobConf
   at 
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
   at 
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
   ```
   Added a fix to resolve this.
   
   ### What changes were proposed in this pull request?
   The beeline script can be run with/without hadoop installed. All the 
required dependencies are bundled into a single downloadable tar file. 
   `mvn clean package install -Pdist -Pitests -DskipTests -Denforcer.skip=true` 
generates something along the lines of 
   **apache-hive-beeline-4.0.0-SNAPSHOT.tar.gz** in the **packaging/target** 
folder.
   
   
   
   ### Does this PR introduce _any_ user-facing change?
   No.
   
   
   ### How was this patch tested?
   
   Created a docker container using the command
   `sudo docker run  --rm -it -v /Users/achennagiri/Downloads:/container --user 
root docker-private.infra.cloudera.com/cloudera_base/ubi8/python-38:1-68 
/bin/bash`
   
   Need to install `yum install -y java-11-openjdk` java in the container. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 729351)
Time Spent: 1h 40m  (was: 1.5h)

> Beeline: Creating a standalone tarball by isolating dependencies
> 
>
> Key: HIVE-25750
> URL: https://issues.apache.org/jira/browse/HIVE-25750
> Project: Hive
>  Issue Type: Bug
>Reporter: Abhay
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> The code to create a standalone beeline tarball was created as part of this 
> ticket https://issues.apache.org/jira/browse/HIVE-24348. However, a bug was 
> reported in the case when the beeline is tried to install without the hadoop 
> installed. 
> The beeline script complains of missing dependencies when it is run.
> The ask as part of this ticket is to fix that bug. 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25959) Expose Compaction Observability delta metrics using the JsonReporter

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25959?focusedWorklogId=729225=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-729225
 ]

ASF GitHub Bot logged work on HIVE-25959:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 20:30
Start Date: 17/Feb/22 20:30
Worklog Time Spent: 10m 
  Work Description: vcsomor commented on a change in pull request #3032:
URL: https://github.com/apache/hive/pull/3032#discussion_r809442897



##
File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/Metrics.java
##
@@ -160,6 +159,47 @@ public Integer getValue() {
 }
   }
 
+  /**
+   * Get a Map that represents a multi-field metric,
+   * or create a new one if it does not already exist.
+   * @param name Name of map metric.  This should come from MetricConstants
+   * @return MapMetric .
+   */
+  public static MapMetrics getOrCreateMapMetrics(String name) {
+if (self == null) {
+  return dummyMapMetrics;
+}
+
+Map metrics = self.registry.getMetrics();
+Metric map = metrics.get(name);
+if (map instanceof MapMetrics) {
+  return (MapMetrics) map;
+}
+
+// Looks like it doesn't exist.  Lock so that two threads don't create it 
at once.
+synchronized (Metrics.class) {
+  // Recheck to make sure someone didn't create it while we waited.

Review comment:
   Fixed and thanks




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 729225)
Time Spent: 2h 20m  (was: 2h 10m)

> Expose Compaction Observability delta metrics using the JsonReporter
> 
>
> Key: HIVE-25959
> URL: https://issues.apache.org/jira/browse/HIVE-25959
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 4.0.0
>Reporter: Viktor Csomor
>Assignee: Viktor Csomor
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> As part of the HIVE-25897 the Delta metrics collection has been moved from 
> HiveServer2 to HMS. Hence the HMS does not have any jmx endpoint exposed by 
> default we'd like to expose these metrics through the JsonReporter



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25947) Compactor job queue cannot be set per table via compactor.mapred.job.queue.name

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25947?focusedWorklogId=729214=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-729214
 ]

ASF GitHub Bot logged work on HIVE-25947:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 20:03
Start Date: 17/Feb/22 20:03
Worklog Time Spent: 10m 
  Work Description: deniskuzZ commented on a change in pull request #3027:
URL: https://github.com/apache/hive/pull/3027#discussion_r809424757



##
File path: 
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorUtil.java
##
@@ -62,31 +75,29 @@ public static ExecutorService 
createExecutorWithThreadFactory(int parallelism, S
* @param conf global hive conf
* @param ci compaction info object
* @param table instance of table
-   * @return name of the queue, can be null
+   * @return name of the queue
*/
   static String getCompactorJobQueueName(HiveConf conf, CompactionInfo ci, 
Table table) {
 // Get queue name from the ci. This is passed through
 // ALTER TABLE table_name COMPACT 'major' WITH OVERWRITE 
TBLPROPERTIES('compactor.hive.compactor.job.queue'='some_queue')
+List> propertyGetters = new ArrayList<>(2);
 if (ci.properties != null) {
   StringableMap ciProperties = new StringableMap(ci.properties);
-  String queueName = ciProperties.get(COMPACTOR_PREFIX + 
MAPRED_QUEUE_NAME);
-  if (queueName != null && queueName.length() > 0) {
-return queueName;
-  }
+  propertyGetters.add(ciProperties::get);
 }
-
-// Get queue name from the table properties
-String queueName = table.getParameters().get(COMPACTOR_PREFIX + 
MAPRED_QUEUE_NAME);
-if (queueName != null && queueName.length() > 0) {
-  return queueName;
+if (table.getParameters() != null) {
+  propertyGetters.add(table.getParameters()::get);
 }
 
-// Get queue name from global hive conf
-queueName = conf.get(HiveConf.ConfVars.COMPACTOR_JOB_QUEUE.varname);
-if (queueName != null && queueName.length() > 0) {
-  return queueName;
+for (Function getter : propertyGetters) {

Review comment:
   that's ok




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 729214)
Time Spent: 2h 10m  (was: 2h)

> Compactor job queue cannot be set per table via 
> compactor.mapred.job.queue.name
> ---
>
> Key: HIVE-25947
> URL: https://issues.apache.org/jira/browse/HIVE-25947
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Before HIVE-20723 it was possible to schedule the compaction for each table 
> on specific job queues by putting {{compactor.mapred.job.queue.name}} in the 
> table properties. 
> {code:sql}
> CREATE TABLE person (name STRING, age INT) STORED AS ORC TBLPROPERTIES(
> 'transactional'='true',
> 'compactor.mapred.job.queue.name'='root.user2);
> ALTER TABLE person COMPACT 'major' WITH OVERWRITE 
> TBLPROPERTIES('compactor.mapred.job.queue.name'='root.user2')
> {code}
> This is no longer possible (after HIVE-20723) and in order to achieve the 
> same effect someone needs to use the {{compactor.hive.compactor.job.queue}}.
> {code:sql}
> CREATE TABLE person (name STRING, age INT) STORED AS ORC TBLPROPERTIES(
> 'transactional'='true',
> 'compactor.hive.compactor.job.queue'='root.user2);
> ALTER TABLE person COMPACT 'major' WITH OVERWRITE 
> TBLPROPERTIES('compactor.hive.compactor.job.queue'='root.user2')
> {code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25957) Fix password based authentication with SAML enabled

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25957?focusedWorklogId=729213=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-729213
 ]

ASF GitHub Bot logged work on HIVE-25957:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 20:02
Start Date: 17/Feb/22 20:02
Worklog Time Spent: 10m 
  Work Description: yongzhi merged pull request #3028:
URL: https://github.com/apache/hive/pull/3028


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 729213)
Time Spent: 1.5h  (was: 1h 20m)

> Fix password based authentication with SAML enabled
> ---
>
> Key: HIVE-25957
> URL: https://issues.apache.org/jira/browse/HIVE-25957
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Yu-Wen Lai
>Assignee: Yu-Wen Lai
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> In HIVE-25875, we allowed SAML to be set with other password based 
> authentication, but we pass NONE to the function doPasswordAuth. That is, any 
> requests use basic authentication header can bypass the password verification 
> because NONE means a no-op authentication.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25959) Expose Compaction Observability delta metrics using the JsonReporter

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25959?focusedWorklogId=729199=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-729199
 ]

ASF GitHub Bot logged work on HIVE-25959:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 19:47
Start Date: 17/Feb/22 19:47
Worklog Time Spent: 10m 
  Work Description: vcsomor commented on a change in pull request #3032:
URL: https://github.com/apache/hive/pull/3032#discussion_r809413256



##
File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/Metrics.java
##
@@ -160,6 +159,47 @@ public Integer getValue() {
 }
   }
 
+  /**
+   * Get a Map that represents a multi-field metric,
+   * or create a new one if it does not already exist.
+   * @param name Name of map metric.  This should come from MetricConstants
+   * @return MapMetric .
+   */
+  public static MapMetrics getOrCreateMapMetrics(String name) {
+if (self == null) {
+  return dummyMapMetrics;
+}
+
+Map metrics = self.registry.getMetrics();
+Metric map = metrics.get(name);
+if (map instanceof MapMetrics) {
+  return (MapMetrics) map;
+}
+
+// Looks like it doesn't exist.  Lock so that two threads don't create it 
at once.
+synchronized (Metrics.class) {
+  // Recheck to make sure someone didn't create it while we waited.

Review comment:
   ohh sorry I've missed that
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 729199)
Time Spent: 2h 10m  (was: 2h)

> Expose Compaction Observability delta metrics using the JsonReporter
> 
>
> Key: HIVE-25959
> URL: https://issues.apache.org/jira/browse/HIVE-25959
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 4.0.0
>Reporter: Viktor Csomor
>Assignee: Viktor Csomor
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> As part of the HIVE-25897 the Delta metrics collection has been moved from 
> HiveServer2 to HMS. Hence the HMS does not have any jmx endpoint exposed by 
> default we'd like to expose these metrics through the JsonReporter



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25178) Reduce number of getPartition calls during loadDynamicPartitions for External Tables

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25178?focusedWorklogId=729166=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-729166
 ]

ASF GitHub Bot logged work on HIVE-25178:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 19:00
Start Date: 17/Feb/22 19:00
Worklog Time Spent: 10m 
  Work Description: sourabh912 commented on pull request #3042:
URL: https://github.com/apache/hive/pull/3042#issuecomment-1043307276


   @nrg4878 : Please review and share your feedback


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 729166)
Time Spent: 40m  (was: 0.5h)

> Reduce number of getPartition calls during loadDynamicPartitions for External 
> Tables
> 
>
> Key: HIVE-25178
> URL: https://issues.apache.org/jira/browse/HIVE-25178
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: Rajesh Balamohan
>Assignee: Narayanan Venkateswaran
>Priority: Major
>  Labels: performance, pull-request-available
> Attachments: 
> 0001-HIVE-25178-Reduce-number-of-getPartition-calls-durin.patch
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> When dynamic partitions are loaded, Hive::loadDynamicPartition loads all 
> partitions from HMS causing heavy load on it. This becomes worse when large 
> number of partitions are present in tables.
> Only relevant partitions being loaded in dynamic partitions can be queried 
> from HMS for partition existence.
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2958]



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25178) Reduce number of getPartition calls during loadDynamicPartitions for External Tables

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25178?focusedWorklogId=729165=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-729165
 ]

ASF GitHub Bot logged work on HIVE-25178:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 18:59
Start Date: 17/Feb/22 18:59
Worklog Time Spent: 10m 
  Work Description: sourabh912 opened a new pull request #3042:
URL: https://github.com/apache/hive/pull/3042


   
   
   
   ### What changes were proposed in this pull request?
   
   Enabling the property hive.load.dynamic.partitions.scan.specific.partitions 
othewise a user sees significant performance degradation in insert/insert 
overwrite queries for large partitioned tables
   
   ### Why are the changes needed?
   
   Improves performance for insert/insert overwrite queries for large 
partitioned tables. 
   ### Does this PR introduce _any_ user-facing change?
   
   No
   
   
   ### How was this patch tested?
   
   Relying on existing tests since it is just a configuration change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 729165)
Time Spent: 0.5h  (was: 20m)

> Reduce number of getPartition calls during loadDynamicPartitions for External 
> Tables
> 
>
> Key: HIVE-25178
> URL: https://issues.apache.org/jira/browse/HIVE-25178
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: Rajesh Balamohan
>Assignee: Narayanan Venkateswaran
>Priority: Major
>  Labels: performance, pull-request-available
> Attachments: 
> 0001-HIVE-25178-Reduce-number-of-getPartition-calls-durin.patch
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> When dynamic partitions are loaded, Hive::loadDynamicPartition loads all 
> partitions from HMS causing heavy load on it. This becomes worse when large 
> number of partitions are present in tables.
> Only relevant partitions being loaded in dynamic partitions can be queried 
> from HMS for partition existence.
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2958]



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work started] (HIVE-25953) Drop HiveRelMdPredicates::getPredicates(Join...) to use that of RelMdPredicates

2022-02-17 Thread Alessandro Solimando (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-25953 started by Alessandro Solimando.
---
> Drop HiveRelMdPredicates::getPredicates(Join...) to use that of 
> RelMdPredicates
> ---
>
> Key: HIVE-25953
> URL: https://issues.apache.org/jira/browse/HIVE-25953
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The goal of the ticket is to unify the two implementations and remove the 
> override in HiveRelMdPredicates.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25967) Prevent residual expressions from getting serialized in Iceberg splits

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25967?focusedWorklogId=729111=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-729111
 ]

ASF GitHub Bot logged work on HIVE-25967:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 17:53
Start Date: 17/Feb/22 17:53
Worklog Time Spent: 10m 
  Work Description: szlta opened a new pull request #3041:
URL: https://github.com/apache/hive/pull/3041


   testing only for now..


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 729111)
Remaining Estimate: 0h
Time Spent: 10m

> Prevent residual expressions from getting serialized in Iceberg splits
> --
>
> Key: HIVE-25967
> URL: https://issues.apache.org/jira/browse/HIVE-25967
> Project: Hive
>  Issue Type: Bug
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (HIVE-25967) Prevent residual expressions from getting serialized in Iceberg splits

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-25967:
--
Labels: pull-request-available  (was: )

> Prevent residual expressions from getting serialized in Iceberg splits
> --
>
> Key: HIVE-25967
> URL: https://issues.apache.org/jira/browse/HIVE-25967
> Project: Hive
>  Issue Type: Bug
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Assigned] (HIVE-25967) Prevent residual expressions from getting serialized in Iceberg splits

2022-02-17 Thread Jira



 [ 
https://issues.apache.org/jira/browse/HIVE-25967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ádám Szita reassigned HIVE-25967:
-


> Prevent residual expressions from getting serialized in Iceberg splits
> --
>
> Key: HIVE-25967
> URL: https://issues.apache.org/jira/browse/HIVE-25967
> Project: Hive
>  Issue Type: Bug
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25963) Temporary table creation with not null constraint gets converted to external table

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25963?focusedWorklogId=729109=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-729109
 ]

ASF GitHub Bot logged work on HIVE-25963:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 17:45
Start Date: 17/Feb/22 17:45
Worklog Time Spent: 10m 
  Work Description: sourabh912 commented on pull request #3040:
URL: https://github.com/apache/hive/pull/3040#issuecomment-1043233911


   @nrg4878 : Please review and provide your feedback.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 729109)
Time Spent: 20m  (was: 10m)

> Temporary table creation with not null constraint gets converted to external 
> table 
> ---
>
> Key: HIVE-25963
> URL: https://issues.apache.org/jira/browse/HIVE-25963
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, Standalone Metastore
>Reporter: Sourabh Goyal
>Assignee: Sourabh Goyal
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> When creating a temporary table with not null, constraint it gets covered to 
> external table. For example: 
> create temporary table t2 (a int not null);
> table t2' metadata looks like: 
> {code:java}
> +---+++
> |   col_name| data_type   
>|  comment   |
> +---+++
> | a | int 
>||
> |   | NULL
>| NULL   |
> | # Detailed Table Information  | NULL
>| NULL   |
> | Database: | default 
>| NULL   |
> | OwnerType:| USER
>| NULL   |
> | Owner:| sourabh 
>| NULL   |
> | CreateTime:   | Tue Feb 15 15:20:13 PST 2022
>| NULL   |
> | LastAccessTime:   | UNKNOWN 
>| NULL   |
> | Retention:| 0   
>| NULL   |
> | Location: | 
> hdfs://localhost:9000/tmp/hive/sourabh/80d374a8-cd7a-4fcf-ae72-51b04ff9c3d8/_tmp_space.db/4574446d-c144-48f9-b4b6-2e9ee0ce5be4
>  | NULL   |
> | Table Type:   | EXTERNAL_TABLE  
>| NULL   |
> | Table Parameters: | NULL
>| NULL   |
> |   | COLUMN_STATS_ACCURATE   
>| {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"a\":\"true\"}} |
> |   | EXTERNAL
>| TRUE   |
> |   | TRANSLATED_TO_EXTERNAL  
>| TRUE   |
> |   | bucketing_version   
>| 2  |
> |   | external.table.purge
>| TRUE   |
> |   | numFiles
>| 0  |
> |   |

[jira] [Updated] (HIVE-25963) Temporary table creation with not null constraint gets converted to external table

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-25963:
--
Labels: pull-request-available  (was: )

> Temporary table creation with not null constraint gets converted to external 
> table 
> ---
>
> Key: HIVE-25963
> URL: https://issues.apache.org/jira/browse/HIVE-25963
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, Standalone Metastore
>Reporter: Sourabh Goyal
>Assignee: Sourabh Goyal
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> When creating a temporary table with not null, constraint it gets covered to 
> external table. For example: 
> create temporary table t2 (a int not null);
> table t2' metadata looks like: 
> {code:java}
> +---+++
> |   col_name| data_type   
>|  comment   |
> +---+++
> | a | int 
>||
> |   | NULL
>| NULL   |
> | # Detailed Table Information  | NULL
>| NULL   |
> | Database: | default 
>| NULL   |
> | OwnerType:| USER
>| NULL   |
> | Owner:| sourabh 
>| NULL   |
> | CreateTime:   | Tue Feb 15 15:20:13 PST 2022
>| NULL   |
> | LastAccessTime:   | UNKNOWN 
>| NULL   |
> | Retention:| 0   
>| NULL   |
> | Location: | 
> hdfs://localhost:9000/tmp/hive/sourabh/80d374a8-cd7a-4fcf-ae72-51b04ff9c3d8/_tmp_space.db/4574446d-c144-48f9-b4b6-2e9ee0ce5be4
>  | NULL   |
> | Table Type:   | EXTERNAL_TABLE  
>| NULL   |
> | Table Parameters: | NULL
>| NULL   |
> |   | COLUMN_STATS_ACCURATE   
>| {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"a\":\"true\"}} |
> |   | EXTERNAL
>| TRUE   |
> |   | TRANSLATED_TO_EXTERNAL  
>| TRUE   |
> |   | bucketing_version   
>| 2  |
> |   | external.table.purge
>| TRUE   |
> |   | numFiles
>| 0  |
> |   | numRows 
>| 0  |
> |   | rawDataSize 
>| 0  |
> |   | totalSize   
>| 0  |
> |   | transient_lastDdlTime   
>| 1644967213 |
> |   | NULL
>| NULL   |
> | # Storage Information | NULL
>| NULL

[jira] [Work logged] (HIVE-25963) Temporary table creation with not null constraint gets converted to external table

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25963?focusedWorklogId=729108=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-729108
 ]

ASF GitHub Bot logged work on HIVE-25963:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 17:44
Start Date: 17/Feb/22 17:44
Worklog Time Spent: 10m 
  Work Description: sourabh912 opened a new pull request #3040:
URL: https://github.com/apache/hive/pull/3040


   
   
   ### What changes were proposed in this pull request?
   
   This PR fixes an issue in which a temporary table when created with not null 
column constraint, gets converted to external table.  It is because HMSClient 
api createTableWithConstraints() call createTable(createTableRequest) which is 
not overriden by SessionHiveMetaStoreClient (which handles temporary tables 
creation/fetch/deletion). As a result of which HMS client makes create_table() 
RPC call to HMS server which in turn changes the table to external table and 
also persists its metadata. This is not the desired behavior because temporary 
tables are stored in memory and the metadata for these should not persisted in 
HMS.   
   
   ### Why are the changes needed?
   
   It fixes a bug as described in HIVE-25963
   
   ### Does this PR introduce _any_ user-facing change?
   
   No
   
   ### How was this patch tested?
   
   Enhanced the existing unit tests by adding positive cases. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 729108)
Remaining Estimate: 0h
Time Spent: 10m

> Temporary table creation with not null constraint gets converted to external 
> table 
> ---
>
> Key: HIVE-25963
> URL: https://issues.apache.org/jira/browse/HIVE-25963
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, Standalone Metastore
>Reporter: Sourabh Goyal
>Assignee: Sourabh Goyal
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> When creating a temporary table with not null, constraint it gets covered to 
> external table. For example: 
> create temporary table t2 (a int not null);
> table t2' metadata looks like: 
> {code:java}
> +---+++
> |   col_name| data_type   
>|  comment   |
> +---+++
> | a | int 
>||
> |   | NULL
>| NULL   |
> | # Detailed Table Information  | NULL
>| NULL   |
> | Database: | default 
>| NULL   |
> | OwnerType:| USER
>| NULL   |
> | Owner:| sourabh 
>| NULL   |
> | CreateTime:   | Tue Feb 15 15:20:13 PST 2022
>| NULL   |
> | LastAccessTime:   | UNKNOWN 
>| NULL   |
> | Retention:| 0   
>| NULL   |
> | Location: | 
> hdfs://localhost:9000/tmp/hive/sourabh/80d374a8-cd7a-4fcf-ae72-51b04ff9c3d8/_tmp_space.db/4574446d-c144-48f9-b4b6-2e9ee0ce5be4
>  | NULL   |
> | Table Type:   | EXTERNAL_TABLE  
>| NULL   |
> | Table Parameters: | NULL
>| NULL   |
> |   |

[jira] [Updated] (HIVE-21152) Rewrite if expression to case and recognize simple case as an if

2022-02-17 Thread Zoltan Haindrich (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-21152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich updated HIVE-21152:

Fix Version/s: 4.0.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

merged into master. Thank you [~kkasa] for revieweing the changes!

> Rewrite if expression to case and recognize simple case as an if
> 
>
> Key: HIVE-21152
> URL: https://issues.apache.org/jira/browse/HIVE-21152
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21152.01.patch, HIVE-21152.02.patch, 
> HIVE-21152.03.patch, HIVE-21152.04.patch, HIVE-21152.05.patch, 
> HIVE-21152.06.patch, HIVE-21152.07.patch
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> * {{IF}} is not part of the sql standard; however given its special form its 
> simpler - and currently in Hive it also has vectorized support
> * people writing standard sql may write: {{CASE WHEN member=1 THEN attr+1 
> else attr+2 END}} which is essentially an if.
> The idea is to rewrite IFs to CASEs for the cbo; and recognize simple 
> "CASE"-s as IFs to get vectorization on them if possible



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-21152) Rewrite if expression to case and recognize simple case as an if

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-21152?focusedWorklogId=729087=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-729087
 ]

ASF GitHub Bot logged work on HIVE-21152:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 17:12
Start Date: 17/Feb/22 17:12
Worklog Time Spent: 10m 
  Work Description: kgyrtkirk merged pull request #2791:
URL: https://github.com/apache/hive/pull/2791


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 729087)
Time Spent: 20m  (was: 10m)

> Rewrite if expression to case and recognize simple case as an if
> 
>
> Key: HIVE-21152
> URL: https://issues.apache.org/jira/browse/HIVE-21152
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-21152.01.patch, HIVE-21152.02.patch, 
> HIVE-21152.03.patch, HIVE-21152.04.patch, HIVE-21152.05.patch, 
> HIVE-21152.06.patch, HIVE-21152.07.patch
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> * {{IF}} is not part of the sql standard; however given its special form its 
> simpler - and currently in Hive it also has vectorized support
> * people writing standard sql may write: {{CASE WHEN member=1 THEN attr+1 
> else attr+2 END}} which is essentially an if.
> The idea is to rewrite IFs to CASEs for the cbo; and recognize simple 
> "CASE"-s as IFs to get vectorization on them if possible



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Resolved] (HIVE-25715) Provide nightly builds

2022-02-17 Thread Zoltan Haindrich (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich resolved HIVE-25715.
-
Resolution: Fixed

merged into master. Thank you [~kkasa] for reviewing the changes!

> Provide nightly builds
> --
>
> Key: HIVE-25715
> URL: https://issues.apache.org/jira/browse/HIVE-25715
> Project: Hive
>  Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> provide nightly builds for the master branch



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25715) Provide nightly builds

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25715?focusedWorklogId=729086=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-729086
 ]

ASF GitHub Bot logged work on HIVE-25715:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 17:06
Start Date: 17/Feb/22 17:06
Worklog Time Spent: 10m 
  Work Description: kgyrtkirk merged pull request #3013:
URL: https://github.com/apache/hive/pull/3013


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 729086)
Time Spent: 20m  (was: 10m)

> Provide nightly builds
> --
>
> Key: HIVE-25715
> URL: https://issues.apache.org/jira/browse/HIVE-25715
> Project: Hive
>  Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> provide nightly builds for the master branch



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25947) Compactor job queue cannot be set per table via compactor.mapred.job.queue.name

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25947?focusedWorklogId=729079=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-729079
 ]

ASF GitHub Bot logged work on HIVE-25947:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 16:54
Start Date: 17/Feb/22 16:54
Worklog Time Spent: 10m 
  Work Description: zabetak commented on a change in pull request #3027:
URL: https://github.com/apache/hive/pull/3027#discussion_r809266279



##
File path: 
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorUtil.java
##
@@ -62,31 +75,29 @@ public static ExecutorService 
createExecutorWithThreadFactory(int parallelism, S
* @param conf global hive conf
* @param ci compaction info object
* @param table instance of table
-   * @return name of the queue, can be null
+   * @return name of the queue
*/
   static String getCompactorJobQueueName(HiveConf conf, CompactionInfo ci, 
Table table) {
 // Get queue name from the ci. This is passed through
 // ALTER TABLE table_name COMPACT 'major' WITH OVERWRITE 
TBLPROPERTIES('compactor.hive.compactor.job.queue'='some_queue')
+List> propertyGetters = new ArrayList<>(2);
 if (ci.properties != null) {
   StringableMap ciProperties = new StringableMap(ci.properties);
-  String queueName = ciProperties.get(COMPACTOR_PREFIX + 
MAPRED_QUEUE_NAME);
-  if (queueName != null && queueName.length() > 0) {
-return queueName;
-  }
+  propertyGetters.add(ciProperties::get);
 }
-
-// Get queue name from the table properties
-String queueName = table.getParameters().get(COMPACTOR_PREFIX + 
MAPRED_QUEUE_NAME);
-if (queueName != null && queueName.length() > 0) {
-  return queueName;
+if (table.getParameters() != null) {
+  propertyGetters.add(table.getParameters()::get);
 }
 
-// Get queue name from global hive conf
-queueName = conf.get(HiveConf.ConfVars.COMPACTOR_JOB_QUEUE.varname);
-if (queueName != null && queueName.length() > 0) {
-  return queueName;
+for (Function getter : propertyGetters) {

Review comment:
   The order in this piece of code is important thus I think the imperative 
style fits better here. I guess it boils down to personal preferences so unless 
you feel strongly about this I am inclined to keep it as is.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 729079)
Time Spent: 2h  (was: 1h 50m)

> Compactor job queue cannot be set per table via 
> compactor.mapred.job.queue.name
> ---
>
> Key: HIVE-25947
> URL: https://issues.apache.org/jira/browse/HIVE-25947
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Before HIVE-20723 it was possible to schedule the compaction for each table 
> on specific job queues by putting {{compactor.mapred.job.queue.name}} in the 
> table properties. 
> {code:sql}
> CREATE TABLE person (name STRING, age INT) STORED AS ORC TBLPROPERTIES(
> 'transactional'='true',
> 'compactor.mapred.job.queue.name'='root.user2);
> ALTER TABLE person COMPACT 'major' WITH OVERWRITE 
> TBLPROPERTIES('compactor.mapred.job.queue.name'='root.user2')
> {code}
> This is no longer possible (after HIVE-20723) and in order to achieve the 
> same effect someone needs to use the {{compactor.hive.compactor.job.queue}}.
> {code:sql}
> CREATE TABLE person (name STRING, age INT) STORED AS ORC TBLPROPERTIES(
> 'transactional'='true',
> 'compactor.hive.compactor.job.queue'='root.user2);
> ALTER TABLE person COMPACT 'major' WITH OVERWRITE 
> TBLPROPERTIES('compactor.hive.compactor.job.queue'='root.user2')
> {code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25947) Compactor job queue cannot be set per table via compactor.mapred.job.queue.name

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25947?focusedWorklogId=729073=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-729073
 ]

ASF GitHub Bot logged work on HIVE-25947:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 16:39
Start Date: 17/Feb/22 16:39
Worklog Time Spent: 10m 
  Work Description: deniskuzZ commented on a change in pull request #3027:
URL: https://github.com/apache/hive/pull/3027#discussion_r809249306



##
File path: 
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorUtil.java
##
@@ -62,31 +75,29 @@ public static ExecutorService 
createExecutorWithThreadFactory(int parallelism, S
* @param conf global hive conf
* @param ci compaction info object
* @param table instance of table
-   * @return name of the queue, can be null
+   * @return name of the queue
*/
   static String getCompactorJobQueueName(HiveConf conf, CompactionInfo ci, 
Table table) {
 // Get queue name from the ci. This is passed through
 // ALTER TABLE table_name COMPACT 'major' WITH OVERWRITE 
TBLPROPERTIES('compactor.hive.compactor.job.queue'='some_queue')
+List> propertyGetters = new ArrayList<>(2);
 if (ci.properties != null) {
   StringableMap ciProperties = new StringableMap(ci.properties);
-  String queueName = ciProperties.get(COMPACTOR_PREFIX + 
MAPRED_QUEUE_NAME);
-  if (queueName != null && queueName.length() > 0) {
-return queueName;
-  }
+  propertyGetters.add(ciProperties::get);
 }
-
-// Get queue name from the table properties
-String queueName = table.getParameters().get(COMPACTOR_PREFIX + 
MAPRED_QUEUE_NAME);
-if (queueName != null && queueName.length() > 0) {
-  return queueName;
+if (table.getParameters() != null) {
+  propertyGetters.add(table.getParameters()::get);
 }
 
-// Get queue name from global hive conf
-queueName = conf.get(HiveConf.ConfVars.COMPACTOR_JOB_QUEUE.varname);
-if (queueName != null && queueName.length() > 0) {
-  return queueName;
+for (Function getter : propertyGetters) {

Review comment:
   minor, however, it looks like a mix of imperative and declarative 
styles, I would prefer streaming API here:
   
   return propertyGetters.stream()
 .flatMap(getter -> QUEUE_PROPERTIES.stream()
   .filter(p -> !StringUtils.isEmpty(getter.apply(p
 .findFirst()
 .orElse(conf.getVar(HiveConf.ConfVars.COMPACTOR_JOB_QUEUE));
    




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 729073)
Time Spent: 1h 50m  (was: 1h 40m)

> Compactor job queue cannot be set per table via 
> compactor.mapred.job.queue.name
> ---
>
> Key: HIVE-25947
> URL: https://issues.apache.org/jira/browse/HIVE-25947
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Before HIVE-20723 it was possible to schedule the compaction for each table 
> on specific job queues by putting {{compactor.mapred.job.queue.name}} in the 
> table properties. 
> {code:sql}
> CREATE TABLE person (name STRING, age INT) STORED AS ORC TBLPROPERTIES(
> 'transactional'='true',
> 'compactor.mapred.job.queue.name'='root.user2);
> ALTER TABLE person COMPACT 'major' WITH OVERWRITE 
> TBLPROPERTIES('compactor.mapred.job.queue.name'='root.user2')
> {code}
> This is no longer possible (after HIVE-20723) and in order to achieve the 
> same effect someone needs to use the {{compactor.hive.compactor.job.queue}}.
> {code:sql}
> CREATE TABLE person (name STRING, age INT) STORED AS ORC TBLPROPERTIES(
> 'transactional'='true',
> 'compactor.hive.compactor.job.queue'='root.user2);
> ALTER TABLE person COMPACT 'major' WITH OVERWRITE 
> TBLPROPERTIES('compactor.hive.compactor.job.queue'='root.user2')
> {code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25952) Drop HiveRelMdPredicates::getPredicates(Project...) to use that of RelMdPredicates

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25952?focusedWorklogId=729070=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-729070
 ]

ASF GitHub Bot logged work on HIVE-25952:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 16:27
Start Date: 17/Feb/22 16:27
Worklog Time Spent: 10m 
  Work Description: asolimando closed pull request #3025:
URL: https://github.com/apache/hive/pull/3025


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 729070)
Time Spent: 1h  (was: 50m)

> Drop HiveRelMdPredicates::getPredicates(Project...) to use that of 
> RelMdPredicates
> --
>
> Key: HIVE-25952
> URL: https://issues.apache.org/jira/browse/HIVE-25952
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> There are some differences on this method between Hive and Calcite, the idea 
> of this ticket is to unify the two methods, and then drop the override in 
> HiveRelMdPredicates in favour of the method of RelMdPredicates.
> After applying HIVE-25966, the only difference is in the test for constant 
> expressions, which can be summarized as follows:
> ||Expression Type|Is Constant for Hive?||Is Constant for Calcite?||
> |InputRef|False|False|
> |Call|True if function is deterministic (arguments are not checked), false 
> otherwise|True if function is deterministic and all operands are constants, 
> false otherwise|
> |CorrelatedVariable|False|False|
> |LocalRef|False|False|
> |Over|False|False|
> |DymanicParameter|False|True|
> |RangeRef|False|False|
> |FieldAccess|False|Given expr.field, true if expr is constant, false 
> otherwise|



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25952) Drop HiveRelMdPredicates::getPredicates(Project...) to use that of RelMdPredicates

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25952?focusedWorklogId=729062=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-729062
 ]

ASF GitHub Bot logged work on HIVE-25952:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 16:18
Start Date: 17/Feb/22 16:18
Worklog Time Spent: 10m 
  Work Description: asolimando closed pull request #3023:
URL: https://github.com/apache/hive/pull/3023


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 729062)
Time Spent: 50m  (was: 40m)

> Drop HiveRelMdPredicates::getPredicates(Project...) to use that of 
> RelMdPredicates
> --
>
> Key: HIVE-25952
> URL: https://issues.apache.org/jira/browse/HIVE-25952
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> There are some differences on this method between Hive and Calcite, the idea 
> of this ticket is to unify the two methods, and then drop the override in 
> HiveRelMdPredicates in favour of the method of RelMdPredicates.
> After applying HIVE-25966, the only difference is in the test for constant 
> expressions, which can be summarized as follows:
> ||Expression Type|Is Constant for Hive?||Is Constant for Calcite?||
> |InputRef|False|False|
> |Call|True if function is deterministic (arguments are not checked), false 
> otherwise|True if function is deterministic and all operands are constants, 
> false otherwise|
> |CorrelatedVariable|False|False|
> |LocalRef|False|False|
> |Over|False|False|
> |DymanicParameter|False|True|
> |RangeRef|False|False|
> |FieldAccess|False|Given expr.field, true if expr is constant, false 
> otherwise|



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (HIVE-25952) Drop HiveRelMdPredicates::getPredicates(Project...) to use that of RelMdPredicates

2022-02-17 Thread Alessandro Solimando (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alessandro Solimando updated HIVE-25952:

Description: 
There are some differences on this method between Hive and Calcite, the idea of 
this ticket is to unify the two methods, and then drop the override in 
HiveRelMdPredicates in favour of the method of RelMdPredicates.

After applying HIVE-25966, the only difference is in the test for constant 
expressions, which can be summarized as follows:


||Expression Type|Is Constant for Hive?||Is Constant for Calcite?||
|InputRef|False|False|
|Call|True if function is deterministic (arguments are not checked), false 
otherwise|True if function is deterministic and all operands are constants, 
false otherwise|
|CorrelatedVariable|False|False|
|LocalRef|False|False|
|Over|False|False|
|DymanicParameter|False|True|
|RangeRef|False|False|
|FieldAccess|False|Given expr.field, true if expr is constant, false otherwise|

  was:
There are some differences on this method between Hive and Calcite, the idea of 
this ticket is to unify the two methods, and then drop the override in 
HiveRelMdPredicates in favour of the method of RelMdPredicates.

The only real


> Drop HiveRelMdPredicates::getPredicates(Project...) to use that of 
> RelMdPredicates
> --
>
> Key: HIVE-25952
> URL: https://issues.apache.org/jira/browse/HIVE-25952
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> There are some differences on this method between Hive and Calcite, the idea 
> of this ticket is to unify the two methods, and then drop the override in 
> HiveRelMdPredicates in favour of the method of RelMdPredicates.
> After applying HIVE-25966, the only difference is in the test for constant 
> expressions, which can be summarized as follows:
> ||Expression Type|Is Constant for Hive?||Is Constant for Calcite?||
> |InputRef|False|False|
> |Call|True if function is deterministic (arguments are not checked), false 
> otherwise|True if function is deterministic and all operands are constants, 
> false otherwise|
> |CorrelatedVariable|False|False|
> |LocalRef|False|False|
> |Over|False|False|
> |DymanicParameter|False|True|
> |RangeRef|False|False|
> |FieldAccess|False|Given expr.field, true if expr is constant, false 
> otherwise|



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (HIVE-25953) Drop HiveRelMdPredicates::getPredicates(Join...) to use that of RelMdPredicates

2022-02-17 Thread Alessandro Solimando (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alessandro Solimando updated HIVE-25953:

Summary: Drop HiveRelMdPredicates::getPredicates(Join...) to use that of 
RelMdPredicates  (was: Drop getPredicates() for Join in HiveRelMdPredicates to 
use that of RelMdPredicates)

> Drop HiveRelMdPredicates::getPredicates(Join...) to use that of 
> RelMdPredicates
> ---
>
> Key: HIVE-25953
> URL: https://issues.apache.org/jira/browse/HIVE-25953
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The goal of the ticket is to unify the two implementations and remove the 
> override in HiveRelMdPredicates.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (HIVE-25953) Drop getPredicates() for Join in HiveRelMdPredicates to use that of RelMdPredicates

2022-02-17 Thread Alessandro Solimando (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alessandro Solimando updated HIVE-25953:

Summary: Drop getPredicates() for Join in HiveRelMdPredicates to use that 
of RelMdPredicates  (was: Unify getPredicates() for Join between 
HiveRelMdPredicates and RelMdPredicates)

> Drop getPredicates() for Join in HiveRelMdPredicates to use that of 
> RelMdPredicates
> ---
>
> Key: HIVE-25953
> URL: https://issues.apache.org/jira/browse/HIVE-25953
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The goal of the ticket is to unify the two implementations and remove the 
> override in HiveRelMdPredicates.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (HIVE-25952) Drop HiveRelMdPredicates::getPredicates(Project...) to use that of RelMdPredicates

2022-02-17 Thread Alessandro Solimando (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alessandro Solimando updated HIVE-25952:

Description: 
There are some differences on this method between Hive and Calcite, the idea of 
this ticket is to unify the two methods, and then drop the override in 
HiveRelMdPredicates in favour of the method of RelMdPredicates.

The only real

  was:There are some differences on this method between Hive and Calcite, the 
idea of this ticket is to unify the two methods, and then drop the override in 
HiveRelMdPredicates in favour of the method of RelMdPredicates.


> Drop HiveRelMdPredicates::getPredicates(Project...) to use that of 
> RelMdPredicates
> --
>
> Key: HIVE-25952
> URL: https://issues.apache.org/jira/browse/HIVE-25952
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> There are some differences on this method between Hive and Calcite, the idea 
> of this ticket is to unify the two methods, and then drop the override in 
> HiveRelMdPredicates in favour of the method of RelMdPredicates.
> The only real



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (HIVE-25952) Drop HiveRelMdPredicates::getPredicates(Project...) to use that of RelMdPredicates

2022-02-17 Thread Alessandro Solimando (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alessandro Solimando updated HIVE-25952:

Summary: Drop HiveRelMdPredicates::getPredicates(Project...) to use that of 
RelMdPredicates  (was: Unify getPredicates() for Projection between 
HiveRelMdPredicates and RelMdPredicates)

> Drop HiveRelMdPredicates::getPredicates(Project...) to use that of 
> RelMdPredicates
> --
>
> Key: HIVE-25952
> URL: https://issues.apache.org/jira/browse/HIVE-25952
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> There are some differences on this method between Hive and Calcite, the idea 
> of this ticket is to unify the two methods, and then drop the override in 
> HiveRelMdPredicates in favour of the method of RelMdPredicates.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (HIVE-25953) Unify getPredicates() for Join between HiveRelMdPredicates and RelMdPredicates

2022-02-17 Thread Alessandro Solimando (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alessandro Solimando updated HIVE-25953:

Description: The goal of the ticket is to unify the two implementations and 
remove the override in HiveRelMdPredicates.  (was: The goal of the ticket is to 
unify the two implementations and remove the override in HiveRelMdPredicates. 

At the moment, the main blocker is that the Hive variant still relies in 
RexNode's comparison via its String digest, while Calcite does not need that 
anymore.)

> Unify getPredicates() for Join between HiveRelMdPredicates and RelMdPredicates
> --
>
> Key: HIVE-25953
> URL: https://issues.apache.org/jira/browse/HIVE-25953
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The goal of the ticket is to unify the two implementations and remove the 
> override in HiveRelMdPredicates.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25966) Align HiveRelMdPredicates getPredicates() for Project with RelMdPredicates

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25966?focusedWorklogId=729049=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-729049
 ]

ASF GitHub Bot logged work on HIVE-25966:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 16:03
Start Date: 17/Feb/22 16:03
Worklog Time Spent: 10m 
  Work Description: asolimando opened a new pull request #3039:
URL: https://github.com/apache/hive/pull/3039


   …h RelMdPredicates
   
   
   
   ### What changes were proposed in this pull request?
   
   
   Refactor the method to bring it close to what's in Calcite.
   
   ### Why are the changes needed?
   
   
   To benefit from the advances in the Calcite version and to facilitate 
dropping `HiveRelMdPredicates` at a later stage.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No
   
   ### How was this patch tested?
   
   
   Existing tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 729049)
Remaining Estimate: 0h
Time Spent: 10m

> Align HiveRelMdPredicates getPredicates() for Project with RelMdPredicates
> --
>
> Key: HIVE-25966
> URL: https://issues.apache.org/jira/browse/HIVE-25966
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The forked version of _HiveRelMdPredicates::getPredicates(Projection ...)_ 
> should be aligned with the current version of _RelMdPredicates_ in order to 
> facilitate dropping it in a later step (the difference at the moment is for 
> the visitor used to identify constants, which behaves slightly different 
> between the two).
> The ticket aims at refactoring the method to bring it close to the current 
> version in Calcite.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (HIVE-25966) Align HiveRelMdPredicates getPredicates() for Project with RelMdPredicates

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-25966:
--
Labels: pull-request-available  (was: )

> Align HiveRelMdPredicates getPredicates() for Project with RelMdPredicates
> --
>
> Key: HIVE-25966
> URL: https://issues.apache.org/jira/browse/HIVE-25966
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The forked version of _HiveRelMdPredicates::getPredicates(Projection ...)_ 
> should be aligned with the current version of _RelMdPredicates_ in order to 
> facilitate dropping it in a later step (the difference at the moment is for 
> the visitor used to identify constants, which behaves slightly different 
> between the two).
> The ticket aims at refactoring the method to bring it close to the current 
> version in Calcite.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work started] (HIVE-25966) Align HiveRelMdPredicates getPredicates() for Project with RelMdPredicates

2022-02-17 Thread Alessandro Solimando (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-25966 started by Alessandro Solimando.
---
> Align HiveRelMdPredicates getPredicates() for Project with RelMdPredicates
> --
>
> Key: HIVE-25966
> URL: https://issues.apache.org/jira/browse/HIVE-25966
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
>
> The forked version of _HiveRelMdPredicates::getPredicates(Projection ...)_ 
> should be aligned with the current version of _RelMdPredicates_ in order to 
> facilitate dropping it in a later step (the difference at the moment is for 
> the visitor used to identify constants, which behaves slightly different 
> between the two).
> The ticket aims at refactoring the method to bring it close to the current 
> version in Calcite.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Assigned] (HIVE-25966) Align HiveRelMdPredicates getPredicates() for Project with RelMdPredicates

2022-02-17 Thread Alessandro Solimando (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alessandro Solimando reassigned HIVE-25966:
---


> Align HiveRelMdPredicates getPredicates() for Project with RelMdPredicates
> --
>
> Key: HIVE-25966
> URL: https://issues.apache.org/jira/browse/HIVE-25966
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
>
> The forked version of _HiveRelMdPredicates::getPredicates(Projection ...)_ 
> should be aligned with the current version of _RelMdPredicates_ in order to 
> facilitate dropping it in a later step (the difference at the moment is for 
> the visitor used to identify constants, which behaves slightly different 
> between the two).
> The ticket aims at refactoring the method to bring it close to the current 
> version in Calcite.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25959) Expose Compaction Observability delta metrics using the JsonReporter

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25959?focusedWorklogId=729011=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-729011
 ]

ASF GitHub Bot logged work on HIVE-25959:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 15:19
Start Date: 17/Feb/22 15:19
Worklog Time Spent: 10m 
  Work Description: klcopp commented on a change in pull request #3032:
URL: https://github.com/apache/hive/pull/3032#discussion_r809164718



##
File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/Metrics.java
##
@@ -160,6 +159,47 @@ public Integer getValue() {
 }
   }
 
+  /**
+   * Get a Map that represents a multi-field metric,
+   * or create a new one if it does not already exist.
+   * @param name Name of map metric.  This should come from MetricConstants
+   * @return MapMetric .
+   */
+  public static MapMetrics getOrCreateMapMetrics(String name) {
+if (self == null) {
+  return dummyMapMetrics;
+}
+
+Map metrics = self.registry.getMetrics();
+Metric map = metrics.get(name);
+if (map instanceof MapMetrics) {
+  return (MapMetrics) map;
+}
+
+// Looks like it doesn't exist.  Lock so that two threads don't create it 
at once.
+synchronized (Metrics.class) {
+  // Recheck to make sure someone didn't create it while we waited.

Review comment:
   Okay, the "Recheck to make sure someone didn't create it while we 
waited" is still not the case




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 729011)
Time Spent: 2h  (was: 1h 50m)

> Expose Compaction Observability delta metrics using the JsonReporter
> 
>
> Key: HIVE-25959
> URL: https://issues.apache.org/jira/browse/HIVE-25959
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 4.0.0
>Reporter: Viktor Csomor
>Assignee: Viktor Csomor
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> As part of the HIVE-25897 the Delta metrics collection has been moved from 
> HiveServer2 to HMS. Hence the HMS does not have any jmx endpoint exposed by 
> default we'd like to expose these metrics through the JsonReporter



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (HIVE-25965) SQLDataException when obtaining partitions from HMS via direct SQL over Derby

2022-02-17 Thread Stamatis Zampetakis (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-25965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493968#comment-17493968
 ] 

Stamatis Zampetakis commented on HIVE-25965:


The derby-dump.tar.gz contains a Derby database where the problem can be 
reproduced outside of Hive. It suffices to uncompress the archive, connect to 
the database, and run the query in the description.

{code:java}
try (Connection c = 
DriverManager.getConnection("jdbc:derby:;databaseName=repro_derby_db")) {
try (PreparedStatement ps = c.prepareStatement(sql)) {
  try (ResultSet rs = ps.executeQuery()) {
while (rs.next()) {
  System.out.println(rs.getInt(1));
}
  }
}
  }
{code}


> SQLDataException when obtaining partitions from HMS via direct SQL over Derby
> -
>
> Key: HIVE-25965
> URL: https://issues.apache.org/jira/browse/HIVE-25965
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Stamatis Zampetakis
>Priority: Major
> Attachments: derby-dump.tar.gz
>
>
> In certain cases fetching the partition information from the metastore using 
> direct SQL fails with the stack trace below.
> {noformat}
> javax.jdo.JDODataStoreException: Error executing SQL query "select 
> "PARTITIONS"."PART_ID" from "PARTITIONS"  inner join "TBLS" on 
> "PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID" and "TBLS"."TBL_NAME" = ?   inner 
> join "DBS" on "TBLS"."DB_ID" = "DBS"."DB_ID"  and "DBS"."NAME" = ? inner 
> join "PARTITION_KEY_VALS" "FILTER0" on "FILTER0"."PART_ID" = 
> "PARTITIONS"."PART_ID" and "FILTER0"."INTEGER_IDX" = 0 where 
> "DBS"."CTLG_NAME" = ?  and (((case when "FILTER0"."PART_KEY_VAL" <> ? and 
> "TBLS"."TBL_NAME" = ? and "DBS"."NAME" = ? and "DBS"."CTLG_NAME" = ? and 
> "FILTER0"."PART_ID" = "PARTITIONS"."PART_ID" and "FILTER0"."INTEGER_IDX" = 0 
> then cast("FILTER0"."PART_KEY_VAL" as decimal(21,0)) else null end) = ?))".
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:542)
>  ~[datanucleus-api-jdo-5.2.4.jar:?]
>   at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:456) 
> ~[datanucleus-api-jdo-5.2.4.jar:?]
>   at org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:318) 
> ~[datanucleus-api-jdo-5.2.4.jar:?]
>   at 
> org.apache.hadoop.hive.metastore.QueryWrapper.executeWithArray(QueryWrapper.java:137)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray(MetastoreDirectSqlUtils.java:69)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:2156)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionIdsViaSqlFilter(MetaStoreDirectSql.java:894)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:663)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore$11.getSqlResult(ObjectStore.java:3962)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore$11.getSqlResult(ObjectStore.java:3953)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:4269)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByExprInternal(ObjectStore.java:3989)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.VerifyingObjectStore.getPartitionsByExpr(VerifyingObjectStore.java:80)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT-tests.jar:4.0.0-SNAPSHOT]
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:1.8.0_261]
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[?:1.8.0_261]
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_261]
>   at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_261]
>   at 
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) 
> ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at

[jira] [Updated] (HIVE-25965) SQLDataException when obtaining partitions from HMS via direct SQL over Derby

2022-02-17 Thread Stamatis Zampetakis (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stamatis Zampetakis updated HIVE-25965:
---
Attachment: derby-dump.tar.gz

> SQLDataException when obtaining partitions from HMS via direct SQL over Derby
> -
>
> Key: HIVE-25965
> URL: https://issues.apache.org/jira/browse/HIVE-25965
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Stamatis Zampetakis
>Priority: Major
> Attachments: derby-dump.tar.gz
>
>
> In certain cases fetching the partition information from the metastore using 
> direct SQL fails with the stack trace below.
> {noformat}
> javax.jdo.JDODataStoreException: Error executing SQL query "select 
> "PARTITIONS"."PART_ID" from "PARTITIONS"  inner join "TBLS" on 
> "PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID" and "TBLS"."TBL_NAME" = ?   inner 
> join "DBS" on "TBLS"."DB_ID" = "DBS"."DB_ID"  and "DBS"."NAME" = ? inner 
> join "PARTITION_KEY_VALS" "FILTER0" on "FILTER0"."PART_ID" = 
> "PARTITIONS"."PART_ID" and "FILTER0"."INTEGER_IDX" = 0 where 
> "DBS"."CTLG_NAME" = ?  and (((case when "FILTER0"."PART_KEY_VAL" <> ? and 
> "TBLS"."TBL_NAME" = ? and "DBS"."NAME" = ? and "DBS"."CTLG_NAME" = ? and 
> "FILTER0"."PART_ID" = "PARTITIONS"."PART_ID" and "FILTER0"."INTEGER_IDX" = 0 
> then cast("FILTER0"."PART_KEY_VAL" as decimal(21,0)) else null end) = ?))".
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:542)
>  ~[datanucleus-api-jdo-5.2.4.jar:?]
>   at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:456) 
> ~[datanucleus-api-jdo-5.2.4.jar:?]
>   at org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:318) 
> ~[datanucleus-api-jdo-5.2.4.jar:?]
>   at 
> org.apache.hadoop.hive.metastore.QueryWrapper.executeWithArray(QueryWrapper.java:137)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray(MetastoreDirectSqlUtils.java:69)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:2156)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionIdsViaSqlFilter(MetaStoreDirectSql.java:894)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:663)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore$11.getSqlResult(ObjectStore.java:3962)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore$11.getSqlResult(ObjectStore.java:3953)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:4269)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByExprInternal(ObjectStore.java:3989)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.VerifyingObjectStore.getPartitionsByExpr(VerifyingObjectStore.java:80)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT-tests.jar:4.0.0-SNAPSHOT]
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:1.8.0_261]
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[?:1.8.0_261]
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_261]
>   at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_261]
>   at 
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) 
> ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at com.sun.proxy.$Proxy60.getPartitionsByExpr(Unknown Source) ~[?:?]
>   at 
> org.apache.hadoop.hive.metastore.HMSHandler.get_partitions_spec_by_expr(HMSHandler.java:7346)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:1.8.0_261]
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[?:1.8.0_261]
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_261]
>   at

[jira] [Resolved] (HIVE-25915) Query based MINOR compaction fails with NPE if the data is loaded into the ACID table

2022-02-17 Thread Jira



 [ 
https://issues.apache.org/jira/browse/HIVE-25915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

László Végh resolved HIVE-25915.

Resolution: Fixed

> Query based MINOR compaction fails with NPE if the data is loaded into the 
> ACID table
> -
>
> Key: HIVE-25915
> URL: https://issues.apache.org/jira/browse/HIVE-25915
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: László Végh
>Assignee: László Végh
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Steps to reproduce:
>  #  Create a table with import:
> {{CREATE TABLE temp_acid(id string, value string) CLUSTERED BY(id) INTO 10 
> BUCKETS STORED AS ORC TBLPROPERTIES('transactional'='true');}}
>  # {{insert into temp_acid values 
> ('1','one'),('2','two'),('3','three'),('4','four'),('5','five'),('6','six'),('7','seven'),('8','eight'),('9','nine'),('10','ten'),('11','eleven'),('12','twelve'),('13','thirteen'),('14','fourteen'),('15','fifteen'),('16','sixteen'),('17','seventeen'),('18','eighteen'),('19','nineteen'),('20','twenty');}}
> {{export table temp_acid to '/tmp/temp_acid';}}
> {{{}i{}}}{{{}mport table imported from '/tmp/temp_acid';{}}}
>  # Do some inserts:
> {{insert into imported values ('21', 'value21'),('84', 'value84'),('66', 
> 'value66'),('54', 'value54');
> insert into imported values ('22', 'value22'),('34', 'value34'),('35', 
> 'value35');
> insert into imported values ('75', 'value75'),('99', 'value99');}}
>  # {{Run a minor compaction}}
> If the data is loaded or imported into the table they way it is described 
> above, the rows in the ORC file don't contain the ACID metadata. The 
> query-based MINOR compaction fails on this kind of table, because when the 
> FileSinkOperator tries to read out the bucket metadata from the rows it will 
> throw a NPE. But deleting and updating a table like this is possible. So 
> somehow the bucketId can be calculated for rows like this.
> The non-query based MINOR compaction works fine on a table like this.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25915) Query based MINOR compaction fails with NPE if the data is loaded into the ACID table

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25915?focusedWorklogId=728942=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-728942
 ]

ASF GitHub Bot logged work on HIVE-25915:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 13:51
Start Date: 17/Feb/22 13:51
Worklog Time Spent: 10m 
  Work Description: deniskuzZ merged pull request #3000:
URL: https://github.com/apache/hive/pull/3000


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 728942)
Time Spent: 2h 10m  (was: 2h)

> Query based MINOR compaction fails with NPE if the data is loaded into the 
> ACID table
> -
>
> Key: HIVE-25915
> URL: https://issues.apache.org/jira/browse/HIVE-25915
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: László Végh
>Assignee: László Végh
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Steps to reproduce:
>  #  Create a table with import:
> {{CREATE TABLE temp_acid(id string, value string) CLUSTERED BY(id) INTO 10 
> BUCKETS STORED AS ORC TBLPROPERTIES('transactional'='true');}}
>  # {{insert into temp_acid values 
> ('1','one'),('2','two'),('3','three'),('4','four'),('5','five'),('6','six'),('7','seven'),('8','eight'),('9','nine'),('10','ten'),('11','eleven'),('12','twelve'),('13','thirteen'),('14','fourteen'),('15','fifteen'),('16','sixteen'),('17','seventeen'),('18','eighteen'),('19','nineteen'),('20','twenty');}}
> {{export table temp_acid to '/tmp/temp_acid';}}
> {{{}i{}}}{{{}mport table imported from '/tmp/temp_acid';{}}}
>  # Do some inserts:
> {{insert into imported values ('21', 'value21'),('84', 'value84'),('66', 
> 'value66'),('54', 'value54');
> insert into imported values ('22', 'value22'),('34', 'value34'),('35', 
> 'value35');
> insert into imported values ('75', 'value75'),('99', 'value99');}}
>  # {{Run a minor compaction}}
> If the data is loaded or imported into the table they way it is described 
> above, the rows in the ORC file don't contain the ACID metadata. The 
> query-based MINOR compaction fails on this kind of table, because when the 
> FileSinkOperator tries to read out the bucket metadata from the rows it will 
> throw a NPE. But deleting and updating a table like this is possible. So 
> somehow the bucketId can be calculated for rows like this.
> The non-query based MINOR compaction works fine on a table like this.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Commented] (HIVE-25965) SQLDataException when obtaining partitions from HMS via direct SQL over Derby

2022-02-17 Thread Stamatis Zampetakis (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-25965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493944#comment-17493944
 ] 

Stamatis Zampetakis commented on HIVE-25965:


The failure is caused by the {{cast("FILTER0"."PART_KEY_VAL" AS decimal(21, 
0)}} expression since a value in PART_KEY_VAL cannot be cast to decimal since 
it is not a number. In principle, the CAST expression is guarded by the 
predicate in the CASE WHEN clase so we should never pass a value in the CAST 
that is not a decimal. For some reason, Derby wrongly re-orders the operations 
and ends up calling the CAST before ensuring that the CASE WHEN predicate holds 
resulting to the exception above.

> SQLDataException when obtaining partitions from HMS via direct SQL over Derby
> -
>
> Key: HIVE-25965
> URL: https://issues.apache.org/jira/browse/HIVE-25965
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Stamatis Zampetakis
>Priority: Major
>
> In certain cases fetching the partition information from the metastore using 
> direct SQL fails with the stack trace below.
> {noformat}
> javax.jdo.JDODataStoreException: Error executing SQL query "select 
> "PARTITIONS"."PART_ID" from "PARTITIONS"  inner join "TBLS" on 
> "PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID" and "TBLS"."TBL_NAME" = ?   inner 
> join "DBS" on "TBLS"."DB_ID" = "DBS"."DB_ID"  and "DBS"."NAME" = ? inner 
> join "PARTITION_KEY_VALS" "FILTER0" on "FILTER0"."PART_ID" = 
> "PARTITIONS"."PART_ID" and "FILTER0"."INTEGER_IDX" = 0 where 
> "DBS"."CTLG_NAME" = ?  and (((case when "FILTER0"."PART_KEY_VAL" <> ? and 
> "TBLS"."TBL_NAME" = ? and "DBS"."NAME" = ? and "DBS"."CTLG_NAME" = ? and 
> "FILTER0"."PART_ID" = "PARTITIONS"."PART_ID" and "FILTER0"."INTEGER_IDX" = 0 
> then cast("FILTER0"."PART_KEY_VAL" as decimal(21,0)) else null end) = ?))".
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:542)
>  ~[datanucleus-api-jdo-5.2.4.jar:?]
>   at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:456) 
> ~[datanucleus-api-jdo-5.2.4.jar:?]
>   at org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:318) 
> ~[datanucleus-api-jdo-5.2.4.jar:?]
>   at 
> org.apache.hadoop.hive.metastore.QueryWrapper.executeWithArray(QueryWrapper.java:137)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray(MetastoreDirectSqlUtils.java:69)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:2156)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionIdsViaSqlFilter(MetaStoreDirectSql.java:894)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:663)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore$11.getSqlResult(ObjectStore.java:3962)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore$11.getSqlResult(ObjectStore.java:3953)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:4269)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByExprInternal(ObjectStore.java:3989)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at 
> org.apache.hadoop.hive.metastore.VerifyingObjectStore.getPartitionsByExpr(VerifyingObjectStore.java:80)
>  ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT-tests.jar:4.0.0-SNAPSHOT]
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:1.8.0_261]
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[?:1.8.0_261]
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_261]
>   at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_261]
>   at 
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) 
> ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>   at com.sun.proxy.$Proxy60.getPartitionsByExpr(Unknown Source) ~[?:?]
>   at 
>

[jira] [Resolved] (HIVE-25766) java.util.NoSuchElementException in HiveFilterProjectTransposeRule if predicate has no InputRef

2022-02-17 Thread Alessandro Solimando (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alessandro Solimando resolved HIVE-25766.
-
Fix Version/s: 4.0.0
   Resolution: Fixed

Fixed via 
[4b7a948e45fd88372fef573be321cda40d189cc7|[https://github.com/apache/hive/commit/4b7a948e45fd88372fef573be321cda40d189cc7],]
 thanks [~kgyrtkirk] for the review!

> java.util.NoSuchElementException in HiveFilterProjectTransposeRule if 
> predicate has no InputRef
> ---
>
> Key: HIVE-25766
> URL: https://issues.apache.org/jira/browse/HIVE-25766
> Project: Hive
>  Issue Type: Bug
>  Components: CBO, Query Planning
>Affects Versions: 4.0.0
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> The issue can be reproduced with the following query:
> {code:java}
> create table test1 (s string);
> create table test2 (m string);
> EXPLAIN CBO SELECT c.m
> FROM (
>   SELECT substr(from_unixtime(unix_timestamp(), '-MM-dd'), 1, 1) as m
>   FROM test1
>   WHERE substr(from_unixtime(unix_timestamp(), '-MM-dd'), 1, 1) = '2') c
> JOIN test2 d ON c.m = d.m;
> {code}
> It fails with the following exception:
> {noformat}
>  java.util.NoSuchElementException
>     at java.util.HashMap$HashIterator.nextNode(HashMap.java:1447)
>     at java.util.HashMap$KeyIterator.next(HashMap.java:1469)
>     at 
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveFilterProjectTransposeRule$RedundancyChecker.check(HiveFilterProjectTransposeRule.java:348)
>     at 
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveFilterProjectTransposeRule$RedundancyChecker.visit(HiveFilterProjectTransposeRule.java:306)
>     at 
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveFilterProjectTransposeRule$RedundancyChecker.visit(HiveFilterProjectTransposeRule.java:303)
>     at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72)
>     at org.apache.calcite.rel.RelVisitor.visit(RelVisitor.java:44)
>     at 
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveFilterProjectTransposeRule$RedundancyChecker.visit(HiveFilterProjectTransposeRule.java:316)
>     at org.apache.calcite.rel.RelVisitor.go(RelVisitor.java:61)
>     at 
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveFilterProjectTransposeRule.isRedundantIsNotNull(HiveFilterProjectTransposeRule.java:276)
>     at 
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveFilterProjectTransposeRule.onMatch(HiveFilterProjectTransposeRule.java:191){noformat}
> The current implementation, while checking if the predicate to be transposed 
> is redundant or not, it expects at least one InputRef, but the predicate can 
> have none as in this case.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Resolved] (HIVE-25884) Improve rule description for rules defined as subclasses

2022-02-17 Thread Alessandro Solimando (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alessandro Solimando resolved HIVE-25884.
-
Fix Version/s: 4.0.0
   Resolution: Fixed

Fixed via 
[aafced6cd2b1bf31bd74c563680e02fd54cd1e01|[https://github.com/apache/hive/commit/aafced6cd2b1bf31bd74c563680e02fd54cd1e01],]
 thanks [~kgyrtkirk] and [~kkasa] for the review!

> Improve rule description for rules defined as subclasses
> 
>
> Key: HIVE-25884
> URL: https://issues.apache.org/jira/browse/HIVE-25884
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Consider the instances of _HivePointLookupOptimizerRule_ (for joins, filters 
> and projects). 
> They use the [default 
> constructor|https://github.com/apache/calcite/blob/0065d7c179b98698f018f83b0af0845a6698fc54/core/src/main/java/org/apache/calcite/plan/RelOptRule.java#L79]
>  for _RelOptRule_, which builds the rule description from the class name, and 
> in case of nested classes, it takes only the inner class name.
> In this case, the names do not refer to _HivePointLookupOptimizerRule_ and 
> are too generic (e.g.,_FilerCondition_), it's hard to link them back to the 
> rule they belong to without looking at the source code.
> This is particularly problematic now that we have more detailed logging for 
> CBO (see [HIVE-25816|https://issues.apache.org/jira/browse/HIVE-25816]), 
> where rule descriptions are printed.
> The aim of the PR is to improve the rule description by passing an explicit 
> string whenever the rule (class) name alone is not enough.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Resolved] (HIVE-25951) Re-use methods from RelMdPredicates in HiveRelMdPredicates

2022-02-17 Thread Alessandro Solimando (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alessandro Solimando resolved HIVE-25951.
-
Fix Version/s: 4.0.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

Fixed via 
[c835e4254452662b88376bb9480145500c6a0065|https://github.com/apache/hive/commit/c835e4254452662b88376bb9480145500c6a0065],
 thanks [~kkasa] for the review!

> Re-use methods from RelMdPredicates in HiveRelMdPredicates
> --
>
> Key: HIVE-25951
> URL: https://issues.apache.org/jira/browse/HIVE-25951
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> This ticket makes HiveRelMdPredicates extend RelMdPredicates, and remove the 
> duplicate methods which share the same behaviour, while overriding those for 
> which there is a difference.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (HIVE-25951) Re-use methods from RelMdPredicates in HiveRelMdPredicates

2022-02-17 Thread Alessandro Solimando (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alessandro Solimando updated HIVE-25951:

Hadoop Flags:   (was: Reviewed)

> Re-use methods from RelMdPredicates in HiveRelMdPredicates
> --
>
> Key: HIVE-25951
> URL: https://issues.apache.org/jira/browse/HIVE-25951
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> This ticket makes HiveRelMdPredicates extend RelMdPredicates, and remove the 
> duplicate methods which share the same behaviour, while overriding those for 
> which there is a difference.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25951) Re-use methods from RelMdPredicates in HiveRelMdPredicates

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25951?focusedWorklogId=728910=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-728910
 ]

ASF GitHub Bot logged work on HIVE-25951:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 12:45
Start Date: 17/Feb/22 12:45
Worklog Time Spent: 10m 
  Work Description: kasakrisz merged pull request #3016:
URL: https://github.com/apache/hive/pull/3016


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 728910)
Remaining Estimate: 0h
Time Spent: 10m

> Re-use methods from RelMdPredicates in HiveRelMdPredicates
> --
>
> Key: HIVE-25951
> URL: https://issues.apache.org/jira/browse/HIVE-25951
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> This ticket makes HiveRelMdPredicates extend RelMdPredicates, and remove the 
> duplicate methods which share the same behaviour, while overriding those for 
> which there is a difference.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25884) Improve rule description for rules defined as subclasses

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25884?focusedWorklogId=728907=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-728907
 ]

ASF GitHub Bot logged work on HIVE-25884:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 12:25
Start Date: 17/Feb/22 12:25
Worklog Time Spent: 10m 
  Work Description: kasakrisz merged pull request #2957:
URL: https://github.com/apache/hive/pull/2957


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 728907)
Time Spent: 50m  (was: 40m)

> Improve rule description for rules defined as subclasses
> 
>
> Key: HIVE-25884
> URL: https://issues.apache.org/jira/browse/HIVE-25884
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Affects Versions: 4.0.0
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Consider the instances of _HivePointLookupOptimizerRule_ (for joins, filters 
> and projects). 
> They use the [default 
> constructor|https://github.com/apache/calcite/blob/0065d7c179b98698f018f83b0af0845a6698fc54/core/src/main/java/org/apache/calcite/plan/RelOptRule.java#L79]
>  for _RelOptRule_, which builds the rule description from the class name, and 
> in case of nested classes, it takes only the inner class name.
> In this case, the names do not refer to _HivePointLookupOptimizerRule_ and 
> are too generic (e.g.,_FilerCondition_), it's hard to link them back to the 
> rule they belong to without looking at the source code.
> This is particularly problematic now that we have more detailed logging for 
> CBO (see [HIVE-25816|https://issues.apache.org/jira/browse/HIVE-25816]), 
> where rule descriptions are printed.
> The aim of the PR is to improve the rule description by passing an explicit 
> string whenever the rule (class) name alone is not enough.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25959) Expose Compaction Observability delta metrics using the JsonReporter

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25959?focusedWorklogId=728852=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-728852
 ]

ASF GitHub Bot logged work on HIVE-25959:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 11:02
Start Date: 17/Feb/22 11:02
Worklog Time Spent: 10m 
  Work Description: vcsomor commented on a change in pull request #3032:
URL: https://github.com/apache/hive/pull/3032#discussion_r808928144



##
File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/Metrics.java
##
@@ -160,6 +159,47 @@ public Integer getValue() {
 }
   }
 
+  /**
+   * Get a Map that represents a multi-field metric,
+   * or create a new one if it does not already exist.
+   * @param name Name of map metric.  This should come from MetricConstants
+   * @return MapMetric .
+   */
+  public static MapMetrics getOrCreateMapMetrics(String name) {
+if (self == null) {
+  return dummyMapMetrics;
+}
+
+Map metrics = self.registry.getMetrics();
+Metric map = metrics.get(name);
+if (map instanceof MapMetrics) {
+  return (MapMetrics) map;
+}
+
+// Looks like it doesn't exist.  Lock so that two threads don't create it 
at once.
+synchronized (Metrics.class) {
+  // Recheck to make sure someone didn't create it while we waited.

Review comment:
   ```java
   Object val = null;
   if (val instanceof String) {
 LOG.info("Not null safe");
   } else {
 LOG.info("Null safe");
   }
   ```
   output
   ```log
   [INFO ] OMITTED - Null safe
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 728852)
Time Spent: 1h 50m  (was: 1h 40m)

> Expose Compaction Observability delta metrics using the JsonReporter
> 
>
> Key: HIVE-25959
> URL: https://issues.apache.org/jira/browse/HIVE-25959
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 4.0.0
>Reporter: Viktor Csomor
>Assignee: Viktor Csomor
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> As part of the HIVE-25897 the Delta metrics collection has been moved from 
> HiveServer2 to HMS. Hence the HMS does not have any jmx endpoint exposed by 
> default we'd like to expose these metrics through the JsonReporter



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25959) Expose Compaction Observability delta metrics using the JsonReporter

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25959?focusedWorklogId=728850=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-728850
 ]

ASF GitHub Bot logged work on HIVE-25959:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 11:00
Start Date: 17/Feb/22 11:00
Worklog Time Spent: 10m 
  Work Description: vcsomor commented on a change in pull request #3032:
URL: https://github.com/apache/hive/pull/3032#discussion_r808925697



##
File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/JsonReporter.java
##
@@ -160,7 +158,7 @@ public void report(SortedMap sortedMap, 
SortedMap Expose Compaction Observability delta metrics using the JsonReporter
> 
>
> Key: HIVE-25959
> URL: https://issues.apache.org/jira/browse/HIVE-25959
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 4.0.0
>Reporter: Viktor Csomor
>Assignee: Viktor Csomor
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> As part of the HIVE-25897 the Delta metrics collection has been moved from 
> HiveServer2 to HMS. Hence the HMS does not have any jmx endpoint exposed by 
> default we'd like to expose these metrics through the JsonReporter



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25959) Expose Compaction Observability delta metrics using the JsonReporter

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25959?focusedWorklogId=728847=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-728847
 ]

ASF GitHub Bot logged work on HIVE-25959:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 10:59
Start Date: 17/Feb/22 10:59
Worklog Time Spent: 10m 
  Work Description: vcsomor commented on a change in pull request #3032:
URL: https://github.com/apache/hive/pull/3032#discussion_r808925697



##
File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/JsonReporter.java
##
@@ -160,7 +158,7 @@ public void report(SortedMap sortedMap, 
SortedMap Expose Compaction Observability delta metrics using the JsonReporter
> 
>
> Key: HIVE-25959
> URL: https://issues.apache.org/jira/browse/HIVE-25959
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 4.0.0
>Reporter: Viktor Csomor
>Assignee: Viktor Csomor
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> As part of the HIVE-25897 the Delta metrics collection has been moved from 
> HiveServer2 to HMS. Hence the HMS does not have any jmx endpoint exposed by 
> default we'd like to expose these metrics through the JsonReporter



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25959) Expose Compaction Observability delta metrics using the JsonReporter

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25959?focusedWorklogId=728848=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-728848
 ]

ASF GitHub Bot logged work on HIVE-25959:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 10:59
Start Date: 17/Feb/22 10:59
Worklog Time Spent: 10m 
  Work Description: vcsomor commented on a change in pull request #3032:
URL: https://github.com/apache/hive/pull/3032#discussion_r808925697



##
File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/JsonReporter.java
##
@@ -160,7 +158,7 @@ public void report(SortedMap sortedMap, 
SortedMap Expose Compaction Observability delta metrics using the JsonReporter
> 
>
> Key: HIVE-25959
> URL: https://issues.apache.org/jira/browse/HIVE-25959
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 4.0.0
>Reporter: Viktor Csomor
>Assignee: Viktor Csomor
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> As part of the HIVE-25897 the Delta metrics collection has been moved from 
> HiveServer2 to HMS. Hence the HMS does not have any jmx endpoint exposed by 
> default we'd like to expose these metrics through the JsonReporter



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25959) Expose Compaction Observability delta metrics using the JsonReporter

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25959?focusedWorklogId=728841=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-728841
 ]

ASF GitHub Bot logged work on HIVE-25959:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 10:53
Start Date: 17/Feb/22 10:53
Worklog Time Spent: 10m 
  Work Description: vcsomor commented on a change in pull request #3032:
URL: https://github.com/apache/hive/pull/3032#discussion_r808916354



##
File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/AcidMetricService.java
##
@@ -93,12 +93,9 @@
 public class AcidMetricService implements MetastoreTaskThread {
 
   private static final Logger LOG = 
LoggerFactory.getLogger(AcidMetricService.class);
-  public static final String OBJECT_NAME_PREFIX = 
"metrics:type=compaction,name=";

Review comment:
   The `OBJECT_NAME_PREFIX` was only required for JMX exposure. Not needed 
at all for the json format. All the metrics following this pattern (see the 
below json).
   ```json
   {
   "version" : "1.0",
   "gauges" : {
 "compaction_num_workers" : {
   "value" : 0
 },
 "compaction_num_working" : {
   "value" : 0
 }
   },
   "mbeans" : {
 "compaction_num_active_deltas" : {
   "db.table1/partition2" : 1000,
   "db.table1/partition1" : 100
 },
 "compaction_num_obsolete_deltas" : {
   "db.table3/partition" : 300
 },
 "compaction_num_small_deltas" : {
   "db.table2/partition" : 200
 }
   }
   ```
   
   As you can see the type and all the name not required at all because it i 
defined by the object_key (e.g.: compaction_num_active_deltas).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 728841)
Time Spent: 1h 10m  (was: 1h)

> Expose Compaction Observability delta metrics using the JsonReporter
> 
>
> Key: HIVE-25959
> URL: https://issues.apache.org/jira/browse/HIVE-25959
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 4.0.0
>Reporter: Viktor Csomor
>Assignee: Viktor Csomor
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> As part of the HIVE-25897 the Delta metrics collection has been moved from 
> HiveServer2 to HMS. Hence the HMS does not have any jmx endpoint exposed by 
> default we'd like to expose these metrics through the JsonReporter



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25959) Expose Compaction Observability delta metrics using the JsonReporter

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25959?focusedWorklogId=728838=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-728838
 ]

ASF GitHub Bot logged work on HIVE-25959:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 10:49
Start Date: 17/Feb/22 10:49
Worklog Time Spent: 10m 
  Work Description: vcsomor commented on a change in pull request #3032:
URL: https://github.com/apache/hive/pull/3032#discussion_r808916354



##
File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/AcidMetricService.java
##
@@ -93,12 +93,9 @@
 public class AcidMetricService implements MetastoreTaskThread {
 
   private static final Logger LOG = 
LoggerFactory.getLogger(AcidMetricService.class);
-  public static final String OBJECT_NAME_PREFIX = 
"metrics:type=compaction,name=";

Review comment:
   The `OBJECT_NAME_PREFIX` was only required for JMX exposure. Not needed 
at all for the json format. All the metrics following this pattern (see the 
below json).
   ```json
   {
   "version" : "1.0",
   "gauges" : {
 "compaction_num_workers" : {
   "value" : 0
 },
 "compaction_num_working" : {
   "value" : 0
 }
   },
   "mbeans" : {
 "compaction_num_active_deltas" : {
   "db.table1/partition2" : 1000,
   "db.table1/partition1" : 100
 },
 "compaction_num_obsolete_deltas" : {
   "db.table3/partition" : 300
 },
 "compaction_num_small_deltas" : {
   "db.table2/partition" : 200
 }
   }
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 728838)
Time Spent: 1h  (was: 50m)

> Expose Compaction Observability delta metrics using the JsonReporter
> 
>
> Key: HIVE-25959
> URL: https://issues.apache.org/jira/browse/HIVE-25959
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 4.0.0
>Reporter: Viktor Csomor
>Assignee: Viktor Csomor
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> As part of the HIVE-25897 the Delta metrics collection has been moved from 
> HiveServer2 to HMS. Hence the HMS does not have any jmx endpoint exposed by 
> default we'd like to expose these metrics through the JsonReporter



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25959) Expose Compaction Observability delta metrics using the JsonReporter

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25959?focusedWorklogId=728836=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-728836
 ]

ASF GitHub Bot logged work on HIVE-25959:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 10:44
Start Date: 17/Feb/22 10:44
Worklog Time Spent: 10m 
  Work Description: vcsomor commented on a change in pull request #3032:
URL: https://github.com/apache/hive/pull/3032#discussion_r808912719



##
File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/Metrics.java
##
@@ -160,6 +159,47 @@ public Integer getValue() {
 }
   }
 
+  /**
+   * Get a Map that represents a multi-field metric,
+   * or create a new one if it does not already exist.
+   * @param name Name of map metric.  This should come from MetricConstants
+   * @return MapMetric .
+   */
+  public static MapMetrics getOrCreateMapMetrics(String name) {
+if (self == null) {
+  return dummyMapMetrics;
+}
+
+Map metrics = self.registry.getMetrics();
+Metric map = metrics.get(name);
+if (map instanceof MapMetrics) {
+  return (MapMetrics) map;
+}
+
+// Looks like it doesn't exist.  Lock so that two threads don't create it 
at once.
+synchronized (Metrics.class) {
+  // Recheck to make sure someone didn't create it while we waited.

Review comment:
   The `instanceof` is nullsafe




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 728836)
Time Spent: 50m  (was: 40m)

> Expose Compaction Observability delta metrics using the JsonReporter
> 
>
> Key: HIVE-25959
> URL: https://issues.apache.org/jira/browse/HIVE-25959
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 4.0.0
>Reporter: Viktor Csomor
>Assignee: Viktor Csomor
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> As part of the HIVE-25897 the Delta metrics collection has been moved from 
> HiveServer2 to HMS. Hence the HMS does not have any jmx endpoint exposed by 
> default we'd like to expose these metrics through the JsonReporter



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25959) Expose Compaction Observability delta metrics using the JsonReporter

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25959?focusedWorklogId=728835=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-728835
 ]

ASF GitHub Bot logged work on HIVE-25959:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 10:44
Start Date: 17/Feb/22 10:44
Worklog Time Spent: 10m 
  Work Description: vcsomor commented on a change in pull request #3032:
URL: https://github.com/apache/hive/pull/3032#discussion_r808912041



##
File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/Metrics.java
##
@@ -160,6 +159,47 @@ public Integer getValue() {
 }
   }
 
+  /**
+   * Get a Map that represents a multi-field metric,
+   * or create a new one if it does not already exist.
+   * @param name Name of map metric.  This should come from MetricConstants
+   * @return MapMetric .
+   */
+  public static MapMetrics getOrCreateMapMetrics(String name) {
+if (self == null) {
+  return dummyMapMetrics;
+}
+
+Map metrics = self.registry.getMetrics();
+Metric map = metrics.get(name);
+if (map instanceof MapMetrics) {

Review comment:
   The `instanceof` is nullsafe. if `map` is `null` the the `if` becomes 
`false`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 728835)
Time Spent: 40m  (was: 0.5h)

> Expose Compaction Observability delta metrics using the JsonReporter
> 
>
> Key: HIVE-25959
> URL: https://issues.apache.org/jira/browse/HIVE-25959
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 4.0.0
>Reporter: Viktor Csomor
>Assignee: Viktor Csomor
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> As part of the HIVE-25897 the Delta metrics collection has been moved from 
> HiveServer2 to HMS. Hence the HMS does not have any jmx endpoint exposed by 
> default we'd like to expose these metrics through the JsonReporter



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25959) Expose Compaction Observability delta metrics using the JsonReporter

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25959?focusedWorklogId=728823=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-728823
 ]

ASF GitHub Bot logged work on HIVE-25959:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 10:26
Start Date: 17/Feb/22 10:26
Worklog Time Spent: 10m 
  Work Description: klcopp commented on a change in pull request #3032:
URL: https://github.com/apache/hive/pull/3032#discussion_r808886769



##
File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/Metrics.java
##
@@ -160,6 +159,47 @@ public Integer getValue() {
 }
   }
 
+  /**
+   * Get a Map that represents a multi-field metric,
+   * or create a new one if it does not already exist.
+   * @param name Name of map metric.  This should come from MetricConstants
+   * @return MapMetric .
+   */
+  public static MapMetrics getOrCreateMapMetrics(String name) {
+if (self == null) {
+  return dummyMapMetrics;
+}
+
+Map metrics = self.registry.getMetrics();
+Metric map = metrics.get(name);
+if (map instanceof MapMetrics) {

Review comment:
   Might need a null check here as well

##
File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/Metrics.java
##
@@ -160,6 +159,47 @@ public Integer getValue() {
 }
   }
 
+  /**
+   * Get a Map that represents a multi-field metric,
+   * or create a new one if it does not already exist.
+   * @param name Name of map metric.  This should come from MetricConstants
+   * @return MapMetric .
+   */
+  public static MapMetrics getOrCreateMapMetrics(String name) {
+if (self == null) {
+  return dummyMapMetrics;
+}
+
+Map metrics = self.registry.getMetrics();
+Metric map = metrics.get(name);
+if (map instanceof MapMetrics) {
+  return (MapMetrics) map;
+}
+
+// Looks like it doesn't exist.  Lock so that two threads don't create it 
at once.
+synchronized (Metrics.class) {
+  // Recheck to make sure someone didn't create it while we waited.

Review comment:
   The recheck is missing. To follow the pattern of the other metrics, this 
block should look like:
   
   ```
   metrics = self.registry.getMetrics();
 map = metrics.get(name);
 if (map != null && map instanceof MapMetrics) {
   return (MapMetrics) map;
 }
   // then register the metric and return it
   ```

##
File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/AcidMetricService.java
##
@@ -93,12 +93,9 @@
 public class AcidMetricService implements MetastoreTaskThread {
 
   private static final Logger LOG = 
LoggerFactory.getLogger(AcidMetricService.class);
-  public static final String OBJECT_NAME_PREFIX = 
"metrics:type=compaction,name=";

Review comment:
   Is this prefix omitted completely? If left out, it might introduce some 
backwards incompatibility

##
File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/JsonReporter.java
##
@@ -160,7 +158,7 @@ public void report(SortedMap sortedMap, 
SortedMap Expose Compaction Observability delta metrics using the JsonReporter
> 
>
> Key: HIVE-25959
> URL: https://issues.apache.org/jira/browse/HIVE-25959
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 4.0.0
>Reporter: Viktor Csomor
>Assignee: Viktor Csomor
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> As part of the HIVE-25897 the Delta metrics collection has been moved from 
> HiveServer2 to HMS. Hence the HMS does not have any jmx endpoint exposed by 
> default we'd like to expose these metrics through the JsonReporter



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Work logged] (HIVE-25964) Create iceberg table with ranger authorization failed with storage URI NullPointerException

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25964?focusedWorklogId=728812=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-728812
 ]

ASF GitHub Bot logged work on HIVE-25964:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 10:12
Start Date: 17/Feb/22 10:12
Worklog Time Spent: 10m 
  Work Description: zhangbutao commented on pull request #3038:
URL: https://github.com/apache/hive/pull/3038#issuecomment-1042783746


   cc @pvary @marton-bod Please take a look when you have spare time. thx


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 728812)
Time Spent: 20m  (was: 10m)

> Create iceberg table  with ranger authorization failed with storage URI 
> NullPointerException 
> -
>
> Key: HIVE-25964
> URL: https://issues.apache.org/jira/browse/HIVE-25964
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, HiveServer2
>Affects Versions: 4.0.0
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> set hive.security.authorization.enabled=true;
> set 
> hive.security.authorization.managerorg.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizerFactory;
> create table test_ice(id int) stored by iceberg;
> {code:java}
> 2022-02-17T16:32:58,304 ERROR [6bf2c99a-72eb-4608-9189-b64bd59df590 
> HiveServer2-Handler-Pool: Thread-83] command.CommandAuthorizerV2: Exception 
> occurred while getting the URI from storage handler: null
> java.lang.NullPointerException: null
>         at java.util.Hashtable.put(Hashtable.java:459) ~[?:1.8.0_112]
>         at java.util.Properties.setProperty(Properties.java:166) 
> ~[?:1.8.0_112]
>         at 
> org.apache.iceberg.mr.hive.IcebergTableUtil.getTable(IcebergTableUtil.java:87)
>  ~[hive-iceberg-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.iceberg.mr.hive.HiveIcebergStorageHandler.getURIForAuth(HiveIcebergStorageHandler.java:372)
>  ~[hive-iceberg-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizerV2.addHivePrivObject(CommandAuthorizerV2.java:197)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizerV2.getHivePrivObjects(CommandAuthorizerV2.java:142)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizerV2.doAuthorization(CommandAuthorizerV2.java:76)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizer.doAuthorization(CommandAuthorizer.java:58)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at org.apache.hadoop.hive.ql.Compiler.authorize(Compiler.java:426) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:112) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:501) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:453) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:417) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:411) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:121)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:204)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:267)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:281) 
> ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:545)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
>

[jira] [Updated] (HIVE-25964) Create iceberg table with ranger authorization failed with storage URI NullPointerException

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-25964:
--
Labels: pull-request-available  (was: )

> Create iceberg table  with ranger authorization failed with storage URI 
> NullPointerException 
> -
>
> Key: HIVE-25964
> URL: https://issues.apache.org/jira/browse/HIVE-25964
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, HiveServer2
>Affects Versions: 4.0.0
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> set hive.security.authorization.enabled=true;
> set 
> hive.security.authorization.managerorg.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizerFactory;
> create table test_ice(id int) stored by iceberg;
> {code:java}
> 2022-02-17T16:32:58,304 ERROR [6bf2c99a-72eb-4608-9189-b64bd59df590 
> HiveServer2-Handler-Pool: Thread-83] command.CommandAuthorizerV2: Exception 
> occurred while getting the URI from storage handler: null
> java.lang.NullPointerException: null
>         at java.util.Hashtable.put(Hashtable.java:459) ~[?:1.8.0_112]
>         at java.util.Properties.setProperty(Properties.java:166) 
> ~[?:1.8.0_112]
>         at 
> org.apache.iceberg.mr.hive.IcebergTableUtil.getTable(IcebergTableUtil.java:87)
>  ~[hive-iceberg-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.iceberg.mr.hive.HiveIcebergStorageHandler.getURIForAuth(HiveIcebergStorageHandler.java:372)
>  ~[hive-iceberg-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizerV2.addHivePrivObject(CommandAuthorizerV2.java:197)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizerV2.getHivePrivObjects(CommandAuthorizerV2.java:142)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizerV2.doAuthorization(CommandAuthorizerV2.java:76)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizer.doAuthorization(CommandAuthorizer.java:58)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at org.apache.hadoop.hive.ql.Compiler.authorize(Compiler.java:426) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:112) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:501) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:453) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:417) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:411) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:121)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:204)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:267)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:281) 
> ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:545)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:530)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:1.8.0_112]
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[?:1.8.0_112]
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_112]
>         at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
>         at 
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
>

[jira] [Work logged] (HIVE-25964) Create iceberg table with ranger authorization failed with storage URI NullPointerException

2022-02-17 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25964?focusedWorklogId=728780=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-728780
 ]

ASF GitHub Bot logged work on HIVE-25964:
-

Author: ASF GitHub Bot
Created on: 17/Feb/22 09:21
Start Date: 17/Feb/22 09:21
Worklog Time Spent: 10m 
  Work Description: zhangbutao opened a new pull request #3038:
URL: https://github.com/apache/hive/pull/3038


   …h storage URI NullPointerException
   
   
   
   ### What changes were proposed in this pull request?
   
   
   
   ### Why are the changes needed?
   
   
   
   ### Does this PR introduce _any_ user-facing change?
   
   
   
   ### How was this patch tested?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 728780)
Remaining Estimate: 0h
Time Spent: 10m

> Create iceberg table  with ranger authorization failed with storage URI 
> NullPointerException 
> -
>
> Key: HIVE-25964
> URL: https://issues.apache.org/jira/browse/HIVE-25964
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, HiveServer2
>Affects Versions: 4.0.0
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> set hive.security.authorization.enabled=true;
> set 
> hive.security.authorization.managerorg.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizerFactory;
> create table test_ice(id int) stored by iceberg;
> {code:java}
> 2022-02-17T16:32:58,304 ERROR [6bf2c99a-72eb-4608-9189-b64bd59df590 
> HiveServer2-Handler-Pool: Thread-83] command.CommandAuthorizerV2: Exception 
> occurred while getting the URI from storage handler: null
> java.lang.NullPointerException: null
>         at java.util.Hashtable.put(Hashtable.java:459) ~[?:1.8.0_112]
>         at java.util.Properties.setProperty(Properties.java:166) 
> ~[?:1.8.0_112]
>         at 
> org.apache.iceberg.mr.hive.IcebergTableUtil.getTable(IcebergTableUtil.java:87)
>  ~[hive-iceberg-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.iceberg.mr.hive.HiveIcebergStorageHandler.getURIForAuth(HiveIcebergStorageHandler.java:372)
>  ~[hive-iceberg-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizerV2.addHivePrivObject(CommandAuthorizerV2.java:197)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizerV2.getHivePrivObjects(CommandAuthorizerV2.java:142)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizerV2.doAuthorization(CommandAuthorizerV2.java:76)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizer.doAuthorization(CommandAuthorizer.java:58)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at org.apache.hadoop.hive.ql.Compiler.authorize(Compiler.java:426) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:112) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:501) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:453) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:417) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:411) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:121)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:204)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:267)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:281) 
> ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
>

[jira] [Commented] (HIVE-25964) Create iceberg table with ranger authorization failed with storage URI NullPointerException

2022-02-17 Thread zhangbutao (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-25964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493796#comment-17493796
 ] 

zhangbutao commented on HIVE-25964:
---

StorageDescriptor won't initialize table location in the compile stage. 
Therefore, it will cause null exception when invoke StorageDescriptor 
getLocation method in IcebergTableUtil.

 
{code:java}
  static Table getTable(Configuration configuration, 
org.apache.hadoop.hive.metastore.api.Table hmsTable) {
    Properties properties = new Properties();

  //An exception occurs on the following line
    properties.setProperty(Catalogs.NAME, 
TableIdentifier.of(hmsTable.getDbName(), hmsTable.getTableName()).toString());

    properties.setProperty(Catalogs.LOCATION, hmsTable.getSd().getLocation());
    if (hmsTable.getParameters().containsKey(InputFormatConfig.CATALOG_NAME)) {
      properties.setProperty(
          InputFormatConfig.CATALOG_NAME, 
hmsTable.getParameters().get(InputFormatConfig.CATALOG_NAME));
    }
    return getTable(configuration, properties);
  } {code}
 

 

> Create iceberg table  with ranger authorization failed with storage URI 
> NullPointerException 
> -
>
> Key: HIVE-25964
> URL: https://issues.apache.org/jira/browse/HIVE-25964
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, HiveServer2
>Affects Versions: 4.0.0
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
>
> set hive.security.authorization.enabled=true;
> set 
> hive.security.authorization.managerorg.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizerFactory;
> create table test_ice(id int) stored by iceberg;
> {code:java}
> 2022-02-17T16:32:58,304 ERROR [6bf2c99a-72eb-4608-9189-b64bd59df590 
> HiveServer2-Handler-Pool: Thread-83] command.CommandAuthorizerV2: Exception 
> occurred while getting the URI from storage handler: null
> java.lang.NullPointerException: null
>         at java.util.Hashtable.put(Hashtable.java:459) ~[?:1.8.0_112]
>         at java.util.Properties.setProperty(Properties.java:166) 
> ~[?:1.8.0_112]
>         at 
> org.apache.iceberg.mr.hive.IcebergTableUtil.getTable(IcebergTableUtil.java:87)
>  ~[hive-iceberg-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.iceberg.mr.hive.HiveIcebergStorageHandler.getURIForAuth(HiveIcebergStorageHandler.java:372)
>  ~[hive-iceberg-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizerV2.addHivePrivObject(CommandAuthorizerV2.java:197)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizerV2.getHivePrivObjects(CommandAuthorizerV2.java:142)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizerV2.doAuthorization(CommandAuthorizerV2.java:76)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizer.doAuthorization(CommandAuthorizer.java:58)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at org.apache.hadoop.hive.ql.Compiler.authorize(Compiler.java:426) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:112) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:501) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:453) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:417) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:411) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:121)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:204)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:267)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:281) 
> ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:545)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
>

[jira] [Comment Edited] (HIVE-25964) Create iceberg table with ranger authorization failed with storage URI NullPointerException

2022-02-17 Thread zhangbutao (Jira)



[ 
https://issues.apache.org/jira/browse/HIVE-25964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493796#comment-17493796
 ] 

zhangbutao edited comment on HIVE-25964 at 2/17/22, 9:07 AM:
-

StorageDescriptor won't initialize table location in the compile stage. 
Therefore, it will cause null exception when invoke StorageDescriptor 
getLocation method in IcebergTableUtil.

 
{code:java}
  static Table getTable(Configuration configuration, 
org.apache.hadoop.hive.metastore.api.Table hmsTable) {
    Properties properties = new Properties();

  //Null exception occurs on the following line
    properties.setProperty(Catalogs.NAME, 
TableIdentifier.of(hmsTable.getDbName(), hmsTable.getTableName()).toString());

    properties.setProperty(Catalogs.LOCATION, hmsTable.getSd().getLocation());
    if (hmsTable.getParameters().containsKey(InputFormatConfig.CATALOG_NAME)) {
      properties.setProperty(
          InputFormatConfig.CATALOG_NAME, 
hmsTable.getParameters().get(InputFormatConfig.CATALOG_NAME));
    }
    return getTable(configuration, properties);
  } {code}
 

 


was (Author: zhangbutao):
StorageDescriptor won't initialize table location in the compile stage. 
Therefore, it will cause null exception when invoke StorageDescriptor 
getLocation method in IcebergTableUtil.

 
{code:java}
  static Table getTable(Configuration configuration, 
org.apache.hadoop.hive.metastore.api.Table hmsTable) {
    Properties properties = new Properties();

  //An exception occurs on the following line
    properties.setProperty(Catalogs.NAME, 
TableIdentifier.of(hmsTable.getDbName(), hmsTable.getTableName()).toString());

    properties.setProperty(Catalogs.LOCATION, hmsTable.getSd().getLocation());
    if (hmsTable.getParameters().containsKey(InputFormatConfig.CATALOG_NAME)) {
      properties.setProperty(
          InputFormatConfig.CATALOG_NAME, 
hmsTable.getParameters().get(InputFormatConfig.CATALOG_NAME));
    }
    return getTable(configuration, properties);
  } {code}
 

 

> Create iceberg table  with ranger authorization failed with storage URI 
> NullPointerException 
> -
>
> Key: HIVE-25964
> URL: https://issues.apache.org/jira/browse/HIVE-25964
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, HiveServer2
>Affects Versions: 4.0.0
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
>
> set hive.security.authorization.enabled=true;
> set 
> hive.security.authorization.managerorg.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizerFactory;
> create table test_ice(id int) stored by iceberg;
> {code:java}
> 2022-02-17T16:32:58,304 ERROR [6bf2c99a-72eb-4608-9189-b64bd59df590 
> HiveServer2-Handler-Pool: Thread-83] command.CommandAuthorizerV2: Exception 
> occurred while getting the URI from storage handler: null
> java.lang.NullPointerException: null
>         at java.util.Hashtable.put(Hashtable.java:459) ~[?:1.8.0_112]
>         at java.util.Properties.setProperty(Properties.java:166) 
> ~[?:1.8.0_112]
>         at 
> org.apache.iceberg.mr.hive.IcebergTableUtil.getTable(IcebergTableUtil.java:87)
>  ~[hive-iceberg-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.iceberg.mr.hive.HiveIcebergStorageHandler.getURIForAuth(HiveIcebergStorageHandler.java:372)
>  ~[hive-iceberg-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizerV2.addHivePrivObject(CommandAuthorizerV2.java:197)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizerV2.getHivePrivObjects(CommandAuthorizerV2.java:142)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizerV2.doAuthorization(CommandAuthorizerV2.java:76)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizer.doAuthorization(CommandAuthorizer.java:58)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at org.apache.hadoop.hive.ql.Compiler.authorize(Compiler.java:426) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:112) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:501) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:453) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:417) 
>

[jira] [Updated] (HIVE-25964) Create iceberg table with ranger authorization failed with storage URI NullPointerException

2022-02-17 Thread zhangbutao (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhangbutao updated HIVE-25964:
--
Description: 
set hive.security.authorization.enabled=true;
set 
hive.security.authorization.managerorg.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizerFactory;

create table test_ice(id int) stored by iceberg;
{code:java}
2022-02-17T16:32:58,304 ERROR [6bf2c99a-72eb-4608-9189-b64bd59df590 
HiveServer2-Handler-Pool: Thread-83] command.CommandAuthorizerV2: Exception 
occurred while getting the URI from storage handler: null
java.lang.NullPointerException: null
        at java.util.Hashtable.put(Hashtable.java:459) ~[?:1.8.0_112]
        at java.util.Properties.setProperty(Properties.java:166) ~[?:1.8.0_112]
        at 
org.apache.iceberg.mr.hive.IcebergTableUtil.getTable(IcebergTableUtil.java:87) 
~[hive-iceberg-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at 
org.apache.iceberg.mr.hive.HiveIcebergStorageHandler.getURIForAuth(HiveIcebergStorageHandler.java:372)
 ~[hive-iceberg-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at 
org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizerV2.addHivePrivObject(CommandAuthorizerV2.java:197)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at 
org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizerV2.getHivePrivObjects(CommandAuthorizerV2.java:142)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at 
org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizerV2.doAuthorization(CommandAuthorizerV2.java:76)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at 
org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizer.doAuthorization(CommandAuthorizer.java:58)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.Compiler.authorize(Compiler.java:426) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:112) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:501) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:453) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:417) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:411) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at 
org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:121)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at 
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:204)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at 
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:267)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at 
org.apache.hive.service.cli.operation.Operation.run(Operation.java:281) 
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at 
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:545)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at 
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:530)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
~[?:1.8.0_112]
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
~[?:1.8.0_112]
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_112]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
        at 
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at 
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at 
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at java.security.AccessController.doPrivileged(Native Method) 
~[?:1.8.0_112]
        at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112]
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
 ~[hadoop-common-3.1.0-bc3.2.0.jar:?]
        at 
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at com.sun.proxy.$Proxy59.executeStatementAsync(Unknown Source) ~[?:?]
        at 
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:316)

[jira] [Assigned] (HIVE-25964) Create iceberg table with ranger authorization failed with storage URI NullPointerException

2022-02-17 Thread zhangbutao (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-25964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhangbutao reassigned HIVE-25964:
-

Assignee: zhangbutao

> Create iceberg table  with ranger authorization failed with storage URI 
> NullPointerException 
> -
>
> Key: HIVE-25964
> URL: https://issues.apache.org/jira/browse/HIVE-25964
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, HiveServer2
>Affects Versions: 4.0.0
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
>
> set hive.security.authorization.enabled=true;
> set 
> hive.security.authorization.managerorg.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizerFactory;
> create table test_ice(id int) stored by iceberg;
> {code:java}
> 2022-02-17T16:32:58,304 ERROR [6bf2c99a-72eb-4608-9189-b64bd59df590 
> HiveServer2-Handler-Pool: Thread-83] command.CommandAuthorizerV2: Exception 
> occurred while getting the URI from storage handler: null
> java.lang.NullPointerException: null
>         at java.util.Hashtable.put(Hashtable.java:459) ~[?:1.8.0_112]
>         at java.util.Properties.setProperty(Properties.java:166) 
> ~[?:1.8.0_112]
>         at 
> org.apache.iceberg.mr.hive.IcebergTableUtil.getTable(IcebergTableUtil.java:87)
>  ~[hive-iceberg-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.iceberg.mr.hive.HiveIcebergStorageHandler.getURIForAuth(HiveIcebergStorageHandler.java:372)
>  ~[hive-iceberg-handler-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizerV2.addHivePrivObject(CommandAuthorizerV2.java:197)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizerV2.getHivePrivObjects(CommandAuthorizerV2.java:142)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizerV2.doAuthorization(CommandAuthorizerV2.java:76)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.security.authorization.command.CommandAuthorizer.doAuthorization(CommandAuthorizer.java:58)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at org.apache.hadoop.hive.ql.Compiler.authorize(Compiler.java:426) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:112) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:501) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:453) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:417) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:411) 
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:121)
>  ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:204)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:267)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:281) 
> ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:545)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:530)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[?:1.8.0_112]
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[?:1.8.0_112]
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[?:1.8.0_112]
>         at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
>         at 
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
>  ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
>         at 
>

70 matches

Mail list logo