[jira] [Commented] (HIVE-23551) Acid: Update queries should treat dirCache as read-only in AcidUtils
[
https://issues.apache.org/jira/browse/HIVE-23551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120756#comment-17120756
]
Hive QA commented on HIVE-23551:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004459/HIVE-23551.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 17213 tests
executed
*Failed tests:*
{noformat}
TestStatsReplicationScenariosACID - did not produce a TEST-*.xml file (likely
timed out) (batchId=186)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_insert_overwrite_update]
(batchId=38)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=172)
org.apache.hive.service.server.TestHS2HttpServerPam.org.apache.hive.service.server.TestHS2HttpServerPam
(batchId=170)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22698/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22698/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22698/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004459 - PreCommit-HIVE-Build
> Acid: Update queries should treat dirCache as read-only in AcidUtils
>
>
> Key: HIVE-23551
> URL: https://issues.apache.org/jira/browse/HIVE-23551
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-23551.1.patch, HIVE-23551.2.patch,
> HIVE-23551.3.patch, HIVE-23551.4.patch, HIVE-23551.5.patch
>
>
> Update statements create delta folders at the end of the execution. When
> {{insert overwrite}} followed by {{update}} is executed, it does not get any
> open txns and ends up caching the {{base}} folder. However, the delta folder
> which gets created at the end of the statement never makes it to the cache.
> This creates wrong results.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23576) Getting partition of type int from metastore sometimes fail on cast error
[
https://issues.apache.org/jira/browse/HIVE-23576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lev Katzav updated HIVE-23576:
--
Description:
+given the following situation:+
there are 2 tables (in db "intpartitionbugtest"), each with a few rows:
# *test_table_int_1* partitioned by *y* of type *int*
# *test_table_string_1* partitioned by *x* of type *string*
here is the output of the following query on the metastore db:
{code:sql}
select
"PARTITIONS"."PART_ID",
"TBLS"."TBL_NAME",
"FILTER0"."PART_KEY_VAL",
"PART_NAME"
from
"PARTITIONS"
inner join "TBLS" on
"PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID"
inner join "DBS" on
"TBLS"."DB_ID" = "DBS"."DB_ID"
inner join "PARTITION_KEY_VALS" "FILTER0" on
"FILTER0"."PART_ID" = "PARTITIONS"."PART_ID"
{code}
!image-2020-05-29-14-16-29-356.png!
+the problem+
when running a hive query on the table *test_table_int_1* that filters on *y=1*
sometimes the following exception will happen on the metastore
{code:java}
javax.jdo.JDODataStoreException: Error executing SQL query "select
"PARTITIONS"."PART_ID" from "PARTITIONS" inner join "TBLS" on
"PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID" and "TBLS"."TBL_NAME" = ? inner
join "DBS" on "TBLS"."DB_ID" = "DBS"."DB_ID" and "DBS"."NAME" = ? inner
join "PARTITION_KEY_VALS" "FILTER0" on "FILTER0"."PART_ID" =
"PARTITIONS"."PART_ID" and "FILTER0"."INTEGER_IDX" = 0 where "DBS"."CTLG_NAME"
= ? and (((case when "FILTER0"."PART_KEY_VAL" <> ? then
cast("FILTER0"."PART_KEY_VAL" as decimal(21,0)) else null end) = ?))".
at
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
~[datanucleus-api-jdo-4.2.4.jar:?]
at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
~[datanucleus-api-jdo-4.2.4.jar:?]
at org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:267)
~[datanucleus-api-jdo-4.2.4.jar:?]
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:2003)
[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:593)
[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:481)
[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.metastore.ObjectStore$11.getSqlResult(ObjectStore.java:3853)
[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.metastore.ObjectStore$11.getSqlResult(ObjectStore.java:3843)
[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:3577)
[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByFilterInternal(ObjectStore.java:3861)
[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByFilter(ObjectStore.java:3516)
[hive-exec-3.1.2.jar:3.1.2]
at sun.reflect.GeneratedMethodAccessor70.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_112]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
[hive-exec-3.1.2.jar:3.1.2]
at com.sun.proxy.$Proxy28.getPartitionsByFilter(Unknown Source) [?:?]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions_by_filter(HiveMetaStore.java:5883)
[hive-exec-3.1.2.jar:3.1.2]
at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_112]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
[hive-exec-3.1.2.jar:3.1.2]
at com.sun.proxy.$Proxy30.get_partitions_by_filter(Unknown Source) [?:?]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_filter.getResult(ThriftHiveMetastore.java:16234)
[hive-exec-3.1.2.jar:3.1.2]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_filter.getResult(ThriftHiveMetastore.java:16218)
[hive-exec-3.1.2.jar:3.1.2]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
[hive-exec-3.1.2.jar:3.1.2]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
[hive-exec-3.1.2.jar:3.1.2]
at
[jira] [Updated] (HIVE-23551) Acid: Update queries should treat dirCache as read-only in AcidUtils
[
https://issues.apache.org/jira/browse/HIVE-23551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-23551:
Attachment: HIVE-23551.5.patch
> Acid: Update queries should treat dirCache as read-only in AcidUtils
>
>
> Key: HIVE-23551
> URL: https://issues.apache.org/jira/browse/HIVE-23551
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-23551.1.patch, HIVE-23551.2.patch,
> HIVE-23551.3.patch, HIVE-23551.4.patch, HIVE-23551.5.patch
>
>
> Update statements create delta folders at the end of the execution. When
> {{insert overwrite}} followed by {{update}} is executed, it does not get any
> open txns and ends up caching the {{base}} folder. However, the delta folder
> which gets created at the end of the statement never makes it to the cache.
> This creates wrong results.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23498) Disable HTTP Trace method on ThriftHttpCliService
[
https://issues.apache.org/jira/browse/HIVE-23498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120642#comment-17120642
]
Hive QA commented on HIVE-23498:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004453/HIVE-23498.04.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17219 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22697/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22697/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22697/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004453 - PreCommit-HIVE-Build
> Disable HTTP Trace method on ThriftHttpCliService
> -
>
> Key: HIVE-23498
> URL: https://issues.apache.org/jira/browse/HIVE-23498
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.2
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-23498.01.patch, HIVE-23498.01.patch,
> HIVE-23498.02.patch, HIVE-23498.03.patch, HIVE-23498.04.patch,
> HIVE-23498.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23498) Disable HTTP Trace method on ThriftHttpCliService
[ https://issues.apache.org/jira/browse/HIVE-23498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-23498: -- Attachment: HIVE-23498.04.patch Status: Patch Available (was: Open) > Disable HTTP Trace method on ThriftHttpCliService > - > > Key: HIVE-23498 > URL: https://issues.apache.org/jira/browse/HIVE-23498 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.1.2 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-23498.01.patch, HIVE-23498.01.patch, > HIVE-23498.02.patch, HIVE-23498.03.patch, HIVE-23498.04.patch, > HIVE-23498.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23498) Disable HTTP Trace method on ThriftHttpCliService
[ https://issues.apache.org/jira/browse/HIVE-23498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-23498: -- Status: Open (was: Patch Available) > Disable HTTP Trace method on ThriftHttpCliService > - > > Key: HIVE-23498 > URL: https://issues.apache.org/jira/browse/HIVE-23498 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.1.2 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-23498.01.patch, HIVE-23498.01.patch, > HIVE-23498.02.patch, HIVE-23498.03.patch, HIVE-23498.04.patch, > HIVE-23498.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23498) Disable HTTP Trace method on ThriftHttpCliService
[ https://issues.apache.org/jira/browse/HIVE-23498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120632#comment-17120632 ] Rajkumar Singh commented on HIVE-23498: --- Tests failure are unrelated but uploading the fresh patch for a clean run. > Disable HTTP Trace method on ThriftHttpCliService > - > > Key: HIVE-23498 > URL: https://issues.apache.org/jira/browse/HIVE-23498 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.1.2 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-23498.01.patch, HIVE-23498.01.patch, > HIVE-23498.02.patch, HIVE-23498.03.patch, HIVE-23498.04.patch, > HIVE-23498.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23498) Disable HTTP Trace method on ThriftHttpCliService
[
https://issues.apache.org/jira/browse/HIVE-23498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120438#comment-17120438
]
Hive QA commented on HIVE-23498:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004436/HIVE-23498.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17201 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=23)
[intersect_all.q,unionDistinct_1.q,orc_ppd_schema_evol_3a.q,macro_1.q,tez_union_dynamic_partition.q,vector_offset_limit.q,tez_union_dynamic_partition_2.q,whroot_external1.q,cte_2.q,rcfile_createas1.q,merge_test_dummy_operator.q,transform3.q,regexp_extract.q,cttl.q,load_binary_data.q]
TestStatsReplicationScenariosACID - did not produce a TEST-*.xml file (likely
timed out) (batchId=186)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22696/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22696/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22696/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004436 - PreCommit-HIVE-Build
> Disable HTTP Trace method on ThriftHttpCliService
> -
>
> Key: HIVE-23498
> URL: https://issues.apache.org/jira/browse/HIVE-23498
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.2
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-23498.01.patch, HIVE-23498.01.patch,
> HIVE-23498.02.patch, HIVE-23498.03.patch, HIVE-23498.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23584) Dont send default configs(HiveConf) to AM and tasks
[ https://issues.apache.org/jira/browse/HIVE-23584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mustafa Iman reassigned HIVE-23584: --- Assignee: Mustafa Iman > Dont send default configs(HiveConf) to AM and tasks > --- > > Key: HIVE-23584 > URL: https://issues.apache.org/jira/browse/HIVE-23584 > Project: Hive > Issue Type: Improvement >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Major > Attachments: hiveconf.wip.patch > > > About 80% of the configs left after HIVE-23175 are default settings coming > from HiveConf object. We can remove these from payload also. Only problem is > that TezTask relies on some of these configs when building dag. We can > explicitly add those settings that are needed in dag build phase in > HiveServer2. The rest is ok to be removed from payload. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23175) Skip serializing hadoop and tez config on HS side
[
https://issues.apache.org/jira/browse/HIVE-23175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120416#comment-17120416
]
Mustafa Iman commented on HIVE-23175:
-
[~ashutoshc] there is also one non-static method exposed in TEZ-4137,
InputInitializerContext#getVertexConfiguration:
{code:java}
this.conf = new Configuration(initializerContext.getVertexConfiguration());
{code}
I cannot get this patch working without the tez side.
> Skip serializing hadoop and tez config on HS side
> -
>
> Key: HIVE-23175
> URL: https://issues.apache.org/jira/browse/HIVE-23175
> Project: Hive
> Issue Type: Improvement
> Components: Tez
>Reporter: Mustafa Iman
>Assignee: Mustafa Iman
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23175.1.patch, HIVE-23175.2.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> HiveServer spends a lot of time serializing configuration objects. We can
> skip putting hadoop and tez config xml files in payload assuming that the
> configs are the same on both HS and Task side. This depends on Tez to load
> local xml configs when creating config objects
> [https://issues.apache.org/jira/browse/TEZ-4137]
> Ideally we should be able to skip hive-site.xml too. However, if we skip
> hive-site.xml at that stage, then we make wrong choices at tez dag build
> stage due to missing configs.
> In the ideal version of this, we should not be both looking up configs and
> putting new configs from and to the same config object at DAG and Vertex
> build phases. Instead we should be looking up from a HS2's HiveConf object
> and writing to a brand new JobConf for each vertex. That way we would not
> have any unnecessary item in the jobconf for any vertex. However Dag and
> Vertex build stages (TezTask#build) and a lot of other components called from
> there treat a single config object both the source of HS2 side config and the
> target JobConf that they are putting vertex level options into. It is very
> hard to separate these concerns now.
> With this patch, we are reducing the size of JobConf (per vertex) by ~65%. It
> should improve the transmit latency. However, most significant gains are at
> CPU time while compressing job configs as the config objects are much smaller
> now.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
