[jira] [Work logged] (HIVE-26446) HiveProtoLoggingHook fails to populate TablesWritten field for partitioned tables.
[
https://issues.apache.org/jira/browse/HIVE-26446?focusedWorklogId=797819=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-797819
]
ASF GitHub Bot logged work on HIVE-26446:
-
Author: ASF GitHub Bot
Created on: 04/Aug/22 00:28
Start Date: 04/Aug/22 00:28
Worklog Time Spent: 10m
Work Description: simhadri-g opened a new pull request, #3499:
URL: https://github.com/apache/hive/pull/3499
…d for partitioned tables
### What changes were proposed in this pull request?
entity.getType() returns the value as "PARTITION" for partitioned tables
instead of "TABLE" as a result the above check returns false and the
tablesWritten field in the hiveProtologger is left unpopulated for partitioned
tables.
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Unit Test and manual test.
Issue Time Tracking
---
Worklog Id: (was: 797819)
Remaining Estimate: 0h
Time Spent: 10m
> HiveProtoLoggingHook fails to populate TablesWritten field for partitioned
> tables.
> --
>
> Key: HIVE-26446
> URL: https://issues.apache.org/jira/browse/HIVE-26446
> Project: Hive
> Issue Type: Bug
>Reporter: Simhadri Govindappa
>Assignee: Simhadri Govindappa
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> From
> [here|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java#L490]
> :
> {code:java}
> if (entity.getType() == Entity.Type.TABLE) {code}
> entity.getType() returns the value as "PARTITION" for partitioned tables
> instead of "TABLE" as a result the above check returns false and the
> tablesWritten field in the hiveProtologger is left unpopulated for
> partitioned tables.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26446) HiveProtoLoggingHook fails to populate TablesWritten field for partitioned tables.
[
https://issues.apache.org/jira/browse/HIVE-26446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-26446:
--
Labels: pull-request-available (was: )
> HiveProtoLoggingHook fails to populate TablesWritten field for partitioned
> tables.
> --
>
> Key: HIVE-26446
> URL: https://issues.apache.org/jira/browse/HIVE-26446
> Project: Hive
> Issue Type: Bug
>Reporter: Simhadri Govindappa
>Assignee: Simhadri Govindappa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> From
> [here|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java#L490]
> :
> {code:java}
> if (entity.getType() == Entity.Type.TABLE) {code}
> entity.getType() returns the value as "PARTITION" for partitioned tables
> instead of "TABLE" as a result the above check returns false and the
> tablesWritten field in the hiveProtologger is left unpopulated for
> partitioned tables.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26079) Upgrade protobuf to 3.16.1
[ https://issues.apache.org/jira/browse/HIVE-26079?focusedWorklogId=797814=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-797814 ] ASF GitHub Bot logged work on HIVE-26079: - Author: ASF GitHub Bot Created on: 04/Aug/22 00:22 Start Date: 04/Aug/22 00:22 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #3309: HIVE-26079: Upgrade Protobuf to 3.21.0 URL: https://github.com/apache/hive/pull/3309 Issue Time Tracking --- Worklog Id: (was: 797814) Time Spent: 1h 40m (was: 1.5h) > Upgrade protobuf to 3.16.1 > -- > > Key: HIVE-26079 > URL: https://issues.apache.org/jira/browse/HIVE-26079 > Project: Hive > Issue Type: Task >Reporter: Ashish Sharma >Assignee: Ashish Sharma >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > Upgrade com.google.protobuf:protobuf-java from 2.5.0 to 3.16.1 to fix > CVE-2021-22569 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26444) Use tez.local.mode.without.network for unit tests
[
https://issues.apache.org/jira/browse/HIVE-26444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17574911#comment-17574911
]
László Bodor commented on HIVE-26444:
-
merged to master, thanks [~ayushtkn] and [~zabetak] for the review!
> Use tez.local.mode.without.network for unit tests
> -
>
> Key: HIVE-26444
> URL: https://issues.apache.org/jira/browse/HIVE-26444
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Since TEZ-4236 (in Tez 0.10.1), tez local mode can run without even starting
> an RPC server in the DAGAppMaster, which is in the same JVM as the client.
> Adapting tez.local.mode.without.network=true could make tez.local.mode=true
> unit tests more stable.
> here is an example where I had no idea why the dag app master connection was
> refused:
> {code}
> 2022-07-29T07:56:24,701 INFO [main_executor] ql.Driver: Executing
> command(queryId=jenkins_20220729075624_b3ba4c8a-82d5-4ebd-b4b0-218325a71b10):
> INSERT into table
> default.tmp_minor_compactor_testmmminorcompaction_1659106584519_result select
> `a`, `b` from default.tmp_minor_compactor_testmmminorcompaction_1659106584519
> 2022-07-29T07:56:24,823 INFO [ServiceThread:DAGClientRPCServer]
> client.DAGClientServer: Instantiated DAGClientRPCServer at
> internal-hive-flaky-check-88-xwmrs-v2h77-knnxx/10.106.3.19:22623
> 2022-07-29T07:56:24,823 INFO
> [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager]
> rm.TaskSchedulerManager: Creating TaskScheduler: Local TaskScheduler with
> clusterIdentifier=1659106584728
> 2022-07-29T07:56:24,825 INFO [DAGAppMaster Thread]
> HistoryEventHandler.criticalEvents: [HISTORY][DAG:N/A][Event:AM_STARTED]:
> appAttemptId=appattempt_1659106584728_0001_00, startTime=1659106584825
> 2022-07-29T07:56:24,825 INFO [DAGAppMaster Thread] app.DAGAppMaster: In
> Session mode. Waiting for DAG over RPC
> 2022-07-29T07:56:24,871 INFO [main_executor] client.LocalClient:
> DAGAppMaster state: IDLE
> 2022-07-29T07:56:24,871 INFO [main_executor] client.TezClient: The url to
> track the Tez Session: N/A
> ...
> 2022-07-29T07:56:46,384 INFO [main_executor] client.TezClient: Failed to
> retrieve AM Status via proxy
> com.google.protobuf.ServiceException: java.net.ConnectException: Call From
> internal-hive-flaky-check-88-xwmrs-v2h77-knnxx/10.106.3.19 to
> internal-hive-flaky-check-88-xwmrs-v2h77-knnxx:22623 failed on connection
> exception: java.net.ConnectException: Connection refused; For more details
> see: http://wiki.apache.org/hadoop/ConnectionRefused
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:247)
> ~[hadoop-common-3.1.1.7.2.15.0-147.jar:?]
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> ~[hadoop-common-3.1.1.7.2.15.0-147.jar:?]
> at com.sun.proxy.$Proxy50.getAMStatus(Unknown Source) ~[?:?]
> {code}
> instead of diving deep into an evil environment related bug, we can simply
> utilize TEZ-4236 in these cases too
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-26444) Use tez.local.mode.without.network for unit tests
[
https://issues.apache.org/jira/browse/HIVE-26444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor resolved HIVE-26444.
-
Resolution: Fixed
> Use tez.local.mode.without.network for unit tests
> -
>
> Key: HIVE-26444
> URL: https://issues.apache.org/jira/browse/HIVE-26444
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Since TEZ-4236 (in Tez 0.10.1), tez local mode can run without even starting
> an RPC server in the DAGAppMaster, which is in the same JVM as the client.
> Adapting tez.local.mode.without.network=true could make tez.local.mode=true
> unit tests more stable.
> here is an example where I had no idea why the dag app master connection was
> refused:
> {code}
> 2022-07-29T07:56:24,701 INFO [main_executor] ql.Driver: Executing
> command(queryId=jenkins_20220729075624_b3ba4c8a-82d5-4ebd-b4b0-218325a71b10):
> INSERT into table
> default.tmp_minor_compactor_testmmminorcompaction_1659106584519_result select
> `a`, `b` from default.tmp_minor_compactor_testmmminorcompaction_1659106584519
> 2022-07-29T07:56:24,823 INFO [ServiceThread:DAGClientRPCServer]
> client.DAGClientServer: Instantiated DAGClientRPCServer at
> internal-hive-flaky-check-88-xwmrs-v2h77-knnxx/10.106.3.19:22623
> 2022-07-29T07:56:24,823 INFO
> [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager]
> rm.TaskSchedulerManager: Creating TaskScheduler: Local TaskScheduler with
> clusterIdentifier=1659106584728
> 2022-07-29T07:56:24,825 INFO [DAGAppMaster Thread]
> HistoryEventHandler.criticalEvents: [HISTORY][DAG:N/A][Event:AM_STARTED]:
> appAttemptId=appattempt_1659106584728_0001_00, startTime=1659106584825
> 2022-07-29T07:56:24,825 INFO [DAGAppMaster Thread] app.DAGAppMaster: In
> Session mode. Waiting for DAG over RPC
> 2022-07-29T07:56:24,871 INFO [main_executor] client.LocalClient:
> DAGAppMaster state: IDLE
> 2022-07-29T07:56:24,871 INFO [main_executor] client.TezClient: The url to
> track the Tez Session: N/A
> ...
> 2022-07-29T07:56:46,384 INFO [main_executor] client.TezClient: Failed to
> retrieve AM Status via proxy
> com.google.protobuf.ServiceException: java.net.ConnectException: Call From
> internal-hive-flaky-check-88-xwmrs-v2h77-knnxx/10.106.3.19 to
> internal-hive-flaky-check-88-xwmrs-v2h77-knnxx:22623 failed on connection
> exception: java.net.ConnectException: Connection refused; For more details
> see: http://wiki.apache.org/hadoop/ConnectionRefused
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:247)
> ~[hadoop-common-3.1.1.7.2.15.0-147.jar:?]
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> ~[hadoop-common-3.1.1.7.2.15.0-147.jar:?]
> at com.sun.proxy.$Proxy50.getAMStatus(Unknown Source) ~[?:?]
> {code}
> instead of diving deep into an evil environment related bug, we can simply
> utilize TEZ-4236 in these cases too
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26444) Use tez.local.mode.without.network for unit tests
[
https://issues.apache.org/jira/browse/HIVE-26444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-26444:
Fix Version/s: 4.0.0-alpha-2
> Use tez.local.mode.without.network for unit tests
> -
>
> Key: HIVE-26444
> URL: https://issues.apache.org/jira/browse/HIVE-26444
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Since TEZ-4236 (in Tez 0.10.1), tez local mode can run without even starting
> an RPC server in the DAGAppMaster, which is in the same JVM as the client.
> Adapting tez.local.mode.without.network=true could make tez.local.mode=true
> unit tests more stable.
> here is an example where I had no idea why the dag app master connection was
> refused:
> {code}
> 2022-07-29T07:56:24,701 INFO [main_executor] ql.Driver: Executing
> command(queryId=jenkins_20220729075624_b3ba4c8a-82d5-4ebd-b4b0-218325a71b10):
> INSERT into table
> default.tmp_minor_compactor_testmmminorcompaction_1659106584519_result select
> `a`, `b` from default.tmp_minor_compactor_testmmminorcompaction_1659106584519
> 2022-07-29T07:56:24,823 INFO [ServiceThread:DAGClientRPCServer]
> client.DAGClientServer: Instantiated DAGClientRPCServer at
> internal-hive-flaky-check-88-xwmrs-v2h77-knnxx/10.106.3.19:22623
> 2022-07-29T07:56:24,823 INFO
> [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager]
> rm.TaskSchedulerManager: Creating TaskScheduler: Local TaskScheduler with
> clusterIdentifier=1659106584728
> 2022-07-29T07:56:24,825 INFO [DAGAppMaster Thread]
> HistoryEventHandler.criticalEvents: [HISTORY][DAG:N/A][Event:AM_STARTED]:
> appAttemptId=appattempt_1659106584728_0001_00, startTime=1659106584825
> 2022-07-29T07:56:24,825 INFO [DAGAppMaster Thread] app.DAGAppMaster: In
> Session mode. Waiting for DAG over RPC
> 2022-07-29T07:56:24,871 INFO [main_executor] client.LocalClient:
> DAGAppMaster state: IDLE
> 2022-07-29T07:56:24,871 INFO [main_executor] client.TezClient: The url to
> track the Tez Session: N/A
> ...
> 2022-07-29T07:56:46,384 INFO [main_executor] client.TezClient: Failed to
> retrieve AM Status via proxy
> com.google.protobuf.ServiceException: java.net.ConnectException: Call From
> internal-hive-flaky-check-88-xwmrs-v2h77-knnxx/10.106.3.19 to
> internal-hive-flaky-check-88-xwmrs-v2h77-knnxx:22623 failed on connection
> exception: java.net.ConnectException: Connection refused; For more details
> see: http://wiki.apache.org/hadoop/ConnectionRefused
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:247)
> ~[hadoop-common-3.1.1.7.2.15.0-147.jar:?]
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> ~[hadoop-common-3.1.1.7.2.15.0-147.jar:?]
> at com.sun.proxy.$Proxy50.getAMStatus(Unknown Source) ~[?:?]
> {code}
> instead of diving deep into an evil environment related bug, we can simply
> utilize TEZ-4236 in these cases too
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26444) Use tez.local.mode.without.network for unit tests
[
https://issues.apache.org/jira/browse/HIVE-26444?focusedWorklogId=797758=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-797758
]
ASF GitHub Bot logged work on HIVE-26444:
-
Author: ASF GitHub Bot
Created on: 03/Aug/22 20:02
Start Date: 03/Aug/22 20:02
Worklog Time Spent: 10m
Work Description: abstractdog merged PR #3491:
URL: https://github.com/apache/hive/pull/3491
Issue Time Tracking
---
Worklog Id: (was: 797758)
Time Spent: 1h (was: 50m)
> Use tez.local.mode.without.network for unit tests
> -
>
> Key: HIVE-26444
> URL: https://issues.apache.org/jira/browse/HIVE-26444
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Since TEZ-4236 (in Tez 0.10.1), tez local mode can run without even starting
> an RPC server in the DAGAppMaster, which is in the same JVM as the client.
> Adapting tez.local.mode.without.network=true could make tez.local.mode=true
> unit tests more stable.
> here is an example where I had no idea why the dag app master connection was
> refused:
> {code}
> 2022-07-29T07:56:24,701 INFO [main_executor] ql.Driver: Executing
> command(queryId=jenkins_20220729075624_b3ba4c8a-82d5-4ebd-b4b0-218325a71b10):
> INSERT into table
> default.tmp_minor_compactor_testmmminorcompaction_1659106584519_result select
> `a`, `b` from default.tmp_minor_compactor_testmmminorcompaction_1659106584519
> 2022-07-29T07:56:24,823 INFO [ServiceThread:DAGClientRPCServer]
> client.DAGClientServer: Instantiated DAGClientRPCServer at
> internal-hive-flaky-check-88-xwmrs-v2h77-knnxx/10.106.3.19:22623
> 2022-07-29T07:56:24,823 INFO
> [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager]
> rm.TaskSchedulerManager: Creating TaskScheduler: Local TaskScheduler with
> clusterIdentifier=1659106584728
> 2022-07-29T07:56:24,825 INFO [DAGAppMaster Thread]
> HistoryEventHandler.criticalEvents: [HISTORY][DAG:N/A][Event:AM_STARTED]:
> appAttemptId=appattempt_1659106584728_0001_00, startTime=1659106584825
> 2022-07-29T07:56:24,825 INFO [DAGAppMaster Thread] app.DAGAppMaster: In
> Session mode. Waiting for DAG over RPC
> 2022-07-29T07:56:24,871 INFO [main_executor] client.LocalClient:
> DAGAppMaster state: IDLE
> 2022-07-29T07:56:24,871 INFO [main_executor] client.TezClient: The url to
> track the Tez Session: N/A
> ...
> 2022-07-29T07:56:46,384 INFO [main_executor] client.TezClient: Failed to
> retrieve AM Status via proxy
> com.google.protobuf.ServiceException: java.net.ConnectException: Call From
> internal-hive-flaky-check-88-xwmrs-v2h77-knnxx/10.106.3.19 to
> internal-hive-flaky-check-88-xwmrs-v2h77-knnxx:22623 failed on connection
> exception: java.net.ConnectException: Connection refused; For more details
> see: http://wiki.apache.org/hadoop/ConnectionRefused
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:247)
> ~[hadoop-common-3.1.1.7.2.15.0-147.jar:?]
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> ~[hadoop-common-3.1.1.7.2.15.0-147.jar:?]
> at com.sun.proxy.$Proxy50.getAMStatus(Unknown Source) ~[?:?]
> {code}
> instead of diving deep into an evil environment related bug, we can simply
> utilize TEZ-4236 in these cases too
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-24483) Bump protobuf version to 3.21.1 (aarch64 support)
[ https://issues.apache.org/jira/browse/HIVE-24483?focusedWorklogId=797714=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-797714 ] ASF GitHub Bot logged work on HIVE-24483: - Author: ASF GitHub Bot Created on: 03/Aug/22 17:59 Start Date: 03/Aug/22 17:59 Worklog Time Spent: 10m Work Description: ayushtkn commented on PR #3498: URL: https://github.com/apache/hive/pull/3498#issuecomment-1204297458 yep, @Noremac201 I updated there and there were some explicit changes for Aarch because of lower version of protobuf, have updated that. Saw 2 Hbase related failure in the last build, may be upgrading Hbase version should solve them, will check post the next build result Issue Time Tracking --- Worklog Id: (was: 797714) Time Spent: 2h 50m (was: 2h 40m) > Bump protobuf version to 3.21.1 (aarch64 support) > - > > Key: HIVE-24483 > URL: https://issues.apache.org/jira/browse/HIVE-24483 > Project: Hive > Issue Type: Improvement >Reporter: Cameron Moberg >Assignee: Ayush Saxena >Priority: Minor > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > The following protoc version's used in hive is very old i.e. 2.5.0 > [https://repo.maven.apache.org/maven2/com/google/protobuf/protoc/] . The > v2.5.0 does not have aarch64 support. But the AArch64 support started from > v3.5.0 on-words in google's protobuf project release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26012) HMS APIs to be enhanced for metadata replication
[
https://issues.apache.org/jira/browse/HIVE-26012?focusedWorklogId=797707=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-797707
]
ASF GitHub Bot logged work on HIVE-26012:
-
Author: ASF GitHub Bot
Created on: 03/Aug/22 17:35
Start Date: 03/Aug/22 17:35
Worklog Time Spent: 10m
Work Description: DanielZhu58 commented on code in PR #3477:
URL: https://github.com/apache/hive/pull/3477#discussion_r936963722
##
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:
##
@@ -3717,4 +3683,103 @@ public void testDropDataConnectorIfNotExistsTrue()
throws Exception {
// No such data connector, ignore NoSuchObjectException
client.dropDataConnector("no_such_data_connector", true, false);
}
+
+ @Test
+ public void testIfFSWritesIsSkippedForDatabase() throws Throwable {
+// create a database, check if the directory is created or not
+// with true, the directory is not created
+// with false, the directory is created
+try {
+ // clear up any existing databases
+ silentDropDatabase(TEST_DB1_NAME);
+
+ String dbLocation =
+ MetastoreConf.getVar(conf, ConfVars.WAREHOUSE_EXTERNAL) +
"/testdb1.db";
+ String mgdLocation =
+ MetastoreConf.getVar(conf, ConfVars.WAREHOUSE) + "/testdb1.db";
+
+ CreateDatabaseRequest req = new CreateDatabaseRequest();
+ req.setSkipFSWrites(true);
+ Database db = new DatabaseBuilder()
+ .setName(TEST_DB1_NAME)
+ .setLocation(dbLocation)
+ .setManagedLocation(mgdLocation)
+ .build(conf);
+ req.setDatabase(db);
+ req.setDatabaseName(TEST_DB1_NAME);
+ client.createDatabase(req);
+
+ Path dbPath = new Path(db.getLocationUri());
+ FileSystem fs = FileSystem.get(new Path(dbLocation).toUri(), conf);
+ assertFalse("Database's file system directory is skipped", fs.exists(new
Path(dbLocation)));
+ fs = FileSystem.get(new Path(mgdLocation).toUri(), conf);
+ assertTrue("Database's managed location is not skipped", fs.exists(new
Path(mgdLocation)));
+} catch (Throwable e) {
+ System.err.println(StringUtils.stringifyException(e));
Review Comment:
Sure, I can change it to e.printStackTrace().
Issue Time Tracking
---
Worklog Id: (was: 797707)
Time Spent: 1h 50m (was: 1h 40m)
> HMS APIs to be enhanced for metadata replication
>
>
> Key: HIVE-26012
> URL: https://issues.apache.org/jira/browse/HIVE-26012
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 3.1.0
>Reporter: Naveen Gangam
>Assignee: Hongdan Zhu
>Priority: Major
> Labels: pull-request-available
> Attachments: HMS APIs to be enhanced for metadata replication.docx
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> HMS currently has APIs like these that automatically create/delete the
> directories on the associated DFS.
> [create/drop]_database
> [create/drop]_table*
> [add/append/drop]_partition*
> This is expected and should be this way when query processors use this APIs.
> However, when tools that replicate hive metadata use this APIs on the target
> cluster, creating these dirs on target side which cause the replication of
> DFS-snapshots to fail.
> So we if provide an option to bypass this creation of dirs, dfs replications
> will be smoother. In the future we will need to restrict users that can use
> these APIs. So we will have some sort of an authorization policy.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-24483) Bump protobuf version to 3.21.1 (aarch64 support)
[ https://issues.apache.org/jira/browse/HIVE-24483?focusedWorklogId=797663=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-797663 ] ASF GitHub Bot logged work on HIVE-24483: - Author: ASF GitHub Bot Created on: 03/Aug/22 15:02 Start Date: 03/Aug/22 15:02 Worklog Time Spent: 10m Work Description: Noremac201 commented on PR #3498: URL: https://github.com/apache/hive/pull/3498#issuecomment-1204068244 Won't standalone-metastore/pom.xml also need to be updated? Issue Time Tracking --- Worklog Id: (was: 797663) Time Spent: 2h 40m (was: 2.5h) > Bump protobuf version to 3.21.1 (aarch64 support) > - > > Key: HIVE-24483 > URL: https://issues.apache.org/jira/browse/HIVE-24483 > Project: Hive > Issue Type: Improvement >Reporter: Cameron Moberg >Assignee: Ayush Saxena >Priority: Minor > Labels: pull-request-available > Time Spent: 2h 40m > Remaining Estimate: 0h > > The following protoc version's used in hive is very old i.e. 2.5.0 > [https://repo.maven.apache.org/maven2/com/google/protobuf/protoc/] . The > v2.5.0 does not have aarch64 support. But the AArch64 support started from > v3.5.0 on-words in google's protobuf project release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-25996) Backport HIVE-25098
[ https://issues.apache.org/jira/browse/HIVE-25996?focusedWorklogId=797662=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-797662 ] ASF GitHub Bot logged work on HIVE-25996: - Author: ASF GitHub Bot Created on: 03/Aug/22 14:58 Start Date: 03/Aug/22 14:58 Worklog Time Spent: 10m Work Description: bgmarsh commented on PR #3066: URL: https://github.com/apache/hive/pull/3066#issuecomment-1204063017 Are there still plans to get this into the Hive 2.3 branch? Issue Time Tracking --- Worklog Id: (was: 797662) Time Spent: 1h (was: 50m) > Backport HIVE-25098 > --- > > Key: HIVE-25996 > URL: https://issues.apache.org/jira/browse/HIVE-25996 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.3.9 >Reporter: Yuming Wang >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26012) HMS APIs to be enhanced for metadata replication
[ https://issues.apache.org/jira/browse/HIVE-26012?focusedWorklogId=797592=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-797592 ] ASF GitHub Bot logged work on HIVE-26012: - Author: ASF GitHub Bot Created on: 03/Aug/22 12:41 Start Date: 03/Aug/22 12:41 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on code in PR #3477: URL: https://github.com/apache/hive/pull/3477#discussion_r936614566 ## standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java: ## @@ -41,15 +41,8 @@ import static org.mockito.Mockito.mock; import com.google.common.collect.Sets; -import org.apache.hadoop.hive.metastore.api.DataConnector; -import org.apache.hadoop.hive.metastore.api.DatabaseType; -import org.apache.hadoop.hive.metastore.api.GetPartitionsFilterSpec; -import org.apache.hadoop.hive.metastore.api.GetProjectionsSpec; -import org.apache.hadoop.hive.metastore.api.GetPartitionsRequest; -import org.apache.hadoop.hive.metastore.api.GetPartitionsResponse; -import org.apache.hadoop.hive.metastore.api.PartitionSpecWithSharedSD; -import org.apache.hadoop.hive.metastore.api.PartitionWithoutSD; -import org.apache.hadoop.hive.metastore.api.SourceTable; +import org.apache.hadoop.hive.metastore.api.*; Review Comment: please do not squash the imports with '*' ## standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java: ## @@ -41,15 +41,8 @@ import static org.mockito.Mockito.mock; import com.google.common.collect.Sets; -import org.apache.hadoop.hive.metastore.api.DataConnector; -import org.apache.hadoop.hive.metastore.api.DatabaseType; -import org.apache.hadoop.hive.metastore.api.GetPartitionsFilterSpec; -import org.apache.hadoop.hive.metastore.api.GetProjectionsSpec; -import org.apache.hadoop.hive.metastore.api.GetPartitionsRequest; -import org.apache.hadoop.hive.metastore.api.GetPartitionsResponse; -import org.apache.hadoop.hive.metastore.api.PartitionSpecWithSharedSD; -import org.apache.hadoop.hive.metastore.api.PartitionWithoutSD; -import org.apache.hadoop.hive.metastore.api.SourceTable; +import org.apache.hadoop.hive.metastore.api.*; Review Comment: nit: please do not squash the imports with '*' Issue Time Tracking --- Worklog Id: (was: 797592) Time Spent: 1h 40m (was: 1.5h) > HMS APIs to be enhanced for metadata replication > > > Key: HIVE-26012 > URL: https://issues.apache.org/jira/browse/HIVE-26012 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.1.0 >Reporter: Naveen Gangam >Assignee: Hongdan Zhu >Priority: Major > Labels: pull-request-available > Attachments: HMS APIs to be enhanced for metadata replication.docx > > Time Spent: 1h 40m > Remaining Estimate: 0h > > HMS currently has APIs like these that automatically create/delete the > directories on the associated DFS. > [create/drop]_database > [create/drop]_table* > [add/append/drop]_partition* > This is expected and should be this way when query processors use this APIs. > However, when tools that replicate hive metadata use this APIs on the target > cluster, creating these dirs on target side which cause the replication of > DFS-snapshots to fail. > So we if provide an option to bypass this creation of dirs, dfs replications > will be smoother. In the future we will need to restrict users that can use > these APIs. So we will have some sort of an authorization policy. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26012) HMS APIs to be enhanced for metadata replication
[
https://issues.apache.org/jira/browse/HIVE-26012?focusedWorklogId=797590=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-797590
]
ASF GitHub Bot logged work on HIVE-26012:
-
Author: ASF GitHub Bot
Created on: 03/Aug/22 12:33
Start Date: 03/Aug/22 12:33
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on code in PR #3477:
URL: https://github.com/apache/hive/pull/3477#discussion_r936607197
##
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:
##
@@ -3717,4 +3683,103 @@ public void testDropDataConnectorIfNotExistsTrue()
throws Exception {
// No such data connector, ignore NoSuchObjectException
client.dropDataConnector("no_such_data_connector", true, false);
}
+
+ @Test
+ public void testIfFSWritesIsSkippedForDatabase() throws Throwable {
+// create a database, check if the directory is created or not
+// with true, the directory is not created
+// with false, the directory is created
+try {
+ // clear up any existing databases
+ silentDropDatabase(TEST_DB1_NAME);
+
+ String dbLocation =
+ MetastoreConf.getVar(conf, ConfVars.WAREHOUSE_EXTERNAL) +
"/testdb1.db";
+ String mgdLocation =
+ MetastoreConf.getVar(conf, ConfVars.WAREHOUSE) + "/testdb1.db";
+
+ CreateDatabaseRequest req = new CreateDatabaseRequest();
+ req.setSkipFSWrites(true);
+ Database db = new DatabaseBuilder()
+ .setName(TEST_DB1_NAME)
+ .setLocation(dbLocation)
+ .setManagedLocation(mgdLocation)
+ .build(conf);
+ req.setDatabase(db);
+ req.setDatabaseName(TEST_DB1_NAME);
+ client.createDatabase(req);
+
+ Path dbPath = new Path(db.getLocationUri());
+ FileSystem fs = FileSystem.get(new Path(dbLocation).toUri(), conf);
+ assertFalse("Database's file system directory is skipped", fs.exists(new
Path(dbLocation)));
+ fs = FileSystem.get(new Path(mgdLocation).toUri(), conf);
+ assertTrue("Database's managed location is not skipped", fs.exists(new
Path(mgdLocation)));
+} catch (Throwable e) {
+ System.err.println(StringUtils.stringifyException(e));
+ System.err.println("testIfFSWritesIsSkippedForDatabase() failed.");
+ throw e;
+}
+ }
+
+ @Test
+ public void testIfFSWritesIsSkippedForTable() throws Throwable {
+// create a database, check if the directory is created or not
+// with true, the directory is not created
+// with false, the directory is created
+try {
+ // clear up any existing databases
+ silentDropDatabase(TEST_DB1_NAME);
+ String tableName1 = "test_table1";
+ String tableName2 = "test_table2";
+
+ String dbLocation =
+ MetastoreConf.getVar(conf, ConfVars.WAREHOUSE_EXTERNAL) +
"/testdb1.db";
+ String mgdLocation =
+ MetastoreConf.getVar(conf, ConfVars.WAREHOUSE) + "/testdb1.db";
+
+ Database db = new DatabaseBuilder()
Review Comment:
maybe should create `db` first in HMS
Issue Time Tracking
---
Worklog Id: (was: 797590)
Time Spent: 1.5h (was: 1h 20m)
> HMS APIs to be enhanced for metadata replication
>
>
> Key: HIVE-26012
> URL: https://issues.apache.org/jira/browse/HIVE-26012
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 3.1.0
>Reporter: Naveen Gangam
>Assignee: Hongdan Zhu
>Priority: Major
> Labels: pull-request-available
> Attachments: HMS APIs to be enhanced for metadata replication.docx
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> HMS currently has APIs like these that automatically create/delete the
> directories on the associated DFS.
> [create/drop]_database
> [create/drop]_table*
> [add/append/drop]_partition*
> This is expected and should be this way when query processors use this APIs.
> However, when tools that replicate hive metadata use this APIs on the target
> cluster, creating these dirs on target side which cause the replication of
> DFS-snapshots to fail.
> So we if provide an option to bypass this creation of dirs, dfs replications
> will be smoother. In the future we will need to restrict users that can use
> these APIs. So we will have some sort of an authorization policy.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26012) HMS APIs to be enhanced for metadata replication
[
https://issues.apache.org/jira/browse/HIVE-26012?focusedWorklogId=797588=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-797588
]
ASF GitHub Bot logged work on HIVE-26012:
-
Author: ASF GitHub Bot
Created on: 03/Aug/22 12:32
Start Date: 03/Aug/22 12:32
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on code in PR #3477:
URL: https://github.com/apache/hive/pull/3477#discussion_r936605923
##
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:
##
@@ -3717,4 +3683,103 @@ public void testDropDataConnectorIfNotExistsTrue()
throws Exception {
// No such data connector, ignore NoSuchObjectException
client.dropDataConnector("no_such_data_connector", true, false);
}
+
+ @Test
+ public void testIfFSWritesIsSkippedForDatabase() throws Throwable {
+// create a database, check if the directory is created or not
+// with true, the directory is not created
+// with false, the directory is created
+try {
+ // clear up any existing databases
+ silentDropDatabase(TEST_DB1_NAME);
+
+ String dbLocation =
+ MetastoreConf.getVar(conf, ConfVars.WAREHOUSE_EXTERNAL) +
"/testdb1.db";
+ String mgdLocation =
+ MetastoreConf.getVar(conf, ConfVars.WAREHOUSE) + "/testdb1.db";
+
+ CreateDatabaseRequest req = new CreateDatabaseRequest();
+ req.setSkipFSWrites(true);
+ Database db = new DatabaseBuilder()
+ .setName(TEST_DB1_NAME)
+ .setLocation(dbLocation)
+ .setManagedLocation(mgdLocation)
+ .build(conf);
+ req.setDatabase(db);
+ req.setDatabaseName(TEST_DB1_NAME);
+ client.createDatabase(req);
+
+ Path dbPath = new Path(db.getLocationUri());
+ FileSystem fs = FileSystem.get(new Path(dbLocation).toUri(), conf);
+ assertFalse("Database's file system directory is skipped", fs.exists(new
Path(dbLocation)));
+ fs = FileSystem.get(new Path(mgdLocation).toUri(), conf);
+ assertTrue("Database's managed location is not skipped", fs.exists(new
Path(mgdLocation)));
Review Comment:
should `mgdLocation` not be created as `skipFsWrites` is true?
Issue Time Tracking
---
Worklog Id: (was: 797588)
Time Spent: 1h 10m (was: 1h)
> HMS APIs to be enhanced for metadata replication
>
>
> Key: HIVE-26012
> URL: https://issues.apache.org/jira/browse/HIVE-26012
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 3.1.0
>Reporter: Naveen Gangam
>Assignee: Hongdan Zhu
>Priority: Major
> Labels: pull-request-available
> Attachments: HMS APIs to be enhanced for metadata replication.docx
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> HMS currently has APIs like these that automatically create/delete the
> directories on the associated DFS.
> [create/drop]_database
> [create/drop]_table*
> [add/append/drop]_partition*
> This is expected and should be this way when query processors use this APIs.
> However, when tools that replicate hive metadata use this APIs on the target
> cluster, creating these dirs on target side which cause the replication of
> DFS-snapshots to fail.
> So we if provide an option to bypass this creation of dirs, dfs replications
> will be smoother. In the future we will need to restrict users that can use
> these APIs. So we will have some sort of an authorization policy.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26012) HMS APIs to be enhanced for metadata replication
[
https://issues.apache.org/jira/browse/HIVE-26012?focusedWorklogId=797589=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-797589
]
ASF GitHub Bot logged work on HIVE-26012:
-
Author: ASF GitHub Bot
Created on: 03/Aug/22 12:32
Start Date: 03/Aug/22 12:32
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on code in PR #3477:
URL: https://github.com/apache/hive/pull/3477#discussion_r936606173
##
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:
##
@@ -3717,4 +3683,103 @@ public void testDropDataConnectorIfNotExistsTrue()
throws Exception {
// No such data connector, ignore NoSuchObjectException
client.dropDataConnector("no_such_data_connector", true, false);
}
+
+ @Test
+ public void testIfFSWritesIsSkippedForDatabase() throws Throwable {
+// create a database, check if the directory is created or not
+// with true, the directory is not created
+// with false, the directory is created
+try {
+ // clear up any existing databases
+ silentDropDatabase(TEST_DB1_NAME);
+
+ String dbLocation =
+ MetastoreConf.getVar(conf, ConfVars.WAREHOUSE_EXTERNAL) +
"/testdb1.db";
+ String mgdLocation =
+ MetastoreConf.getVar(conf, ConfVars.WAREHOUSE) + "/testdb1.db";
+
+ CreateDatabaseRequest req = new CreateDatabaseRequest();
+ req.setSkipFSWrites(true);
+ Database db = new DatabaseBuilder()
+ .setName(TEST_DB1_NAME)
+ .setLocation(dbLocation)
+ .setManagedLocation(mgdLocation)
+ .build(conf);
+ req.setDatabase(db);
+ req.setDatabaseName(TEST_DB1_NAME);
+ client.createDatabase(req);
+
+ Path dbPath = new Path(db.getLocationUri());
+ FileSystem fs = FileSystem.get(new Path(dbLocation).toUri(), conf);
+ assertFalse("Database's file system directory is skipped", fs.exists(new
Path(dbLocation)));
+ fs = FileSystem.get(new Path(mgdLocation).toUri(), conf);
+ assertTrue("Database's managed location is not skipped", fs.exists(new
Path(mgdLocation)));
+} catch (Throwable e) {
+ System.err.println(StringUtils.stringifyException(e));
Review Comment:
how about e.printStackTrace()?
Issue Time Tracking
---
Worklog Id: (was: 797589)
Time Spent: 1h 20m (was: 1h 10m)
> HMS APIs to be enhanced for metadata replication
>
>
> Key: HIVE-26012
> URL: https://issues.apache.org/jira/browse/HIVE-26012
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 3.1.0
>Reporter: Naveen Gangam
>Assignee: Hongdan Zhu
>Priority: Major
> Labels: pull-request-available
> Attachments: HMS APIs to be enhanced for metadata replication.docx
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> HMS currently has APIs like these that automatically create/delete the
> directories on the associated DFS.
> [create/drop]_database
> [create/drop]_table*
> [add/append/drop]_partition*
> This is expected and should be this way when query processors use this APIs.
> However, when tools that replicate hive metadata use this APIs on the target
> cluster, creating these dirs on target side which cause the replication of
> DFS-snapshots to fail.
> So we if provide an option to bypass this creation of dirs, dfs replications
> will be smoother. In the future we will need to restrict users that can use
> these APIs. So we will have some sort of an authorization policy.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26445) Use tez.local.mode.without.network for qtests
[
https://issues.apache.org/jira/browse/HIVE-26445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-26445:
Summary: Use tez.local.mode.without.network for qtests (was: Use
tez.local.mode.without.network for unit tests - qtests)
> Use tez.local.mode.without.network for qtests
> -
>
> Key: HIVE-26445
> URL: https://issues.apache.org/jira/browse/HIVE-26445
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
>
> looks like in case of iceberg, the local dagclient behaves weird:
> {code}
> 2022-08-02T06:54:36,669 ERROR [2f953972-7675-4594-8d6b-d1c295c056a5
> Time-limited test] tez.TezTask: Failed to execute tez graph.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.ql.exec.tez.TezTask.collectCommitInformation(TezTask.java:367)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:279)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:354)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:327)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:244)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:105)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:355)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:205)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:154)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:149)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> {code}
> it's thrown from
> https://github.com/apache/hive/blob/e0f2d287c562423dc2632910aae4f1cd8bcd4b4d/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java#L367
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26445) Use tez.local.mode.without.network for unit tests - qtests
[
https://issues.apache.org/jira/browse/HIVE-26445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-26445:
Summary: Use tez.local.mode.without.network for unit tests - qtests (was:
Use tez.local.mode.without.network for unit tests - iceberg tests)
> Use tez.local.mode.without.network for unit tests - qtests
> --
>
> Key: HIVE-26445
> URL: https://issues.apache.org/jira/browse/HIVE-26445
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Priority: Major
>
> looks like in case of iceberg, the local dagclient behaves weird:
> {code}
> 2022-08-02T06:54:36,669 ERROR [2f953972-7675-4594-8d6b-d1c295c056a5
> Time-limited test] tez.TezTask: Failed to execute tez graph.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.ql.exec.tez.TezTask.collectCommitInformation(TezTask.java:367)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:279)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:354)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:327)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:244)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:105)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:355)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:205)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:154)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:149)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> {code}
> it's thrown from
> https://github.com/apache/hive/blob/e0f2d287c562423dc2632910aae4f1cd8bcd4b4d/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java#L367
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26445) Use tez.local.mode.without.network for unit tests - qtests
[
https://issues.apache.org/jira/browse/HIVE-26445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor reassigned HIVE-26445:
---
Assignee: László Bodor
> Use tez.local.mode.without.network for unit tests - qtests
> --
>
> Key: HIVE-26445
> URL: https://issues.apache.org/jira/browse/HIVE-26445
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
>
> looks like in case of iceberg, the local dagclient behaves weird:
> {code}
> 2022-08-02T06:54:36,669 ERROR [2f953972-7675-4594-8d6b-d1c295c056a5
> Time-limited test] tez.TezTask: Failed to execute tez graph.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.ql.exec.tez.TezTask.collectCommitInformation(TezTask.java:367)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:279)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:354)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:327)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:244)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:105)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:355)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:205)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:154)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:149)
> [hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> {code}
> it's thrown from
> https://github.com/apache/hive/blob/e0f2d287c562423dc2632910aae4f1cd8bcd4b4d/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java#L367
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-24484) Upgrade Hadoop to 3.3.1 And Tez to 0.10.2
[ https://issues.apache.org/jira/browse/HIVE-24484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17574635#comment-17574635 ] Steve Loughran commented on HIVE-24484: --- nice! > Upgrade Hadoop to 3.3.1 And Tez to 0.10.2 > -- > > Key: HIVE-24484 > URL: https://issues.apache.org/jira/browse/HIVE-24484 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 15.05h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26444) Use tez.local.mode.without.network for unit tests
[
https://issues.apache.org/jira/browse/HIVE-26444?focusedWorklogId=797544=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-797544
]
ASF GitHub Bot logged work on HIVE-26444:
-
Author: ASF GitHub Bot
Created on: 03/Aug/22 09:16
Start Date: 03/Aug/22 09:16
Worklog Time Spent: 10m
Work Description: zabetak commented on code in PR #3491:
URL: https://github.com/apache/hive/pull/3491#discussion_r936432254
##
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveShell.java:
##
@@ -201,6 +201,8 @@ private HiveConf initializeConf() {
// Tez configuration
hiveConf.setBoolean("tez.local.mode", true);
+// TODO: enable below option once HIVE-26445 is investigated
+// hiveConf.setBoolean("tez.local.mode.without.network", true);
Review Comment:
Sounds good.
Issue Time Tracking
---
Worklog Id: (was: 797544)
Time Spent: 50m (was: 40m)
> Use tez.local.mode.without.network for unit tests
> -
>
> Key: HIVE-26444
> URL: https://issues.apache.org/jira/browse/HIVE-26444
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Since TEZ-4236 (in Tez 0.10.1), tez local mode can run without even starting
> an RPC server in the DAGAppMaster, which is in the same JVM as the client.
> Adapting tez.local.mode.without.network=true could make tez.local.mode=true
> unit tests more stable.
> here is an example where I had no idea why the dag app master connection was
> refused:
> {code}
> 2022-07-29T07:56:24,701 INFO [main_executor] ql.Driver: Executing
> command(queryId=jenkins_20220729075624_b3ba4c8a-82d5-4ebd-b4b0-218325a71b10):
> INSERT into table
> default.tmp_minor_compactor_testmmminorcompaction_1659106584519_result select
> `a`, `b` from default.tmp_minor_compactor_testmmminorcompaction_1659106584519
> 2022-07-29T07:56:24,823 INFO [ServiceThread:DAGClientRPCServer]
> client.DAGClientServer: Instantiated DAGClientRPCServer at
> internal-hive-flaky-check-88-xwmrs-v2h77-knnxx/10.106.3.19:22623
> 2022-07-29T07:56:24,823 INFO
> [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager]
> rm.TaskSchedulerManager: Creating TaskScheduler: Local TaskScheduler with
> clusterIdentifier=1659106584728
> 2022-07-29T07:56:24,825 INFO [DAGAppMaster Thread]
> HistoryEventHandler.criticalEvents: [HISTORY][DAG:N/A][Event:AM_STARTED]:
> appAttemptId=appattempt_1659106584728_0001_00, startTime=1659106584825
> 2022-07-29T07:56:24,825 INFO [DAGAppMaster Thread] app.DAGAppMaster: In
> Session mode. Waiting for DAG over RPC
> 2022-07-29T07:56:24,871 INFO [main_executor] client.LocalClient:
> DAGAppMaster state: IDLE
> 2022-07-29T07:56:24,871 INFO [main_executor] client.TezClient: The url to
> track the Tez Session: N/A
> ...
> 2022-07-29T07:56:46,384 INFO [main_executor] client.TezClient: Failed to
> retrieve AM Status via proxy
> com.google.protobuf.ServiceException: java.net.ConnectException: Call From
> internal-hive-flaky-check-88-xwmrs-v2h77-knnxx/10.106.3.19 to
> internal-hive-flaky-check-88-xwmrs-v2h77-knnxx:22623 failed on connection
> exception: java.net.ConnectException: Connection refused; For more details
> see: http://wiki.apache.org/hadoop/ConnectionRefused
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:247)
> ~[hadoop-common-3.1.1.7.2.15.0-147.jar:?]
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> ~[hadoop-common-3.1.1.7.2.15.0-147.jar:?]
> at com.sun.proxy.$Proxy50.getAMStatus(Unknown Source) ~[?:?]
> {code}
> instead of diving deep into an evil environment related bug, we can simply
> utilize TEZ-4236 in these cases too
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26414) Aborted/Cancelled CTAS operations must initiate cleanup of uncommitted data
[ https://issues.apache.org/jira/browse/HIVE-26414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17574587#comment-17574587 ] Sourabh Badhya commented on HIVE-26414: --- Thanks [~dkuzmenko] for the review. > Aborted/Cancelled CTAS operations must initiate cleanup of uncommitted data > --- > > Key: HIVE-26414 > URL: https://issues.apache.org/jira/browse/HIVE-26414 > Project: Hive > Issue Type: Improvement >Reporter: Sourabh Badhya >Assignee: Sourabh Badhya >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 13.5h > Remaining Estimate: 0h > > When a CTAS query fails before creation of table and after writing the data, > the data is present in the directory and not cleaned up currently by the > cleaner or any other mechanism currently. This is because the cleaner > requires a table corresponding to what its cleaning. In order surpass such a > situation, we can directly pass the relevant information to the cleaner so > that such uncommitted data is deleted. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26414) Aborted/Cancelled CTAS operations must initiate cleanup of uncommitted data
[ https://issues.apache.org/jira/browse/HIVE-26414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sourabh Badhya resolved HIVE-26414. --- Fix Version/s: 4.0.0-alpha-2 Resolution: Fixed > Aborted/Cancelled CTAS operations must initiate cleanup of uncommitted data > --- > > Key: HIVE-26414 > URL: https://issues.apache.org/jira/browse/HIVE-26414 > Project: Hive > Issue Type: Improvement >Reporter: Sourabh Badhya >Assignee: Sourabh Badhya >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 13.5h > Remaining Estimate: 0h > > When a CTAS query fails before creation of table and after writing the data, > the data is present in the directory and not cleaned up currently by the > cleaner or any other mechanism currently. This is because the cleaner > requires a table corresponding to what its cleaning. In order surpass such a > situation, we can directly pass the relevant information to the cleaner so > that such uncommitted data is deleted. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26437) dump unpartitioned Tables in parallel
[ https://issues.apache.org/jira/browse/HIVE-26437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17574576#comment-17574576 ] Amit Saonerkar commented on HIVE-26437: --- Currently partitions of table is dump in parallel manner. But if table is not partitioned, it is dumped serially. Parallelism at partition level is governed by config param "REPL_PARTITIONS_DUMP_PARALLELISM". Table level parallelism during BootStrapDump() and incrementalDump() is missing. > dump unpartitioned Tables in parallel > - > > Key: HIVE-26437 > URL: https://issues.apache.org/jira/browse/HIVE-26437 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Amit Saonerkar >Assignee: Amit Saonerkar >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26414) Aborted/Cancelled CTAS operations must initiate cleanup of uncommitted data
[ https://issues.apache.org/jira/browse/HIVE-26414?focusedWorklogId=797529=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-797529 ] ASF GitHub Bot logged work on HIVE-26414: - Author: ASF GitHub Bot Created on: 03/Aug/22 07:50 Start Date: 03/Aug/22 07:50 Worklog Time Spent: 10m Work Description: deniskuzZ merged PR #3457: URL: https://github.com/apache/hive/pull/3457 Issue Time Tracking --- Worklog Id: (was: 797529) Time Spent: 13.5h (was: 13h 20m) > Aborted/Cancelled CTAS operations must initiate cleanup of uncommitted data > --- > > Key: HIVE-26414 > URL: https://issues.apache.org/jira/browse/HIVE-26414 > Project: Hive > Issue Type: Improvement >Reporter: Sourabh Badhya >Assignee: Sourabh Badhya >Priority: Major > Labels: pull-request-available > Time Spent: 13.5h > Remaining Estimate: 0h > > When a CTAS query fails before creation of table and after writing the data, > the data is present in the directory and not cleaned up currently by the > cleaner or any other mechanism currently. This is because the cleaner > requires a table corresponding to what its cleaning. In order surpass such a > situation, we can directly pass the relevant information to the cleaner so > that such uncommitted data is deleted. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26444) Use tez.local.mode.without.network for unit tests
[
https://issues.apache.org/jira/browse/HIVE-26444?focusedWorklogId=797519=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-797519
]
ASF GitHub Bot logged work on HIVE-26444:
-
Author: ASF GitHub Bot
Created on: 03/Aug/22 07:00
Start Date: 03/Aug/22 07:00
Worklog Time Spent: 10m
Work Description: abstractdog commented on code in PR #3491:
URL: https://github.com/apache/hive/pull/3491#discussion_r936304092
##
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveShell.java:
##
@@ -201,6 +201,8 @@ private HiveConf initializeConf() {
// Tez configuration
hiveConf.setBoolean("tez.local.mode", true);
+// TODO: enable below option once HIVE-26445 is investigated
+// hiveConf.setBoolean("tez.local.mode.without.network", true);
Review Comment:
recently, only hive iceberg related tests failed
[here](http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-3491/2/pipeline),
and I cannot easily disable this feature only for
TestIcebergLlapLocalCliDriver (but not for other CliDrivers), so I'm about to
make this change only for some unit tests (which I targeted originally), and
handle all the qtests in the followup HIVE-26445, does it make sense?
Issue Time Tracking
---
Worklog Id: (was: 797519)
Time Spent: 40m (was: 0.5h)
> Use tez.local.mode.without.network for unit tests
> -
>
> Key: HIVE-26444
> URL: https://issues.apache.org/jira/browse/HIVE-26444
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Since TEZ-4236 (in Tez 0.10.1), tez local mode can run without even starting
> an RPC server in the DAGAppMaster, which is in the same JVM as the client.
> Adapting tez.local.mode.without.network=true could make tez.local.mode=true
> unit tests more stable.
> here is an example where I had no idea why the dag app master connection was
> refused:
> {code}
> 2022-07-29T07:56:24,701 INFO [main_executor] ql.Driver: Executing
> command(queryId=jenkins_20220729075624_b3ba4c8a-82d5-4ebd-b4b0-218325a71b10):
> INSERT into table
> default.tmp_minor_compactor_testmmminorcompaction_1659106584519_result select
> `a`, `b` from default.tmp_minor_compactor_testmmminorcompaction_1659106584519
> 2022-07-29T07:56:24,823 INFO [ServiceThread:DAGClientRPCServer]
> client.DAGClientServer: Instantiated DAGClientRPCServer at
> internal-hive-flaky-check-88-xwmrs-v2h77-knnxx/10.106.3.19:22623
> 2022-07-29T07:56:24,823 INFO
> [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager]
> rm.TaskSchedulerManager: Creating TaskScheduler: Local TaskScheduler with
> clusterIdentifier=1659106584728
> 2022-07-29T07:56:24,825 INFO [DAGAppMaster Thread]
> HistoryEventHandler.criticalEvents: [HISTORY][DAG:N/A][Event:AM_STARTED]:
> appAttemptId=appattempt_1659106584728_0001_00, startTime=1659106584825
> 2022-07-29T07:56:24,825 INFO [DAGAppMaster Thread] app.DAGAppMaster: In
> Session mode. Waiting for DAG over RPC
> 2022-07-29T07:56:24,871 INFO [main_executor] client.LocalClient:
> DAGAppMaster state: IDLE
> 2022-07-29T07:56:24,871 INFO [main_executor] client.TezClient: The url to
> track the Tez Session: N/A
> ...
> 2022-07-29T07:56:46,384 INFO [main_executor] client.TezClient: Failed to
> retrieve AM Status via proxy
> com.google.protobuf.ServiceException: java.net.ConnectException: Call From
> internal-hive-flaky-check-88-xwmrs-v2h77-knnxx/10.106.3.19 to
> internal-hive-flaky-check-88-xwmrs-v2h77-knnxx:22623 failed on connection
> exception: java.net.ConnectException: Connection refused; For more details
> see: http://wiki.apache.org/hadoop/ConnectionRefused
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:247)
> ~[hadoop-common-3.1.1.7.2.15.0-147.jar:?]
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> ~[hadoop-common-3.1.1.7.2.15.0-147.jar:?]
> at com.sun.proxy.$Proxy50.getAMStatus(Unknown Source) ~[?:?]
> {code}
> instead of diving deep into an evil environment related bug, we can simply
> utilize TEZ-4236 in these cases too
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-24484) Upgrade Hadoop to 3.3.1 And Tez to 0.10.2
[ https://issues.apache.org/jira/browse/HIVE-24484?focusedWorklogId=797517=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-797517 ] ASF GitHub Bot logged work on HIVE-24484: - Author: ASF GitHub Bot Created on: 03/Aug/22 06:46 Start Date: 03/Aug/22 06:46 Worklog Time Spent: 10m Work Description: abstractdog commented on PR #3279: URL: https://github.com/apache/hive/pull/3279#issuecomment-1203550667 also, let me grab the opportunity to thank @belugabehr who put enormous efforts into the hadoop upgrade in the early days! Issue Time Tracking --- Worklog Id: (was: 797517) Time Spent: 15.05h (was: 14h 53m) > Upgrade Hadoop to 3.3.1 And Tez to 0.10.2 > -- > > Key: HIVE-24484 > URL: https://issues.apache.org/jira/browse/HIVE-24484 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 15.05h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26415) Add epoch time in the information_schema.scheduled_executions view
[ https://issues.apache.org/jira/browse/HIVE-26415?focusedWorklogId=797516=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-797516 ] ASF GitHub Bot logged work on HIVE-26415: - Author: ASF GitHub Bot Created on: 03/Aug/22 06:44 Start Date: 03/Aug/22 06:44 Worklog Time Spent: 10m Work Description: pudidic commented on code in PR #3467: URL: https://github.com/apache/hive/pull/3467#discussion_r936292595 ## metastore/scripts/upgrade/hive/hive-schema-4.0.0-alpha-2.hive.sql: ## @@ -1980,7 +1980,8 @@ SELECT FROM_UNIXTIME(END_TIME) as END_TIME, END_TIME-START_TIME as ELAPSED, ERROR_MESSAGE, - FROM_UNIXTIME(LAST_UPDATE_TIME) AS LAST_UPDATE_TIME + FROM_UNIXTIME(LAST_UPDATE_TIME) AS LAST_UPDATE_TIME, + START_TIME as START_TIME_SINCE_EPOCH FROM Review Comment: There is upgrade-4.0.0-alpha1-to-4.0.0-alpha-2.hive.sql, too. Issue Time Tracking --- Worklog Id: (was: 797516) Remaining Estimate: 167h 20m (was: 167.5h) Time Spent: 40m (was: 0.5h) > Add epoch time in the information_schema.scheduled_executions view > -- > > Key: HIVE-26415 > URL: https://issues.apache.org/jira/browse/HIVE-26415 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: Imran >Assignee: Shreenidhi >Priority: Major > Labels: pull-request-available > Original Estimate: 168h > Time Spent: 40m > Remaining Estimate: 167h 20m > > information_schema.scheduled_executions shows time as the System time. > replication_metrics shows time in epoch time. > Only way to corelate the two is using the scheduled_execution id. Looking at > the time at the 2 tables causes some confusion. So we can add a new column in > the information_schema.scheduled_executions view displaying the epoch time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-24483) Bump protobuf version to 3.21.1 (aarch64 support)
[ https://issues.apache.org/jira/browse/HIVE-24483?focusedWorklogId=797511=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-797511 ] ASF GitHub Bot logged work on HIVE-24483: - Author: ASF GitHub Bot Created on: 03/Aug/22 06:04 Start Date: 03/Aug/22 06:04 Worklog Time Spent: 10m Work Description: ayushtkn opened a new pull request, #3498: URL: https://github.com/apache/hive/pull/3498 Exploratory Sate: See what all breaks Issue Time Tracking --- Worklog Id: (was: 797511) Time Spent: 2.5h (was: 2h 20m) > Bump protobuf version to 3.21.1 (aarch64 support) > - > > Key: HIVE-24483 > URL: https://issues.apache.org/jira/browse/HIVE-24483 > Project: Hive > Issue Type: Improvement >Reporter: Cameron Moberg >Assignee: Ayush Saxena >Priority: Minor > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > The following protoc version's used in hive is very old i.e. 2.5.0 > [https://repo.maven.apache.org/maven2/com/google/protobuf/protoc/] . The > v2.5.0 does not have aarch64 support. But the AArch64 support started from > v3.5.0 on-words in google's protobuf project release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
