[jira] [Updated] (HIVE-27767) Copy more data into HIVE_LOCKS for better supportability
[ https://issues.apache.org/jira/browse/HIVE-27767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-27767: -- Labels: pull-request-available (was: ) > Copy more data into HIVE_LOCKS for better supportability > > > Key: HIVE-27767 > URL: https://issues.apache.org/jira/browse/HIVE-27767 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: KIRTI RUGE >Assignee: KIRTI RUGE >Priority: Major > Labels: pull-request-available > > There is some information like ERROR_MESSAGE needs to copy to HIVE_LOCKS . It > would help with supportability if HIVE_LOCKS (and especially the view of it > in the SYS database) also contained this information. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27723) Prevent localizing the same original file more than once if symlinks are present
[
https://issues.apache.org/jira/browse/HIVE-27723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17772076#comment-17772076
]
László Bodor commented on HIVE-27723:
-
merged to master, thanks [~ayushsaxena] and [~dkuzmenko] for all the comments
on this jira!
> Prevent localizing the same original file more than once if symlinks are
> present
>
>
> Key: HIVE-27723
> URL: https://issues.apache.org/jira/browse/HIVE-27723
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> We already calculate SHA hashes for the files to be localized. There is a
> chance, that in some setups, the hive-exec jars are symlinked so it gets
> localized more than once.
> {code}
> [root@lbodor-hiveontez-4 ~]# sudo -u hive hdfs dfs -ls -R
> /tmp/hive/hive/_tez_session_dir
> drwx-- - hive supergroup 0 2023-09-20 12:13
> /tmp/hive/hive/_tez_session_dir/0febf6f5-bacc-4055-b22b-e621c59cd1d6
> drwx-- - hive supergroup 0 2023-09-20 12:19
> /tmp/hive/hive/_tez_session_dir/0febf6f5-bacc-4055-b22b-e621c59cd1d6/.tez
> drwx-- - hive supergroup 0 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/0febf6f5-bacc-4055-b22b-e621c59cd1d6-resources
> -rw-r--r-- 3 hive supergroup 78366781 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/0febf6f5-bacc-4055-b22b-e621c59cd1d6-resources/hive-exec-3.1.3000.7.2.18.0-334.jar
> -rw-r--r-- 3 hive supergroup 78366781 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/0febf6f5-bacc-4055-b22b-e621c59cd1d6-resources/hive-exec.jar
> drwx-- - hive supergroup 0 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/21686e3c-2a00-457b-b84f-1a8db37699d1
> drwx-- - hive supergroup 0 2023-09-20 12:04
> /tmp/hive/hive/_tez_session_dir/21686e3c-2a00-457b-b84f-1a8db37699d1/.tez
> drwx-- - hive supergroup 0 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/21686e3c-2a00-457b-b84f-1a8db37699d1-resources
> -rw-r--r-- 3 hive supergroup 78366781 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/21686e3c-2a00-457b-b84f-1a8db37699d1-resources/hive-exec-3.1.3000.7.2.18.0-334.jar
> -rw-r--r-- 3 hive supergroup 78366781 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/21686e3c-2a00-457b-b84f-1a8db37699d1-resources/hive-exec.jar
> drwx-- - hive supergroup 0 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/40c7fb13-cfa1-4377-8d40-7e19503fbdad
> drwx-- - hive supergroup 0 2023-09-20 13:13
> /tmp/hive/hive/_tez_session_dir/40c7fb13-cfa1-4377-8d40-7e19503fbdad/.tez
> drwx-- - hive supergroup 0 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/40c7fb13-cfa1-4377-8d40-7e19503fbdad-resources
> -rw-r--r-- 3 hive supergroup 78366781 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/40c7fb13-cfa1-4377-8d40-7e19503fbdad-resources/hive-exec-3.1.3000.7.2.18.0-334.jar
> -rw-r--r-- 3 hive supergroup 78366781 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/40c7fb13-cfa1-4377-8d40-7e19503fbdad-resources/hive-exec.jar
> drwx-- - hive supergroup 0 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/5c48d6ab-ed8c-49c9-afe0-465de82c9c57
> drwx-- - hive supergroup 0 2023-09-20 12:04
> /tmp/hive/hive/_tez_session_dir/5c48d6ab-ed8c-49c9-afe0-465de82c9c57/.tez
> drwx-- - hive supergroup 0 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/5c48d6ab-ed8c-49c9-afe0-465de82c9c57-resources
> -rw-r--r-- 3 hive supergroup 78366781 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/5c48d6ab-ed8c-49c9-afe0-465de82c9c57-resources/hive-exec-3.1.3000.7.2.18.0-334.jar
> -rw-r--r-- 3 hive supergroup 78366781 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/5c48d6ab-ed8c-49c9-afe0-465de82c9c57-resources/hive-exec.jar
> {code}
> in the presence of huge amount of sessions, we cannot afford this overhead of
> copying this files to HDFS and localizing to all containers twice
> the root cause can be solved by removing symlinks of the same hive-exec jar,
> -however, as we're already calculating SHA for the files, it's so easy to
> take care of the duplications in the localization codepath, and this takes
> care of any accidental duplications- so if all symlinks point to the same
> jar, resolving those before passing the Path objects to the localization
> codepath would simply solve this issue
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-27723) Prevent localizing the same original file more than once if symlinks are present
[
https://issues.apache.org/jira/browse/HIVE-27723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor resolved HIVE-27723.
-
Resolution: Fixed
> Prevent localizing the same original file more than once if symlinks are
> present
>
>
> Key: HIVE-27723
> URL: https://issues.apache.org/jira/browse/HIVE-27723
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> We already calculate SHA hashes for the files to be localized. There is a
> chance, that in some setups, the hive-exec jars are symlinked so it gets
> localized more than once.
> {code}
> [root@lbodor-hiveontez-4 ~]# sudo -u hive hdfs dfs -ls -R
> /tmp/hive/hive/_tez_session_dir
> drwx-- - hive supergroup 0 2023-09-20 12:13
> /tmp/hive/hive/_tez_session_dir/0febf6f5-bacc-4055-b22b-e621c59cd1d6
> drwx-- - hive supergroup 0 2023-09-20 12:19
> /tmp/hive/hive/_tez_session_dir/0febf6f5-bacc-4055-b22b-e621c59cd1d6/.tez
> drwx-- - hive supergroup 0 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/0febf6f5-bacc-4055-b22b-e621c59cd1d6-resources
> -rw-r--r-- 3 hive supergroup 78366781 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/0febf6f5-bacc-4055-b22b-e621c59cd1d6-resources/hive-exec-3.1.3000.7.2.18.0-334.jar
> -rw-r--r-- 3 hive supergroup 78366781 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/0febf6f5-bacc-4055-b22b-e621c59cd1d6-resources/hive-exec.jar
> drwx-- - hive supergroup 0 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/21686e3c-2a00-457b-b84f-1a8db37699d1
> drwx-- - hive supergroup 0 2023-09-20 12:04
> /tmp/hive/hive/_tez_session_dir/21686e3c-2a00-457b-b84f-1a8db37699d1/.tez
> drwx-- - hive supergroup 0 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/21686e3c-2a00-457b-b84f-1a8db37699d1-resources
> -rw-r--r-- 3 hive supergroup 78366781 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/21686e3c-2a00-457b-b84f-1a8db37699d1-resources/hive-exec-3.1.3000.7.2.18.0-334.jar
> -rw-r--r-- 3 hive supergroup 78366781 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/21686e3c-2a00-457b-b84f-1a8db37699d1-resources/hive-exec.jar
> drwx-- - hive supergroup 0 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/40c7fb13-cfa1-4377-8d40-7e19503fbdad
> drwx-- - hive supergroup 0 2023-09-20 13:13
> /tmp/hive/hive/_tez_session_dir/40c7fb13-cfa1-4377-8d40-7e19503fbdad/.tez
> drwx-- - hive supergroup 0 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/40c7fb13-cfa1-4377-8d40-7e19503fbdad-resources
> -rw-r--r-- 3 hive supergroup 78366781 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/40c7fb13-cfa1-4377-8d40-7e19503fbdad-resources/hive-exec-3.1.3000.7.2.18.0-334.jar
> -rw-r--r-- 3 hive supergroup 78366781 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/40c7fb13-cfa1-4377-8d40-7e19503fbdad-resources/hive-exec.jar
> drwx-- - hive supergroup 0 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/5c48d6ab-ed8c-49c9-afe0-465de82c9c57
> drwx-- - hive supergroup 0 2023-09-20 12:04
> /tmp/hive/hive/_tez_session_dir/5c48d6ab-ed8c-49c9-afe0-465de82c9c57/.tez
> drwx-- - hive supergroup 0 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/5c48d6ab-ed8c-49c9-afe0-465de82c9c57-resources
> -rw-r--r-- 3 hive supergroup 78366781 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/5c48d6ab-ed8c-49c9-afe0-465de82c9c57-resources/hive-exec-3.1.3000.7.2.18.0-334.jar
> -rw-r--r-- 3 hive supergroup 78366781 2023-09-20 11:58
> /tmp/hive/hive/_tez_session_dir/5c48d6ab-ed8c-49c9-afe0-465de82c9c57-resources/hive-exec.jar
> {code}
> in the presence of huge amount of sessions, we cannot afford this overhead of
> copying this files to HDFS and localizing to all containers twice
> the root cause can be solved by removing symlinks of the same hive-exec jar,
> -however, as we're already calculating SHA for the files, it's so easy to
> take care of the duplications in the localization codepath, and this takes
> care of any accidental duplications- so if all symlinks point to the same
> jar, resolving those before passing the Path objects to the localization
> codepath would simply solve this issue
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-27757) Upgrade hadoop to 3.3.6
[ https://issues.apache.org/jira/browse/HIVE-27757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HIVE-27757. - Fix Version/s: 4.0.0 Resolution: Fixed > Upgrade hadoop to 3.3.6 > --- > > Key: HIVE-27757 > URL: https://issues.apache.org/jira/browse/HIVE-27757 > Project: Hive > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Hadoop 3.3.6 is released and comes up with lots of improvements & CVE fixes -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27757) Upgrade hadoop to 3.3.6
[ https://issues.apache.org/jira/browse/HIVE-27757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17772071#comment-17772071 ] Ayush Saxena commented on HIVE-27757: - Committed to master. Thanx [~sbadhya] & [~simhadri-g] for the reviews!!! > Upgrade hadoop to 3.3.6 > --- > > Key: HIVE-27757 > URL: https://issues.apache.org/jira/browse/HIVE-27757 > Project: Hive > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > > Hadoop 3.3.6 is released and comes up with lots of improvements & CVE fixes -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27770) Consider localizing hive-exec jar as a PUBLIC resource
[ https://issues.apache.org/jira/browse/HIVE-27770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-27770: Description: we might want to discover the possibility to localize hive-exec jar as PUBLIC, similarly to the tez.tar.gz here: [https://github.com/apache/tez/blob/2ad10b61a0263770e4e78983939bb9d823d9dc27/tez-api/src/main/java/org/apache/tez/client/TezClientUtils.java#L271-L277] this way, hive-exec.jar will be downloaded once per node instead of once per session, saving lots of HDFS resources and speeding up tez app start a bit > Consider localizing hive-exec jar as a PUBLIC resource > -- > > Key: HIVE-27770 > URL: https://issues.apache.org/jira/browse/HIVE-27770 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Priority: Major > > we might want to discover the possibility to localize hive-exec jar as > PUBLIC, similarly to the tez.tar.gz here: > [https://github.com/apache/tez/blob/2ad10b61a0263770e4e78983939bb9d823d9dc27/tez-api/src/main/java/org/apache/tez/client/TezClientUtils.java#L271-L277] > > this way, hive-exec.jar will be downloaded once per node instead of once per > session, saving lots of HDFS resources and speeding up tez app start a bit -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27770) Consider localizing hive-exec jar as a PUBLIC resource
László Bodor created HIVE-27770: --- Summary: Consider localizing hive-exec jar as a PUBLIC resource Key: HIVE-27770 URL: https://issues.apache.org/jira/browse/HIVE-27770 Project: Hive Issue Type: Improvement Reporter: László Bodor -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-27767) Copy more data into HIVE_LOCKS for better supportability
[ https://issues.apache.org/jira/browse/HIVE-27767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KIRTI RUGE reassigned HIVE-27767: - Assignee: KIRTI RUGE > Copy more data into HIVE_LOCKS for better supportability > > > Key: HIVE-27767 > URL: https://issues.apache.org/jira/browse/HIVE-27767 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: KIRTI RUGE >Assignee: KIRTI RUGE >Priority: Major > > There is some information like ERROR_MESSAGE needs to copy to HIVE_LOCKS . It > would help with supportability if HIVE_LOCKS (and especially the view of it > in the SYS database) also contained this information. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27769) If there has the plan to upgrade jdk version?
sanqingleo created HIVE-27769:
-
Summary: If there has the plan to upgrade jdk version?
Key: HIVE-27769
URL: https://issues.apache.org/jira/browse/HIVE-27769
Project: Hive
Issue Type: New Feature
Environment: Hive 3.1.3
Flink 1.16
Hudi 0.13
Reporter: sanqingleo
Hi guys, I'm now using hive(3.1.3) catalog. When I use flink to create catalog,
It will throw the exception like this
{code:java}
Caused by: MetaException(message:Got exception: java.lang.ClassCastException
class [Ljava.lang.Object; cannot be cast to class [Ljava.net.URI;
([Ljava.lang.Object; and [Ljava.net.URI; are in module java.base of loader
'bootstrap')) at
org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:1342)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:278)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:210)
... 20 more
{code}
I have to downgrade my JDK version to 8
As I know , now many open source softwares are supporting JDK17.(like
flink,hudi..) I want to know if there has the plan to upgrade JDK version? It
will support JDK17 since which hive version?
Thx
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27584) Backport HIVE-21407 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-27584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17771785#comment-17771785 ] Chinna Rao Lalam commented on HIVE-27584: - LGTM +1 > Backport HIVE-21407 to branch-3 > --- > > Key: HIVE-27584 > URL: https://issues.apache.org/jira/browse/HIVE-27584 > Project: Hive > Issue Type: Task >Reporter: Pravin Sinha >Assignee: Pravin Sinha >Priority: Major > Labels: pull-request-available > > HIVE-21407: Parquet predicate pushdown is not working correctly for char > column types (Marta Kuczora reviewed by Peter Vary) -- This message was sent by Atlassian Jira (v8.20.10#820010)
