[jira] [Created] (HIVE-25831) Report Progress on Every Record Read for CompactorMR
David Mollitor created HIVE-25831:
-
Summary: Report Progress on Every Record Read for CompactorMR
Key: HIVE-25831
URL: https://issues.apache.org/jira/browse/HIVE-25831
Project: Hive
Issue Type: Improvement
Reporter: David Mollitor
Progress should be updated for every read of an input
{quote}
reads an input, writes an output, nor updates its status string
{quote}
https://github.com/apache/hive/blob/fffb31f2346df2b8011a9949895de21f506c0117/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java#L813-L828
I think ever loop should simply be calling {{progress()}}. If during a major
compaction there are a lot of deleted values, long gaps of time can occur
without a progress update and the job may be timed out by YARN.
I'm not 100% sure this is happening, but just something I wanted to point out.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HIVE-25830) Hive::loadPartitionInternal occur connection leak
tenglei created HIVE-25830: -- Summary: Hive::loadPartitionInternal occur connection leak Key: HIVE-25830 URL: https://issues.apache.org/jira/browse/HIVE-25830 Project: Hive Issue Type: Bug Reporter: tenglei when Hive::loadPartitionInternal is invoked and there's no metastore connection in Hive.ThreadLocalHive, will create a metastore connection for Hive.ThreadLocalHive, if external part hold the Hive.ThreadLocalHive for cache, and Hive.ThreadLocalHive update in some special scene, this will cause external part hold the different Hive object to Hive.ThreadLocalHive.when external part want to create a metastore connection by invoking Hive::loadPartitionInternal, Hive will create a metastore connection for Hive.ThreadLocalHive, not for external part's holding.so these can cause a connection leaks easliy. I found it will cause a metastore connection leak in spark-beeline when invoke Hive::loadPartitionInternal. there's a commit HIVE-25075 relate before, I think it should remove the Hive.get() in Hive::loadPartitionInternal at the same time. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HIVE-25829) Tez exec mode support for credential provider for jobs
Ádám Szita created HIVE-25829: - Summary: Tez exec mode support for credential provider for jobs Key: HIVE-25829 URL: https://issues.apache.org/jira/browse/HIVE-25829 Project: Hive Issue Type: Improvement Components: HiveServer2 Reporter: Ádám Szita HIVE-14822 introduced support to securely forward a job specific java credential store path, and a corresponding password to the backend executors. This is currently implemented for only MR2 and Spark execution engines. I propose we extend this feature by adding Tez mode to said list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
