[jira] [Commented] (HIVE-25115) Compaction queue entries may accumulate in "ready for cleaning" state
[ https://issues.apache.org/jira/browse/HIVE-25115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17568601#comment-17568601 ] Karen Coppage commented on HIVE-25115: -- That's right, a better way to say this is: "queue entry remains in the queue." > Compaction queue entries may accumulate in "ready for cleaning" state > - > > Key: HIVE-25115 > URL: https://issues.apache.org/jira/browse/HIVE-25115 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 4h 20m > Remaining Estimate: 0h > > If the Cleaner does not delete any files, the compaction queue entry is > thrown back to the queue and remains in "ready for cleaning" state. > Problem: If 2 compactions run on the same table and enter "ready for > cleaning" state at the same time, only one "cleaning" will remove obsolete > files, the other entry will remain in the queue in "ready for cleaning" state. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26280) Copy more data into COMPLETED_COMPACTIONS for better supportability
[ https://issues.apache.org/jira/browse/HIVE-26280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-26280. -- Fix Version/s: 4.0.0-alpha-2 Resolution: Fixed Committed to master branch. Thanks for reviewing [~dkuzmenko]! > Copy more data into COMPLETED_COMPACTIONS for better supportability > --- > > Key: HIVE-26280 > URL: https://issues.apache.org/jira/browse/HIVE-26280 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > There is some information in COMPACTION_QUEUE that doesn't get copied over to > COMPLETED_COMPACTIONS when compaction completes. It would help with > supportability if COMPLETED_COMPACTIONS (and especially the view of it in the > SYS database) also contained this information. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (HIVE-26280) Copy more data into COMPLETED_COMPACTIONS for better supportability
[ https://issues.apache.org/jira/browse/HIVE-26280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reassigned HIVE-26280: > Copy more data into COMPLETED_COMPACTIONS for better supportability > --- > > Key: HIVE-26280 > URL: https://issues.apache.org/jira/browse/HIVE-26280 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Minor > > There is some information in COMPACTION_QUEUE that doesn't get copied over to > COMPLETED_COMPACTIONS when compaction completes. It would help with > supportability if COMPLETED_COMPACTIONS (and especially the view of it in the > SYS database) also contained this information. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HIVE-22633) GroupByOperator may throw NullPointerException when setting data skew optimization parameters
[
https://issues.apache.org/jira/browse/HIVE-22633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17544217#comment-17544217
]
Karen Coppage commented on HIVE-22633:
--
HIVE-23530 is probably the reason why this isn't reproable. It seems to work by
avoiding using UDAFs to compute statistics.
> GroupByOperator may throw NullPointerException when setting data skew
> optimization parameters
> -
>
> Key: HIVE-22633
> URL: https://issues.apache.org/jira/browse/HIVE-22633
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.1.1, 4.0.0
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
>
> if hive.map.aggr and hive.groupby.skewindata set true，exception will be
> thrown.
> step to repro:
> 1. create table:
> set hive.map.aggr=true;
> set hive.groupby.skewindata=true;
> create table test1 (id1 bigint);
> create table test2 (id2 bigint) partitioned by(dt2 string);
> insert into test2 partition(dt2='2020') select a.id1 from test1 a group by
> a.id1;
> 2.NullPointerException:
> {code:java}
> ], TaskAttempt 2 failed, info=[Error: Error while running task ( failure ) :
> attempt_1585641455670_0001_2_03_00_2:java.lang.RuntimeException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFComputeStats$GenericUDAFNumericStatsEvaluator.init(GenericUDAFComputeStats.java:373)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.initializeOp(GroupByOperator.java:373)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:360)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:191)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Resolved] (HIVE-26147) OrcRawRecordMerger throws NPE when hive.acid.key.index is missing for an acid file
[
https://issues.apache.org/jira/browse/HIVE-26147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage resolved HIVE-26147.
--
Fix Version/s: 4.0.0-alpha-2
Resolution: Fixed
> OrcRawRecordMerger throws NPE when hive.acid.key.index is missing for an acid
> file
> --
>
> Key: HIVE-26147
> URL: https://issues.apache.org/jira/browse/HIVE-26147
> Project: Hive
> Issue Type: Bug
> Components: ORC, Transactions
>Affects Versions: 4.0.0-alpha-2
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When _hive.acid.key.index_ is missing for an acid ORC file
> _OrcRawRecordMerger_ throws as follows:
> {noformat}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.discoverKeyBounds(OrcRawRecordMerger.java:795)
> ~[hive-exec-4.0.0-alpha-2-SNAPS
> HOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:1053)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.
> 0.0-alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:2096)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-a
> lpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1991)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4
> .0.0-alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:769)
> ~[hive-exec-4.0.0-alpha
> -2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:335)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-
> alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:560)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha
> -2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:529)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-
> SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:150)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.Driver.getFetchingTableResults(Driver.java:719)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNA
> PSHOT]
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:671)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:233)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha
> -2-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:489)
> ~[hive-service-4.0.0-alpha-2-SNAPSHOT.jar:
> 4.0.0-alpha-2-SNAPSHOT]
> ... 24 more
> {noformat}
> For this situation to happen, the ORC file must have more than one stripe,
> and the offset of the element to seek should either locate it beyond the
> first stripe (but before the last one), or in the first one if not the last
> one, as the code shows:
> {code:java}
> if (firstStripe != 0) {
> minKey = keyIndex[firstStripe - 1];
> }
> if (!isTail) {
> maxKey = keyIndex[firstStripe + stripeCount - 1];
> }
> {code}
> However, in the context of the detection of the original issue, the NPE was
> triggered even by a simple "select *" over a table with ORC files missing the
> _hive.acid.key.index_ metadata information, but it was never failing for ORC
> files with a single stripe. The file was generated after a major compaction
> of acid and non-acid data.
> If the "select *" is not triggering the NPE, either pick the values of the
> row obtained with "select * from $table limit 1", or try to select based on
> different values trying to get into the sought situation with a filter like
> this:
> {code:sql}
> select * from $table where c = $value
> {code}
> _OrcRawRecordMerger_ should simply leave as "null" the min and max keys when
> the _hive.acid.key.index_ metadata is missing.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Resolved] (HIVE-26146) Handle missing hive.acid.key.index in the fixacidkeyindex utility
[
https://issues.apache.org/jira/browse/HIVE-26146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage resolved HIVE-26146.
--
Fix Version/s: 4.0.0-alpha-2
Resolution: Fixed
> Handle missing hive.acid.key.index in the fixacidkeyindex utility
> -
>
> Key: HIVE-26146
> URL: https://issues.apache.org/jira/browse/HIVE-26146
> Project: Hive
> Issue Type: Improvement
> Components: ORC, Transactions
>Affects Versions: 4.0.0-alpha-2
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Critical
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> There is a utility in hive which can validate/fix corrupted
> _hive.acid.key.index_:
> {code:bash}
> hive --service fixacidkeyindex
> {code}
> At the moment the utility throws a NPE if the _hive.acid.key.index_ metadata
> entry is missing:
> {noformat}
> ERROR checking /hive-dev-box/multistripe_ko_acid.orc
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.orc.FixAcidKeyIndex.validate(FixAcidKeyIndex.java:183)
> at
> org.apache.hadoop.hive.ql.io.orc.FixAcidKeyIndex.checkFile(FixAcidKeyIndex.java:147)
> at
> org.apache.hadoop.hive.ql.io.orc.FixAcidKeyIndex.checkFiles(FixAcidKeyIndex.java:130)
> at
> org.apache.hadoop.hive.ql.io.orc.FixAcidKeyIndex.main(FixAcidKeyIndex.java:106)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:308)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:222)
> {noformat}
> The aim of this ticket is to handle such case in order to support
> re-generating this metadata entry even when it is missing.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HIVE-26146) Handle missing hive.acid.key.index in the fixacidkeyindex utility
[
https://issues.apache.org/jira/browse/HIVE-26146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17524379#comment-17524379
]
Karen Coppage commented on HIVE-26146:
--
Committed to master branch. Thanks for your contribution [~asolimando] !
> Handle missing hive.acid.key.index in the fixacidkeyindex utility
> -
>
> Key: HIVE-26146
> URL: https://issues.apache.org/jira/browse/HIVE-26146
> Project: Hive
> Issue Type: Improvement
> Components: ORC, Transactions
>Affects Versions: 4.0.0-alpha-2
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> There is a utility in hive which can validate/fix corrupted
> _hive.acid.key.index_:
> {code:bash}
> hive --service fixacidkeyindex
> {code}
> At the moment the utility throws a NPE if the _hive.acid.key.index_ metadata
> entry is missing:
> {noformat}
> ERROR checking /hive-dev-box/multistripe_ko_acid.orc
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.orc.FixAcidKeyIndex.validate(FixAcidKeyIndex.java:183)
> at
> org.apache.hadoop.hive.ql.io.orc.FixAcidKeyIndex.checkFile(FixAcidKeyIndex.java:147)
> at
> org.apache.hadoop.hive.ql.io.orc.FixAcidKeyIndex.checkFiles(FixAcidKeyIndex.java:130)
> at
> org.apache.hadoop.hive.ql.io.orc.FixAcidKeyIndex.main(FixAcidKeyIndex.java:106)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:308)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:222)
> {noformat}
> The aim of this ticket is to handle such case in order to support
> re-generating this metadata entry even when it is missing.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HIVE-26147) OrcRawRecordMerger throws NPE when hive.acid.key.index is missing for an acid file
[
https://issues.apache.org/jira/browse/HIVE-26147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17524372#comment-17524372
]
Karen Coppage commented on HIVE-26147:
--
Committed to master branch. Thanks for your contribution [~asolimando] !
> OrcRawRecordMerger throws NPE when hive.acid.key.index is missing for an acid
> file
> --
>
> Key: HIVE-26147
> URL: https://issues.apache.org/jira/browse/HIVE-26147
> Project: Hive
> Issue Type: Bug
> Components: ORC, Transactions
>Affects Versions: 4.0.0-alpha-2
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When _hive.acid.key.index_ is missing for an acid ORC file
> _OrcRawRecordMerger_ throws as follows:
> {noformat}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.discoverKeyBounds(OrcRawRecordMerger.java:795)
> ~[hive-exec-4.0.0-alpha-2-SNAPS
> HOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:1053)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.
> 0.0-alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:2096)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-a
> lpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1991)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4
> .0.0-alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:769)
> ~[hive-exec-4.0.0-alpha
> -2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:335)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-
> alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:560)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha
> -2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:529)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-
> SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:150)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.Driver.getFetchingTableResults(Driver.java:719)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNA
> PSHOT]
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:671)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:233)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha
> -2-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:489)
> ~[hive-service-4.0.0-alpha-2-SNAPSHOT.jar:
> 4.0.0-alpha-2-SNAPSHOT]
> ... 24 more
> {noformat}
> For this situation to happen, the ORC file must have more than one stripe,
> and the offset of the element to seek should either locate it beyond the
> first stripe (but before the last one), or in the first one if not the last
> one, as the code shows:
> {code:java}
> if (firstStripe != 0) {
> minKey = keyIndex[firstStripe - 1];
> }
> if (!isTail) {
> maxKey = keyIndex[firstStripe + stripeCount - 1];
> }
> {code}
> However, in the context of the detection of the original issue, the NPE was
> triggered even by a simple "select *" over a table with ORC files missing the
> _hive.acid.key.index_ metadata information, but it was never failing for ORC
> files with a single stripe. The file was generated after a major compaction
> of acid and non-acid data.
> If the "select *" is not triggering the NPE, either pick the values of the
> row obtained with "select * from $table limit 1", or try to select based on
> different values trying to get into the sought situation with a filter like
> this:
> {code:sql}
> select * from $table where c = $value
> {code}
> _OrcRawRecordMerger_ should simply leave as "null" the min and max keys when
> the _hive.acid.key.index_ metadata is missing.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Resolved] (HIVE-26116) Fix handling of compaction requests originating from aborted dynamic partition queries in Initiator
[ https://issues.apache.org/jira/browse/HIVE-26116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-26116. -- Fix Version/s: 4.0.0 Resolution: Fixed Committed to master branch. Thanks for your contribution [~veghlaci05] ! > Fix handling of compaction requests originating from aborted dynamic > partition queries in Initiator > --- > > Key: HIVE-26116 > URL: https://issues.apache.org/jira/browse/HIVE-26116 > Project: Hive > Issue Type: Bug >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Compaction requests originated from an abort of a dynamic partition insert > can cause a NPE in Initiator. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-25943) Introduce compaction cleaner failed attempts threshold
[ https://issues.apache.org/jira/browse/HIVE-25943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-25943. -- Fix Version/s: 4.0.0 Resolution: Fixed Committed to master branch. Thanks for your contribution [~veghlaci05]! > Introduce compaction cleaner failed attempts threshold > -- > > Key: HIVE-25943 > URL: https://issues.apache.org/jira/browse/HIVE-25943 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 6h > Remaining Estimate: 0h > > If the cleaner fails for some reason, the compaction entity status remains in > "ready for cleaning", therefore the cleaner will pick up this entity > resulting in an endless try. The number of failed cleaning attempts should be > counted and if they reach a certain threshold the cleaner must skip all the > cleaning attempts on that compaction entity. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-25645) Query-based compaction doesn't work when partition column type is boolean
[ https://issues.apache.org/jira/browse/HIVE-25645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-25645. -- Fix Version/s: 4.0.0 Resolution: Fixed Committed to master branch. Thanks for your contribution [~veghlaci05]! > Query-based compaction doesn't work when partition column type is boolean > - > > Key: HIVE-25645 > URL: https://issues.apache.org/jira/browse/HIVE-25645 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-26010) Compactions not picked up by cleaner if CQ_COMMIT_TIME is null and delayed cleaning is enabled
[ https://issues.apache.org/jira/browse/HIVE-26010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503382#comment-17503382 ] Karen Coppage commented on HIVE-26010: -- Committed to master branch. Thanks for your contribution [~veghlaci05]! > Compactions not picked up by cleaner if CQ_COMMIT_TIME is null and delayed > cleaning is enabled > -- > > Key: HIVE-26010 > URL: https://issues.apache.org/jira/browse/HIVE-26010 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Compactions are not picked up by the cleaner if the CQ_COMMIT_TIME is null > and the delated cleaning is enabled. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-26010) Compactions not picked up by cleaner if CQ_COMMIT_TIME is null and delayed cleaning is enabled
[ https://issues.apache.org/jira/browse/HIVE-26010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-26010. -- Fix Version/s: 4.0.0 Resolution: Fixed > Compactions not picked up by cleaner if CQ_COMMIT_TIME is null and delayed > cleaning is enabled > -- > > Key: HIVE-26010 > URL: https://issues.apache.org/jira/browse/HIVE-26010 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Compactions are not picked up by the cleaner if the CQ_COMMIT_TIME is null > and the delated cleaning is enabled. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-25959) Expose Compaction Observability delta metrics using the JsonReporter
[ https://issues.apache.org/jira/browse/HIVE-25959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-25959. -- Resolution: Fixed Committed to master branch. Thanks [~vcsomor] for your contribution! > Expose Compaction Observability delta metrics using the JsonReporter > > > Key: HIVE-25959 > URL: https://issues.apache.org/jira/browse/HIVE-25959 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 4.0.0 >Reporter: Viktor Csomor >Assignee: Viktor Csomor >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > As part of the HIVE-25897 the Delta metrics collection has been moved from > HiveServer2 to HMS. Hence the HMS does not have any jmx endpoint exposed by > default we'd like to expose these metrics through the JsonReporter -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-25926) Move all logging from AcidMetricService to AcidMetricLogger
[ https://issues.apache.org/jira/browse/HIVE-25926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-25926. -- Resolution: Fixed Committed to master branch. Thanks for your contribution [~vcsomor]! > Move all logging from AcidMetricService to AcidMetricLogger > --- > > Key: HIVE-25926 > URL: https://issues.apache.org/jira/browse/HIVE-25926 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 4.0.0 >Reporter: Viktor Csomor >Assignee: Viktor Csomor >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Some logging is left over in AcidMetricService (which used to both log > errors/warnings and emit metrics, and now just emits metrics). Logging should > be in the AcidMetricLogger as much as possible. Stragglers left over in > AcidMetricService: > * compaction_oldest_enqueue_age_in_sec > * COMPACTOR_FAILED_COMPACTION_RATIO_THRESHOLD logging > * Multiple Compaction Worker versions (in method: > detectMultipleWorkerVersions) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-25746) Compaction Failure Counter counted incorrectly
[
https://issues.apache.org/jira/browse/HIVE-25746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage resolved HIVE-25746.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Committed to master branch. Thanks for your contribution [~vcsomor]!
> Compaction Failure Counter counted incorrectly
> --
>
> Key: HIVE-25746
> URL: https://issues.apache.org/jira/browse/HIVE-25746
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 4.0.0
>Reporter: Viktor Csomor
>Assignee: Viktor Csomor
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> The count of the below metrics counted incorrectly upon an exception.
> - {{compaction_initator_failure_counter}}
> - {{compaction_cleaner_failure_counter}}
> Reasoning:
> In the {{Initator}}/{{Cleaner}} class creates a list of {{CompletableFuture}}
> which {{Runnable}} core exception is being wrapped to {{RuntimeExceptions}}.
> The below code-snippet waits all cleaners to complete (Initiators does it
> similarly).
> {code:java}
> try {
>
> for (CompactionInfo compactionInfo : readyToClean) {
>
> cleanerList.add(CompletableFuture.runAsync(CompactorUtil.ThrowingRunnable.unchecked(()
> ->
> clean(compactionInfo, cleanerWaterMark,
> metricsEnabled)), cleanerExecutor));
> }
> CompletableFuture.allOf(cleanerList.toArray(new
> CompletableFuture[0])).join();
> }
> } catch (Throwable t) {
> // the lock timeout on AUX lock, should be ignored.
> if (metricsEnabled && handle != null) {
> failuresCounter.inc();
> }
> {code}
> If the {{CompleteableFututre#join}} throws an Exception then the failure
> counter is incremented.
> Docs:
> {code}
> /**
> * Returns the result value when complete, or throws an
> * (unchecked) exception if completed exceptionally. To better
> * conform with the use of common functional forms, if a
> * computation involved in the completion of this
> * CompletableFuture threw an exception, this method throws an
> * (unchecked) {@link CompletionException} with the underlying
> * exception as its cause.
> *
> * @return the result value
> * @throws CancellationException if the computation was cancelled
> * @throws CompletionException if this future completed
> * exceptionally or a completion computation threw an exception
> */
> public T join() {
> Object r;
> return reportJoin((r = result) == null ? waitingGet(false) : r);
> }
> {code}
> (!) Let's suppose we have 10 cleaners and the 2nd throws an exception. The
> {{catch}} block will be initiated and the {{failuresCounter}} will be
> incremented. If there is any consecutive error amongst the remaining cleaners
> the counter won't be incremented.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HIVE-25914) Cleaner updates Initiator cycle metric
[ https://issues.apache.org/jira/browse/HIVE-25914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-25914. -- Fix Version/s: 4.0.0 Resolution: Fixed Committed to master branch. Thank you for your contribution [~vcsomor]! > Cleaner updates Initiator cycle metric > -- > > Key: HIVE-25914 > URL: https://issues.apache.org/jira/browse/HIVE-25914 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 4.0.0 >Reporter: Viktor Csomor >Assignee: Viktor Csomor >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > The Cleaner updates the Initiator metric. > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java#L175 > Create some test for Initiator and Cleaner to avoid this happen. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25550) Increase the RM_PROGRESS column max length to fit metrics stat
[
https://issues.apache.org/jira/browse/HIVE-25550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464384#comment-17464384
]
Karen Coppage commented on HIVE-25550:
--
[~ayushtkn] understood, thanks for explaining!
> Increase the RM_PROGRESS column max length to fit metrics stat
> --
>
> Key: HIVE-25550
> URL: https://issues.apache.org/jira/browse/HIVE-25550
> Project: Hive
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Presently it fails with the following trace:
> {noformat}
> [[Event Name: EVENT_ALLOC_WRITE_ID; Total Number: 213; Total Time: 85347.0;
> Mean: 400.6901408450704; Median: 392.0; Standard Deviation:
> 33.99178239314741; Variance: 1155.4412702630862; Kurtosis: 83.69411620601193;
> Skewness: 83.69411620601193; 25th Percentile: 384.0; 50th Percentile: 392.0;
> 75th Percentile: 408.0; 90th Percentile: 417.0; Top 5 EventIds(EventId=Time)
> {1498476=791, 1498872=533, 1497805=508, 1498808=500, 1499027=492};]]}"}]}" in
> column ""RM_PROGRESS"" that has maximum length of 4000. Please correct your
> data!
> at
> org.datanucleus.store.rdbms.mapping.datastore.CharRDBMSMapping.setString(CharRDBMSMapping.java:254)
> ~[datanucleus-rdbms-4.1.19.jar:?]
> at
> org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.setString(SingleFieldMapping.java:180)
> ~{noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-25550) Increase the RM_PROGRESS column max length to fit metrics stat
[
https://issues.apache.org/jira/browse/HIVE-25550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463813#comment-17463813
]
Karen Coppage commented on HIVE-25550:
--
[~ayushtkn] , I'm a bit confused about why the schema changes were skipped for
Oracle and MsSQL for this commit, is there a specific reason for this?
> Increase the RM_PROGRESS column max length to fit metrics stat
> --
>
> Key: HIVE-25550
> URL: https://issues.apache.org/jira/browse/HIVE-25550
> Project: Hive
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Presently it fails with the following trace:
> {noformat}
> [[Event Name: EVENT_ALLOC_WRITE_ID; Total Number: 213; Total Time: 85347.0;
> Mean: 400.6901408450704; Median: 392.0; Standard Deviation:
> 33.99178239314741; Variance: 1155.4412702630862; Kurtosis: 83.69411620601193;
> Skewness: 83.69411620601193; 25th Percentile: 384.0; 50th Percentile: 392.0;
> 75th Percentile: 408.0; 90th Percentile: 417.0; Top 5 EventIds(EventId=Time)
> {1498476=791, 1498872=533, 1497805=508, 1498808=500, 1499027=492};]]}"}]}" in
> column ""RM_PROGRESS"" that has maximum length of 4000. Please correct your
> data!
> at
> org.datanucleus.store.rdbms.mapping.datastore.CharRDBMSMapping.setString(CharRDBMSMapping.java:254)
> ~[datanucleus-rdbms-4.1.19.jar:?]
> at
> org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.setString(SingleFieldMapping.java:180)
> ~{noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HIVE-25737) Compaction Observability: Initiator/Worker/Cleaner cycle measurement improvements
[
https://issues.apache.org/jira/browse/HIVE-25737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage resolved HIVE-25737.
--
Resolution: Fixed
Committed to master branch. Thanks for your contribution [~vcsomor]!
> Compaction Observability: Initiator/Worker/Cleaner cycle measurement
> improvements
> -
>
> Key: HIVE-25737
> URL: https://issues.apache.org/jira/browse/HIVE-25737
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Viktor Csomor
>Assignee: Viktor Csomor
>Priority: Major
>
> In the Compaction Observability the Initiator/Worker/Cleaner cycle is
> measured with a [Dropwizard
> Timer|https://metrics.dropwizard.io/4.2.0/getting-started.html] metrics.
> {noformat}
> Timers
> A timer measures both the rate that a particular piece of code is called and
> the distribution of its duration.
> {noformat}
> However this is not good to measure simply a duration. Furthermore, one HMS
> can run multiple Worker threads and the duration of the last finished worker
> is not really informative if a Worker thread got stuck.
> Timers do not carry enough information because they only bump the counter if
> a Worker has finished a loop.
> If Initiator/Worker/Cleaner gets stuck, then the metrics is not provided
> hence it didn't bump the counter.
> It'd better to implement the followings:
> - Time passed since Initiator start (single threaded) -> Gauge metric
> - Oldest Working compaction -> Gauge Metric
> - Oldest Working Cleaner -> Gauge metric
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25716) Fix flaky test TestCompactionMetrics#testOldestReadyForCleaningAge
[
https://issues.apache.org/jira/browse/HIVE-25716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage updated HIVE-25716:
-
Labels: flaky-test (was: )
> Fix flaky test TestCompactionMetrics#testOldestReadyForCleaningAge
> --
>
> Key: HIVE-25716
> URL: https://issues.apache.org/jira/browse/HIVE-25716
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Karen Coppage
>Priority: Major
> Labels: flaky-test
>
> Flaky check failed on run #59:
> [http://ci.hive.apache.org/job/hive-flaky-check/467/|http://ci.hive.apache.org/job/hive-flaky-check/467/]
> {code:java}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at
> org.apache.hadoop.hive.ql.txn.compactor.TestCompactionMetrics.testOldestReadyForCleaningAge(TestCompactionMetrics.java:214)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-25662) JDBC driver returns " java.lang.IllegalArgumentException: Unrecognized type name: timestamp with local time zone('Europe/Madrid') in the table's introspection
[
https://issues.apache.org/jira/browse/HIVE-25662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17441211#comment-17441211
]
Karen Coppage commented on HIVE-25662:
--
Are you using Hive CLI by any chance?
> JDBC driver returns " java.lang.IllegalArgumentException: Unrecognized type
> name: timestamp with local time zone('Europe/Madrid') in the table's
> introspection
> --
>
> Key: HIVE-25662
> URL: https://issues.apache.org/jira/browse/HIVE-25662
> Project: Hive
> Issue Type: Bug
> Components: Hive, JDBC, SQL, Types
>Affects Versions: 3.0.0, 3.1.1, 3.1.2
>Reporter: Inmaculada Domínguez Mira
>Priority: Blocker
> Labels: hive3, localtimezone, timestamp
>
> I have created in Hive 3.1.1 the following table:
>
> CREATE TABLE `hive.timestamps_hive3`(
> `localdate_field` date,
> `timestamp_field` timestamp,
> `timestamptz_field` timestamp with local time zone
> )
> Then I hve inserted one row:
>
> insert into `hive.timestamps_hive3` values
> (DATE '2017-03-26',TIMESTAMP '2017-03-26 01:01:02.345000','2017-03-25
> 17:01:02.345-07:00')
>
> Both of these sentences have gone ok but when I try to get the columns from
> the created table, the following error appears:
>
> SQL Error: java.lang.IllegalArgumentException: Unrecognized type name:
> timestamp with local time zone('Europe/Madrid')
> I have used some JDBC clients to connect to this schema/table with the same
> error therefore I understand the problem is in the JDBC driver.
>
> Some help woulb be very appreciate because it is blocking my project.
>
> REgards and thanks in advance
> Inma
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-25578) Tests are failing because operators can't be closed
[ https://issues.apache.org/jira/browse/HIVE-25578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17440972#comment-17440972 ] Karen Coppage commented on HIVE-25578: -- [~kgyrtkirk] what was the outcome of this? > Tests are failing because operators can't be closed > --- > > Key: HIVE-25578 > URL: https://issues.apache.org/jira/browse/HIVE-25578 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Priority: Critical > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The following qtests are failing consistently > ([example|http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-2667/6/tests/]) > on the master branch: > * TestMiniLlapCliDriver > ([http://ci.hive.apache.org/job/hive-flaky-check/420/]) > ** newline > ** groupby_bigdata > ** input20 > ** input33 > ** rcfile_bigdata > ** remote_script > * TestContribCliDriver > ([http://ci.hive.apache.org/job/hive-flaky-check/421/]) > ** serde_typedbytes5 > The failure reason for all seems to be that operators can't be closed. Not > 100% sure that TestContribCliDriver#serde_typedbytes5 failure is related to > the others – the issue seems to be the same, the error message is a bit > different. > I'm about to disable these as they are blocking all work. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25672) Hive isn't purging older compaction entries from show compaction command
[
https://issues.apache.org/jira/browse/HIVE-25672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17440396#comment-17440396
]
Karen Coppage commented on HIVE-25672:
--
Make sure the AcidHouseKeeperService is running:
hive.metastore.housekeeping.threads.on=true.
Does this fix the issue?
> Hive isn't purging older compaction entries from show compaction command
>
>
> Key: HIVE-25672
> URL: https://issues.apache.org/jira/browse/HIVE-25672
> Project: Hive
> Issue Type: Bug
> Components: Hive, Metastore, Transactions
>Affects Versions: 3.1.1
>Reporter: Rohan Nimmagadda
>Priority: Minor
>
> Added below properties in hive-site, but it's not enforced to auto purging.
> When we run show compaction command it takes forever and returns billions of
> rows.
> Result of show compactions command :
> {code:java}
> 752,450 rows selected (198.066 seconds)
> {code}
> {code:java}
> hive.compactor.history.retention.succeeded": "10",
> "hive.compactor.history.retention.failed": "10",
> "hive.compactor.history.retention.attempted": "10",
> "hive.compactor.history.reaper.interval": "10m" {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HIVE-25060) Hive Compactor doesn´t launch cleaner
[ https://issues.apache.org/jira/browse/HIVE-25060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-25060. -- Assignee: Fran Gonzalez Resolution: Abandoned Due to lack of information we cannot proceed. As it's been more than 6 months I will mark this issue abandoned. > Hive Compactor doesn´t launch cleaner > - > > Key: HIVE-25060 > URL: https://issues.apache.org/jira/browse/HIVE-25060 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.1 > Environment: Hive 3.1.0 > Hadoop 3.1.1 >Reporter: Fran Gonzalez >Assignee: Fran Gonzalez >Priority: Major > > Hello, > there are problems with Hive Compactor. We can see in hivemetastore.log this > message "Max block location exceeded for split" and it´s appearing more and > more times. > After that, the "compactor.Cleaner" is not launched. > We observed that after a Hive Metastore restart, the "compactor.Cleaner" has > not been launched nevermore, but logs doesn´t display any message about it. > Could be a degradation of the Hive Compactor when delta files are growing in > the partitions? > Regards. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25549) Wrong results for window function with expression in PARTITION BY or ORDER BY clause
[
https://issues.apache.org/jira/browse/HIVE-25549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage resolved HIVE-25549.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Committed to master branch. Thanks for the feedback [~abstractdog] and for the
review [~szita]!
> Wrong results for window function with expression in PARTITION BY or ORDER BY
> clause
>
>
> Key: HIVE-25549
> URL: https://issues.apache.org/jira/browse/HIVE-25549
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Sometimes the partition in a vectorized PTF needs some sort of
> transformation. For these to work the partition expression may need some
> transient variables initialized.
> Example with row_number:
> {code:java}
> create table test_rownumber (a string, b string) stored as orc;
> insert into test_rownumber values
> ('1', 'a'),
> ('2', 'b'),
> ('3', 'c'),
> ('4', 'd'),
> ('5', 'e');
> CREATE VIEW `test_rownumber_vue` AS SELECT `test_rownumber`.`a` AS
> `a`,CAST(`test_rownumber`.`a` as INT) AS `a_int`,
> `test_rownumber`.`b` as `b` from `default`.`test_rownumber`;
> set hive.vectorized.execution.enabled=true;
> select *, row_number() over(partition by a_int order by b) from
> test_rownumber_vue;
> {code}
> Output is:
> {code:java}
> +---+---+---+--+
> | test_rownumber_vue.a | test_rownumber_vue.a_int | test_rownumber_vue.b |
> row_number_window_0 |
> +---+---+---+--+
> | 1 | 1 | a |
> 1|
> | 2 | 2 | b |
> 2|
> | 3 | 3 | c |
> 3|
> | 4 | 4 | d |
> 4|
> | 5 | 5 | e |
> 5|
> +---+---+---+--+
> {code}
> But it should be this, because we restart the row numbering for each
> partition:
> {code:java}
> +---+---+---+--+
> | test_rownumber_vue.a | test_rownumber_vue.a_int | test_rownumber_vue.b |
> row_number_window_0 |
> +---+---+---+--+
> | 1 | 1 | a |
> 1|
> | 2 | 2 | b |
> 1|
> | 3 | 3 | c |
> 1|
> | 4 | 4 | d |
> 1|
> | 5 | 5 | e |
> 1|
> +---+---+---+--+
> {code}
> Explanation:
> CastStringToLong has to be executed on the partition column (a_int). Because
> CastStringToLong.integerPrimitiveCategory is not initialized, all output of
> CastStringToLong is null - so a_int is interpreted as containing null values
> only and partitioning is ignored.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-25578) Tests are failing because operators can't be closed
[ https://issues.apache.org/jira/browse/HIVE-25578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17422176#comment-17422176 ] Karen Coppage commented on HIVE-25578: -- Okay I reverted... but still not sure how to move forward, as you mentioned that TestMiniLlapCliDriver#*[newline] passed for you locally > Tests are failing because operators can't be closed > --- > > Key: HIVE-25578 > URL: https://issues.apache.org/jira/browse/HIVE-25578 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Priority: Critical > > The following qtests are failing consistently > ([example|http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-2667/6/tests/]) > on the master branch: > * TestMiniLlapCliDriver > ([http://ci.hive.apache.org/job/hive-flaky-check/420/]) > ** newline > ** groupby_bigdata > ** input20 > ** input33 > ** rcfile_bigdata > ** remote_script > * TestContribCliDriver > ([http://ci.hive.apache.org/job/hive-flaky-check/421/]) > ** serde_typedbytes5 > The failure reason for all seems to be that operators can't be closed. Not > 100% sure that TestContribCliDriver#serde_typedbytes5 failure is related to > the others – the issue seems to be the same, the error message is a bit > different. > I'm about to disable these as they are blocking all work. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25549) Wrong results for window function with expression in PARTITION BY or ORDER BY clause
[
https://issues.apache.org/jira/browse/HIVE-25549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage updated HIVE-25549:
-
Summary: Wrong results for window function with expression in PARTITION BY
or ORDER BY clause (was: Wrong results for window function with CAST
expression in PARTITION BY clause)
> Wrong results for window function with expression in PARTITION BY or ORDER BY
> clause
>
>
> Key: HIVE-25549
> URL: https://issues.apache.org/jira/browse/HIVE-25549
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Sometimes the partition in a vectorized PTF needs some sort of
> transformation. For these to work the partition expression may need some
> transient variables initialized.
> Example with row_number:
> {code:java}
> create table test_rownumber (a string, b string) stored as orc;
> insert into test_rownumber values
> ('1', 'a'),
> ('2', 'b'),
> ('3', 'c'),
> ('4', 'd'),
> ('5', 'e');
> CREATE VIEW `test_rownumber_vue` AS SELECT `test_rownumber`.`a` AS
> `a`,CAST(`test_rownumber`.`a` as INT) AS `a_int`,
> `test_rownumber`.`b` as `b` from `default`.`test_rownumber`;
> set hive.vectorized.execution.enabled=true;
> select *, row_number() over(partition by a_int order by b) from
> test_rownumber_vue;
> {code}
> Output is:
> {code:java}
> +---+---+---+--+
> | test_rownumber_vue.a | test_rownumber_vue.a_int | test_rownumber_vue.b |
> row_number_window_0 |
> +---+---+---+--+
> | 1 | 1 | a |
> 1|
> | 2 | 2 | b |
> 2|
> | 3 | 3 | c |
> 3|
> | 4 | 4 | d |
> 4|
> | 5 | 5 | e |
> 5|
> +---+---+---+--+
> {code}
> But it should be this, because we restart the row numbering for each
> partition:
> {code:java}
> +---+---+---+--+
> | test_rownumber_vue.a | test_rownumber_vue.a_int | test_rownumber_vue.b |
> row_number_window_0 |
> +---+---+---+--+
> | 1 | 1 | a |
> 1|
> | 2 | 2 | b |
> 1|
> | 3 | 3 | c |
> 1|
> | 4 | 4 | d |
> 1|
> | 5 | 5 | e |
> 1|
> +---+---+---+--+
> {code}
> Explanation:
> CastStringToLong has to be executed on the partition column (a_int). Because
> CastStringToLong.integerPrimitiveCategory is not initialized, all output of
> CastStringToLong is null - so a_int is interpreted as containing null values
> only and partitioning is ignored.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-25549) Need to transient init partition expressions in vectorized PTFs
[
https://issues.apache.org/jira/browse/HIVE-25549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17419289#comment-17419289
]
Karen Coppage commented on HIVE-25549:
--
Could also result in an NPE:
{code:java}
create table row_number_test as select (posexplode(split(repeat("w,", 2),
","))) as (pos, col);
select row_number() over(partition by cast (pos as string)) from
row_number_test;{code}
results in
{code:java}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing vector batch (tag=0) (vectorizedVertexNum 1)
...
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.vector.expressions.MathExpr.writeLongToUTF8(MathExpr.java:136)
at
org.apache.hadoop.hive.ql.exec.vector.expressions.CastLongToString.func(CastLongToString.java:53)
at
org.apache.hadoop.hive.ql.exec.vector.expressions.LongToStringUnaryUDF.evaluate(LongToStringUnaryUDF.java:111)
at
org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFOperator.process(VectorPTFOperator.java:381)
{code}
because the orderExpressions don't get a transient init either.
(Now I don't quite understand why there are orderExpressions when the function
has no order by clause, but that's beside the point.)
> Need to transient init partition expressions in vectorized PTFs
> ---
>
> Key: HIVE-25549
> URL: https://issues.apache.org/jira/browse/HIVE-25549
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Sometimes the partition in a vectorized PTF needs some sort of
> transformation. For these to work the partition expression may need some
> transient variables initialized.
> Example with row_number:
> {code:java}
> create table test_rownumber (a string, b string) stored as orc;
> insert into test_rownumber values
> ('1', 'a'),
> ('2', 'b'),
> ('3', 'c'),
> ('4', 'd'),
> ('5', 'e');
> CREATE VIEW `test_rownumber_vue` AS SELECT `test_rownumber`.`a` AS
> `a`,CAST(`test_rownumber`.`a` as INT) AS `a_int`,
> `test_rownumber`.`b` as `b` from `default`.`test_rownumber`;
> set hive.vectorized.execution.enabled=true;
> select *, row_number() over(partition by a_int order by b) from
> test_rownumber_vue;
> {code}
> Output is:
> {code:java}
> +---+---+---+--+
> | test_rownumber_vue.a | test_rownumber_vue.a_int | test_rownumber_vue.b |
> row_number_window_0 |
> +---+---+---+--+
> | 1 | 1 | a |
> 1|
> | 2 | 2 | b |
> 2|
> | 3 | 3 | c |
> 3|
> | 4 | 4 | d |
> 4|
> | 5 | 5 | e |
> 5|
> +---+---+---+--+
> {code}
> But it should be this, because we restart the row numbering for each
> partition:
> {code:java}
> +---+---+---+--+
> | test_rownumber_vue.a | test_rownumber_vue.a_int | test_rownumber_vue.b |
> row_number_window_0 |
> +---+---+---+--+
> | 1 | 1 | a |
> 1|
> | 2 | 2 | b |
> 1|
> | 3 | 3 | c |
> 1|
> | 4 | 4 | d |
> 1|
> | 5 | 5 | e |
> 1|
> +---+---+---+--+
> {code}
> Explanation:
> CastStringToLong has to be executed on the partition column (a_int). Because
> CastStringToLong.integerPrimitiveCategory is not initialized, all output of
> CastStringToLong is null - so a_int is interpreted as containing null values
> only and partitioning is ignored.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-25549) Need to transient init partition expressions in vectorized PTFs
[
https://issues.apache.org/jira/browse/HIVE-25549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage updated HIVE-25549:
-
Description:
Sometimes the partition in a vectorized PTF needs some sort of transformation.
For these to work the partition expression may need some transient variables
initialized.
Example with row_number:
{code:java}
create table test_rownumber (a string, b string) stored as orc;
insert into test_rownumber values
('1', 'a'),
('2', 'b'),
('3', 'c'),
('4', 'd'),
('5', 'e');
CREATE VIEW `test_rownumber_vue` AS SELECT `test_rownumber`.`a` AS
`a`,CAST(`test_rownumber`.`a` as INT) AS `a_int`,
`test_rownumber`.`b` as `b` from `default`.`test_rownumber`;
set hive.vectorized.execution.enabled=true;
select *, row_number() over(partition by a_int order by b) from
test_rownumber_vue;
{code}
Output is:
{code:java}
+---+---+---+--+
| test_rownumber_vue.a | test_rownumber_vue.a_int | test_rownumber_vue.b |
row_number_window_0 |
+---+---+---+--+
| 1 | 1 | a | 1
|
| 2 | 2 | b | 2
|
| 3 | 3 | c | 3
|
| 4 | 4 | d | 4
|
| 5 | 5 | e | 5
|
+---+---+---+--+
{code}
But it should be this, because we restart the row numbering for each partition:
{code:java}
+---+---+---+--+
| test_rownumber_vue.a | test_rownumber_vue.a_int | test_rownumber_vue.b |
row_number_window_0 |
+---+---+---+--+
| 1 | 1 | a | 1
|
| 2 | 2 | b | 1
|
| 3 | 3 | c | 1
|
| 4 | 4 | d | 1
|
| 5 | 5 | e | 1
|
+---+---+---+--+
{code}
Explanation:
CastStringToLong has to be executed on the partition column (a_int). Because
CastStringToLong.integerPrimitiveCategory is not initialized, all output of
CastStringToLong is null - so a_int is interpreted as containing null values
only and partitioning is ignored.
was:
Sometimes the partition in a vectorized PTF needs some sort of transformation
(in the example below: CastStringToLong). For these to work the partition
expression may need some transient varia
Example with row_number:
{code:java}
create table test_rownumber (a string, b string) stored as orc;
insert into test_rownumber values
('1', 'a'),
('2', 'b'),
('3', 'c'),
('4', 'd'),
('5', 'e');
CREATE VIEW `test_rownumber_vue` AS SELECT `test_rownumber`.`a` AS
`a`,CAST(`test_rownumber`.`a` as INT) AS `a_int`,
`test_rownumber`.`b` as `b` from `default`.`test_rownumber`;
set hive.vectorized.execution.enabled=true;
select *, row_number() over(partition by a_int order by b) from
test_rownumber_vue;
{code}
Output is:
{code:java}
+---+---+---+--+
| test_rownumber_vue.a | test_rownumber_vue.a_int | test_rownumber_vue.b |
row_number_window_0 |
+---+---+---+--+
| 1 | 1 | a | 1
|
| 2 | 2 | b | 2
|
| 3 | 3 | c | 3
|
| 4 | 4 | d | 4
|
| 5 | 5 | e | 5
|
+---+---+---+--+
{code}
But it should be this, because we restart the row numbering for each partition:
{code:java}
+---+---+---+--+
[jira] [Updated] (HIVE-25549) Need to transient init partition expressions in vectorized PTFs
[
https://issues.apache.org/jira/browse/HIVE-25549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage updated HIVE-25549:
-
Description:
{code:java}
set hive.vectorized.execution.enabled=true;
create table test_rownumber (a string, b string) stored as orc;
insert into test_rownumber values
('1', 'a'),
('2', 'b'),
('3', 'c'),
('4', 'd'),
('5', 'e');
CREATE VIEW `test_rownumber_vue` AS SELECT `test_rownumber`.`a` AS
`a`,CAST(`test_rownumber`.`a` as INT) AS `a_int`,
`test_rownumber`.`b` as `b` from `default`.`test_rownumber`;
set hive.vectorized.execution.enabled=true;
select *, row_number() over(partition by a_int order by b) from
test_rownumber_vue;
{code}
Output is:
{code:java}
+---+---+---+--+
| test_rownumber_vue.a | test_rownumber_vue.a_int | test_rownumber_vue.b |
row_number_window_0 |
+---+---+---+--+
| 1 | 1 | a | 1
|
| 2 | 2 | b | 2
|
| 3 | 3 | c | 3
|
| 4 | 4 | d | 4
|
| 5 | 5 | e | 5
|
+---+---+---+--+{code}
But it should be this, because we should restart row numbering for each
partition:
{code:java}
+---+---+---+--+
| test_rownumber_vue.a | test_rownumber_vue.a_int | test_rownumber_vue.b |
row_number_window_0 |
+---+---+---+--+
| 1 | 1 | a | 1
|
| 2 | 2 | b | 1
|
| 3 | 3 | c | 1
|
| 4 | 4 | d | 1
|
| 5 | 5 | e | 1
|
+---+---+---+--+
{code}
> Need to transient init partition expressions in vectorized PTFs
> ---
>
> Key: HIVE-25549
> URL: https://issues.apache.org/jira/browse/HIVE-25549
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Sometimes the partition in a vectorized PTF needs some sort of transformation
> (in the example below: CastStringToLong). For these to work the partition
> expression may need some transient varia
> Example with row_number:
> {code:java}
> create table test_rownumber (a string, b string) stored as orc;
> insert into test_rownumber values
> ('1', 'a'),
> ('2', 'b'),
> ('3', 'c'),
> ('4', 'd'),
> ('5', 'e');
> CREATE VIEW `test_rownumber_vue` AS SELECT `test_rownumber`.`a` AS
> `a`,CAST(`test_rownumber`.`a` as INT) AS `a_int`,
> `test_rownumber`.`b` as `b` from `default`.`test_rownumber`;
> set hive.vectorized.execution.enabled=true;
> select *, row_number() over(partition by a_int order by b) from
> test_rownumber_vue;
> {code}
> Output is:
> {code:java}
> +---+---+---+--+
> | test_rownumber_vue.a | test_rownumber_vue.a_int | test_rownumber_vue.b |
> row_number_window_0 |
> +---+---+---+--+
> | 1 | 1 | a |
> 1|
> | 2 | 2 | b |
> 2|
> | 3 | 3 | c |
> 3|
> | 4 | 4 | d |
> 4|
> | 5 | 5 | e |
> 5|
> +---+---+---+--+
> {code}
> But it should be this, because we restart the row numbering for
[jira] [Updated] (HIVE-25549) Need to transient init partition expressions in vectorized PTFs
[
https://issues.apache.org/jira/browse/HIVE-25549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage updated HIVE-25549:
-
Description:
Sometimes the partition in a vectorized PTF needs some sort of transformation
(in the example below: CastStringToLong). For these to work the partition
expression may need some transient varia
Example with row_number:
{code:java}
create table test_rownumber (a string, b string) stored as orc;
insert into test_rownumber values
('1', 'a'),
('2', 'b'),
('3', 'c'),
('4', 'd'),
('5', 'e');
CREATE VIEW `test_rownumber_vue` AS SELECT `test_rownumber`.`a` AS
`a`,CAST(`test_rownumber`.`a` as INT) AS `a_int`,
`test_rownumber`.`b` as `b` from `default`.`test_rownumber`;
set hive.vectorized.execution.enabled=true;
select *, row_number() over(partition by a_int order by b) from
test_rownumber_vue;
{code}
Output is:
{code:java}
+---+---+---+--+
| test_rownumber_vue.a | test_rownumber_vue.a_int | test_rownumber_vue.b |
row_number_window_0 |
+---+---+---+--+
| 1 | 1 | a | 1
|
| 2 | 2 | b | 2
|
| 3 | 3 | c | 3
|
| 4 | 4 | d | 4
|
| 5 | 5 | e | 5
|
+---+---+---+--+
{code}
But it should be this, because we restart the row numbering for each partition:
{code:java}
+---+---+---+--+
| test_rownumber_vue.a | test_rownumber_vue.a_int | test_rownumber_vue.b |
row_number_window_0 |
+---+---+---+--+
| 1 | 1 | a | 1
|
| 2 | 2 | b | 1
|
| 3 | 3 | c | 1
|
| 4 | 4 | d | 1
|
| 5 | 5 | e | 1
|
+---+---+---+--+
{code}
> Need to transient init partition expressions in vectorized PTFs
> ---
>
> Key: HIVE-25549
> URL: https://issues.apache.org/jira/browse/HIVE-25549
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Sometimes the partition in a vectorized PTF needs some sort of transformation
> (in the example below: CastStringToLong). For these to work the partition
> expression may need some transient varia
> Example with row_number:
> {code:java}
> create table test_rownumber (a string, b string) stored as orc;
> insert into test_rownumber values
> ('1', 'a'),
> ('2', 'b'),
> ('3', 'c'),
> ('4', 'd'),
> ('5', 'e');
> CREATE VIEW `test_rownumber_vue` AS SELECT `test_rownumber`.`a` AS
> `a`,CAST(`test_rownumber`.`a` as INT) AS `a_int`,
> `test_rownumber`.`b` as `b` from `default`.`test_rownumber`;
> set hive.vectorized.execution.enabled=true;
> select *, row_number() over(partition by a_int order by b) from
> test_rownumber_vue;
> {code}
> Output is:
> {code:java}
> +---+---+---+--+
> | test_rownumber_vue.a | test_rownumber_vue.a_int | test_rownumber_vue.b |
> row_number_window_0 |
> +---+---+---+--+
> | 1 | 1 | a |
> 1|
> | 2 | 2 | b |
> 2|
> | 3 | 3 | c |
> 3|
> | 4 | 4 | d |
> 4|
> | 5 | 5 | e |
> 5|
>
[jira] [Commented] (HIVE-25549) Need to transient init partition expressions in vectorized PTFs
[ https://issues.apache.org/jira/browse/HIVE-25549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17419142#comment-17419142 ] Karen Coppage commented on HIVE-25549: -- [~zabetak] this a WIP right now, just seeing if the tests run. But I will update the description > Need to transient init partition expressions in vectorized PTFs > --- > > Key: HIVE-25549 > URL: https://issues.apache.org/jira/browse/HIVE-25549 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25549) Need to transient init partition expressions in vectorized PTFs
[ https://issues.apache.org/jira/browse/HIVE-25549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reassigned HIVE-25549: > Need to transient init partition expressions in vectorized PTFs > --- > > Key: HIVE-25549 > URL: https://issues.apache.org/jira/browse/HIVE-25549 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23760) Upgrading to Kafka 2.5 Clients
[ https://issues.apache.org/jira/browse/HIVE-23760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-23760: - Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Upgrading to Kafka 2.5 Clients > -- > > Key: HIVE-23760 > URL: https://issues.apache.org/jira/browse/HIVE-23760 > Project: Hive > Issue Type: Improvement > Components: kafka integration >Reporter: Andras Katona >Assignee: Karen Coppage >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 2h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25513) Delta metrics collection may cause NPE
[ https://issues.apache.org/jira/browse/HIVE-25513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-25513: - Parent: HIVE-24824 Issue Type: Sub-task (was: Bug) > Delta metrics collection may cause NPE > -- > > Key: HIVE-25513 > URL: https://issues.apache.org/jira/browse/HIVE-25513 > Project: Hive > Issue Type: Sub-task >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > When collecting metrics about the number of deltas under specific > partitions/tables, information about which partitions/tables are being read > is stored in the Configuration object under key delta.files.metrics.metadata. > This information is retrieved in > DeltaFilesMetricsReporter#mergeDeltaFilesStats when collecting the actual > information about the number of deltas. But if the information was never > stored for some reason, an NPE will be thrown from > DeltaFilesMetricsReporter#mergeDeltaFilesStats. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25513) Delta metrics collection may cause NPE
[ https://issues.apache.org/jira/browse/HIVE-25513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-25513. -- Resolution: Fixed Committed to master branch. Thank you for your review [~lpinter]! > Delta metrics collection may cause NPE > -- > > Key: HIVE-25513 > URL: https://issues.apache.org/jira/browse/HIVE-25513 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > When collecting metrics about the number of deltas under specific > partitions/tables, information about which partitions/tables are being read > is stored in the Configuration object under key delta.files.metrics.metadata. > This information is retrieved in > DeltaFilesMetricsReporter#mergeDeltaFilesStats when collecting the actual > information about the number of deltas. But if the information was never > stored for some reason, an NPE will be thrown from > DeltaFilesMetricsReporter#mergeDeltaFilesStats. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25513) Delta metrics collection may cause NPE
[ https://issues.apache.org/jira/browse/HIVE-25513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reassigned HIVE-25513: > Delta metrics collection may cause NPE > -- > > Key: HIVE-25513 > URL: https://issues.apache.org/jira/browse/HIVE-25513 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > > When collecting metrics about the number of deltas under specific > partitions/tables, information about which partitions/tables are being read > is stored in the Configuration object under key delta.files.metrics.metadata. > This information is retrieved in > DeltaFilesMetricsReporter#mergeDeltaFilesStats when collecting the actual > information about the number of deltas. But if the information was never > stored for some reason, an NPE will be thrown from > DeltaFilesMetricsReporter#mergeDeltaFilesStats. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25450) Delta metrics keys should contain database name
[ https://issues.apache.org/jira/browse/HIVE-25450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-25450. -- Fix Version/s: 4.0.0 Resolution: Fixed Committed to master under PR #2613. Thanks for reviewing [~dkuzmenko]! > Delta metrics keys should contain database name > --- > > Key: HIVE-25450 > URL: https://issues.apache.org/jira/browse/HIVE-25450 > Project: Hive > Issue Type: Sub-task >Reporter: Karen Coppage >Assignee: László Pintér >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Currently metrics about the number of deltas in a given partition or > unpartitioned table include information about the table name and the > partition name (if applicable), but they should also include the database > name, since there could be 2 tables in different databases with the same name. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25430) compactor.Worker.markFailed should catch and log any kind of exception
[ https://issues.apache.org/jira/browse/HIVE-25430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-25430. -- Fix Version/s: 4.0.0 Resolution: Fixed Committed to master branch. Thanks for reviewing [~dkuzmenko]! > compactor.Worker.markFailed should catch and log any kind of exception > -- > > Key: HIVE-25430 > URL: https://issues.apache.org/jira/browse/HIVE-25430 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25429) Delta metrics collection may cause number of tez counters to exceed tez.counters.max limit
[ https://issues.apache.org/jira/browse/HIVE-25429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-25429. -- Fix Version/s: 4.0.0 Resolution: Fixed Committed to master. Thanks [~lpinter] for the +1 and [~dkuzmenko] for the feedback! > Delta metrics collection may cause number of tez counters to exceed > tez.counters.max limit > -- > > Key: HIVE-25429 > URL: https://issues.apache.org/jira/browse/HIVE-25429 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > There's a limit to the number of tez counters allowed (tez.counters.max). > Delta metrics collection (i.e. DeltaFileMetricsReporter) was creating 3 > counters for each partition touched by a given query, which can result in a > huge number of counters, which is unnecessary because we're only interested > in n number of partitions with the most deltas. This change limits the number > of counters created to hive.txn.acid.metrics.max.cache.size*3. > Also when tez.counters.max is reached a LimitExceededException is thrown but > isn't caught on the Hive side and causes the query to fail. We should catch > this and skip delta metrics collection in this case. > Also make sure that metrics are only collected if > hive.metastore.acidmetrics.ext.on=true -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25450) Delta metrics keys should contain database name
[ https://issues.apache.org/jira/browse/HIVE-25450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-25450: - Description: Currently metrics about the number of deltas in a given partition or unpartitioned table include information about the table name and the partition name (if applicable), but they should also include the database name, since there could be 2 tables in different databases with the same name. (was: Currently metrics about the number of deltas in a given partition or unpartitioned table includes information about the table name and the partition name (if applicable), but they should also include the database name, since there could be 2 tables in different databases with the same name.) > Delta metrics keys should contain database name > --- > > Key: HIVE-25450 > URL: https://issues.apache.org/jira/browse/HIVE-25450 > Project: Hive > Issue Type: Sub-task >Reporter: Karen Coppage >Priority: Major > > Currently metrics about the number of deltas in a given partition or > unpartitioned table include information about the table name and the > partition name (if applicable), but they should also include the database > name, since there could be 2 tables in different databases with the same name. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25429) Delta metrics collection may cause number of tez counters to exceed tez.counters.max limit
[ https://issues.apache.org/jira/browse/HIVE-25429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-25429: - Description: There's a limit to the number of tez counters allowed (tez.counters.max). Delta metrics collection (i.e. DeltaFileMetricsReporter) was creating 3 counters for each partition touched by a given query, which can result in a huge number of counters, which is unnecessary because we're only interested in n number of partitions with the most deltas. This change limits the number of counters created to hive.txn.acid.metrics.max.cache.size*3. Also when tez.counters.max is reached a LimitExceededException is thrown but isn't caught on the Hive side and causes the query to fail. We should catch this and skip delta metrics collection in this case. Also make sure that metrics are only collected if hive.metastore.acidmetrics.ext.on=true was: There's a limit to the number of tez counters allowed (tez.counters.max). Delta metrics collection (i.e. DeltaFileMetricsReporter) was creating 3 counters for each partition touched by a given query, which can result in a huge number of counters, which is unnecessary because we're only interested in n number of partitions with the most deltas. This change limits the number of counters created to hive.txn.acid.metrics.max.cache.size*3. Also when tez.counters.max is reached a LimitExceededException is thrown but isn't caught on the Hive side and causes the query to fail. We should catch this and skip delta metrics collection in this case. > Delta metrics collection may cause number of tez counters to exceed > tez.counters.max limit > -- > > Key: HIVE-25429 > URL: https://issues.apache.org/jira/browse/HIVE-25429 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > There's a limit to the number of tez counters allowed (tez.counters.max). > Delta metrics collection (i.e. DeltaFileMetricsReporter) was creating 3 > counters for each partition touched by a given query, which can result in a > huge number of counters, which is unnecessary because we're only interested > in n number of partitions with the most deltas. This change limits the number > of counters created to hive.txn.acid.metrics.max.cache.size*3. > Also when tez.counters.max is reached a LimitExceededException is thrown but > isn't caught on the Hive side and causes the query to fail. We should catch > this and skip delta metrics collection in this case. > Also make sure that metrics are only collected if > hive.metastore.acidmetrics.ext.on=true -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-24235) Drop and recreate table during MR compaction leaves behind base/delta directory
[
https://issues.apache.org/jira/browse/HIVE-24235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17388018#comment-17388018
]
Karen Coppage edited comment on HIVE-24235 at 8/5/21, 12:41 PM:
This is incomplete because if the table is dropped, all related compactions are
cleared from metadata. So here we're trying to fail a compaction that doesn't
exist.
HIVE-24900 and HIVE-25430 are needed to complete this fix.
was (Author: klcopp):
This is useless because if the table is dropped, all related compactions are
cleared from metadata. So here we're trying to fail a compaction that doesn't
exist.
The actual fix for this issue is HIVE-25393.
> Drop and recreate table during MR compaction leaves behind base/delta
> directory
> ---
>
> Key: HIVE-24235
> URL: https://issues.apache.org/jira/browse/HIVE-24235
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> If a table is dropped and recreated during MR compaction, the table directory
> and a base (or delta, if minor compaction) directory could be created, with
> or without data, while the table "does not exist".
> E.g.
> {code:java}
> create table c (i int) stored as orc tblproperties
> ("NO_AUTO_COMPACTION"="true", "transactional"="true");
> insert into c values (9);
> insert into c values (9);
> alter table c compact 'major';
> While compaction job is running: {
> drop table c;
> create table c (i int) stored as orc tblproperties
> ("NO_AUTO_COMPACTION"="true", "transactional"="true");
> }
> {code}
> The table directory should be empty, but table directory could look like this
> after the job is finished:
> {code:java}
> Oct 6 14:23 c/base_002_v101/._orc_acid_version.crc
> Oct 6 14:23 c/base_002_v101/.bucket_0.crc
> Oct 6 14:23 c/base_002_v101/_orc_acid_version
> Oct 6 14:23 c/base_002_v101/bucket_0
> {code}
> or perhaps just:
> {code:java}
> Oct 6 14:23 c/base_002_v101/._orc_acid_version.crc
> Oct 6 14:23 c/base_002_v101/_orc_acid_version
> {code}
> Insert another row and you have:
> {code:java}
> Oct 6 14:33 base_002_v101/
> Oct 6 14:33 base_002_v101/._orc_acid_version.crc
> Oct 6 14:33 base_002_v101/.bucket_0.crc
> Oct 6 14:33 base_002_v101/_orc_acid_version
> Oct 6 14:33 base_002_v101/bucket_0
> Oct 6 14:35 delta_001_001_/._orc_acid_version.crc
> Oct 6 14:35 delta_001_001_/.bucket_0_0.crc
> Oct 6 14:35 delta_001_001_/_orc_acid_version
> Oct 6 14:35 delta_001_001_/bucket_0_0
> {code}
> Selecting from the table will result in this error because the highest valid
> writeId for this table is 1:
> {code:java}
> thrift.ThriftCLIService: Error fetching results:
> org.apache.hive.service.cli.HiveSQLException: Unable to get the next row set
> at
> org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:482)
> ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> ...
> Caused by: java.io.IOException: java.lang.RuntimeException: ORC split
> generation failed with exception: java.io.IOException: Not enough history
> available for (1,x). Oldest available base:
> .../warehouse/b/base_004_v092
> {code}
> Solution: Resolve the table again after compaction is finished; compare the
> id with the table id from when compaction began. If the ids do not match,
> abort the compaction's transaction.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-25430) compactor.Worker.markFailed should catch and log any kind of exception
[ https://issues.apache.org/jira/browse/HIVE-25430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reassigned HIVE-25430: > compactor.Worker.markFailed should catch and log any kind of exception > -- > > Key: HIVE-25430 > URL: https://issues.apache.org/jira/browse/HIVE-25430 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25429) Delta metrics collection may cause number of tez counters to exceed tez.counters.max limit
[ https://issues.apache.org/jira/browse/HIVE-25429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reassigned HIVE-25429: > Delta metrics collection may cause number of tez counters to exceed > tez.counters.max limit > -- > > Key: HIVE-25429 > URL: https://issues.apache.org/jira/browse/HIVE-25429 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > > There's a limit to the number of tez counters allowed (tez.counters.max). > Delta metrics collection (i.e. DeltaFileMetricsReporter) was creating 3 > counters for each partition touched by a given query, which can result in a > huge number of counters, which is unnecessary because we're only interested > in n number of partitions with the most deltas. This change limits the number > of counters created to hive.txn.acid.metrics.max.cache.size*3. > Also when tez.counters.max is reached a LimitExceededException is thrown but > isn't caught on the Hive side and causes the query to fail. We should catch > this and skip delta metrics collection in this case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25390) Metrics compaction_failed_initiator_ratio and compaction_failed_cleaner_ratio should be counters
[ https://issues.apache.org/jira/browse/HIVE-25390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-25390: - Fix Version/s: 4.0.0 > Metrics compaction_failed_initiator_ratio and compaction_failed_cleaner_ratio > should be counters > > > Key: HIVE-25390 > URL: https://issues.apache.org/jira/browse/HIVE-25390 > Project: Hive > Issue Type: Sub-task >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: backward-incompatible, pull-request-available > Fix For: 4.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Metric compaction_failed_initiator_ratio represents the ratio of initiator > failures to the total number of initiator runs, both computed since Metastore > was restarted, represented by a double. This isn't really useable. It would > be better if it counted the number of initiator failures since Metastore was > restarted so other components can keep an eye on things like "number of > initiator failures in the last day". > Same goes for compaction_failed_cleaner_ratio. > This commit removes metrics > * compaction_failed_initiator_ratio > * compaction_failed_cleaner_ratio > and introduces metrics > * compaction_initiator_failure_counter > * compaction_cleaner_failure_counter -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25390) Metrics compaction_failed_initiator_ratio and compaction_failed_cleaner_ratio should be counters
[ https://issues.apache.org/jira/browse/HIVE-25390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-25390. -- Resolution: Fixed Committed to master branch. Thanks for the review [~lpinter]! > Metrics compaction_failed_initiator_ratio and compaction_failed_cleaner_ratio > should be counters > > > Key: HIVE-25390 > URL: https://issues.apache.org/jira/browse/HIVE-25390 > Project: Hive > Issue Type: Sub-task >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: backward-incompatible, pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Metric compaction_failed_initiator_ratio represents the ratio of initiator > failures to the total number of initiator runs, both computed since Metastore > was restarted, represented by a double. This isn't really useable. It would > be better if it counted the number of initiator failures since Metastore was > restarted so other components can keep an eye on things like "number of > initiator failures in the last day". > Same goes for compaction_failed_cleaner_ratio. > This commit removes metrics > * compaction_failed_initiator_ratio > * compaction_failed_cleaner_ratio > and introduces metrics > * compaction_initiator_failure_counter > * compaction_cleaner_failure_counter -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-24235) Drop and recreate table during MR compaction leaves behind base/delta directory
[
https://issues.apache.org/jira/browse/HIVE-24235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17388018#comment-17388018
]
Karen Coppage commented on HIVE-24235:
--
This is useless because if the table is dropped, all related compactions are
cleared from metadata. So here we're trying to fail a compaction that doesn't
exist.
The actual fix for this issue is HIVE-25393.
> Drop and recreate table during MR compaction leaves behind base/delta
> directory
> ---
>
> Key: HIVE-24235
> URL: https://issues.apache.org/jira/browse/HIVE-24235
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> If a table is dropped and recreated during MR compaction, the table directory
> and a base (or delta, if minor compaction) directory could be created, with
> or without data, while the table "does not exist".
> E.g.
> {code:java}
> create table c (i int) stored as orc tblproperties
> ("NO_AUTO_COMPACTION"="true", "transactional"="true");
> insert into c values (9);
> insert into c values (9);
> alter table c compact 'major';
> While compaction job is running: {
> drop table c;
> create table c (i int) stored as orc tblproperties
> ("NO_AUTO_COMPACTION"="true", "transactional"="true");
> }
> {code}
> The table directory should be empty, but table directory could look like this
> after the job is finished:
> {code:java}
> Oct 6 14:23 c/base_002_v101/._orc_acid_version.crc
> Oct 6 14:23 c/base_002_v101/.bucket_0.crc
> Oct 6 14:23 c/base_002_v101/_orc_acid_version
> Oct 6 14:23 c/base_002_v101/bucket_0
> {code}
> or perhaps just:
> {code:java}
> Oct 6 14:23 c/base_002_v101/._orc_acid_version.crc
> Oct 6 14:23 c/base_002_v101/_orc_acid_version
> {code}
> Insert another row and you have:
> {code:java}
> Oct 6 14:33 base_002_v101/
> Oct 6 14:33 base_002_v101/._orc_acid_version.crc
> Oct 6 14:33 base_002_v101/.bucket_0.crc
> Oct 6 14:33 base_002_v101/_orc_acid_version
> Oct 6 14:33 base_002_v101/bucket_0
> Oct 6 14:35 delta_001_001_/._orc_acid_version.crc
> Oct 6 14:35 delta_001_001_/.bucket_0_0.crc
> Oct 6 14:35 delta_001_001_/_orc_acid_version
> Oct 6 14:35 delta_001_001_/bucket_0_0
> {code}
> Selecting from the table will result in this error because the highest valid
> writeId for this table is 1:
> {code:java}
> thrift.ThriftCLIService: Error fetching results:
> org.apache.hive.service.cli.HiveSQLException: Unable to get the next row set
> at
> org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:482)
> ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> ...
> Caused by: java.io.IOException: java.lang.RuntimeException: ORC split
> generation failed with exception: java.io.IOException: Not enough history
> available for (1,x). Oldest available base:
> .../warehouse/b/base_004_v092
> {code}
> Solution: Resolve the table again after compaction is finished; compare the
> id with the table id from when compaction began. If the ids do not match,
> abort the compaction's transaction.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-25390) Metrics compaction_failed_initiator_ratio and compaction_failed_cleaner_ratio should be counters
[ https://issues.apache.org/jira/browse/HIVE-25390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reassigned HIVE-25390: > Metrics compaction_failed_initiator_ratio and compaction_failed_cleaner_ratio > should be counters > > > Key: HIVE-25390 > URL: https://issues.apache.org/jira/browse/HIVE-25390 > Project: Hive > Issue Type: Sub-task >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: backward-incompatible > > Metric compaction_failed_initiator_ratio represents the ratio of initiator > failures to the total number of initiator runs, both computed since Metastore > was restarted, represented by a double. This isn't really useable. It would > be better if it counted the number of initiator failures since Metastore was > restarted so other components can keep an eye on things like "number of > initiator failures in the last day". > Same goes for compaction_failed_cleaner_ratio. > This commit removes metrics > * compaction_failed_initiator_ratio > * compaction_failed_cleaner_ratio > and introduces metrics > * compaction_initiator_failure_counter > * compaction_cleaner_failure_counter -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25359) Changes to metastore API in HIVE-24880 are not backwards compatible
[ https://issues.apache.org/jira/browse/HIVE-25359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-25359. -- Fix Version/s: 4.0.0 Resolution: Fixed Committed to master branch. Thanks for reviewing [~lpinter]! > Changes to metastore API in HIVE-24880 are not backwards compatible > --- > > Key: HIVE-25359 > URL: https://issues.apache.org/jira/browse/HIVE-25359 > Project: Hive > Issue Type: Sub-task >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > With HIVE-24880 find_next_compact(String workerId) was changed to > find_next_compact(String workerId, String workerVersion). This isn't > backwards compatible and could break other components > This commit reverts that change, deprecates find_next_compact, adds a new > method: find_next_compact2(FindNextCompactRequest rqst) where > FindNextCompactRequest has fields workerId and workerVersion, and makes Hive > use find_next_compact2. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25348) Skip metrics collection about writes to tables with tblproperty no_auto_compaction=true if CTAS
[
https://issues.apache.org/jira/browse/HIVE-25348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage resolved HIVE-25348.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Committed to master branch. Thank you for reviewing [~lpinter]!
> Skip metrics collection about writes to tables with tblproperty
> no_auto_compaction=true if CTAS
> ---
>
> Key: HIVE-25348
> URL: https://issues.apache.org/jira/browse/HIVE-25348
> Project: Hive
> Issue Type: Sub-task
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> We collect metrics about writes to tables with no_auto_compaction=true when
> allocating writeids. In the case of CTAS, if ACID is enabled on the new
> table, a writeid is allocated before the table object is created so we can't
> get tblproperties from it when allocating the writeid.
> In this case we should skip collecting the metric.
> This commit fixes errors like this:
> {code:java}
> 2021-07-16 18:48:04,350 ERROR
> org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-9-thread-72]:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.HMSMetricsListener.onAllocWriteId(HMSMetricsListener.java:104)
> at
> org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.lambda$static$6(MetaStoreListenerNotifier.java:229)
> at
> org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:291)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.allocate_table_write_ids(HiveMetaStore.java:8592)
> at sun.reflect.GeneratedMethodAccessor86.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:160)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:121)
> at com.sun.proxy.$Proxy33.allocate_table_write_ids(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$allocate_table_write_ids.getResult(ThriftHiveMetastore.java:21584)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$allocate_table_write_ids.getResult(ThriftHiveMetastore.java:21568)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-25371) Add myself to thrift file reviewers
[ https://issues.apache.org/jira/browse/HIVE-25371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-25371. -- Fix Version/s: 4.0.0 Resolution: Fixed Committed to master. Thanks [~kgyrtkirk] for reviewing! > Add myself to thrift file reviewers > --- > > Key: HIVE-25371 > URL: https://issues.apache.org/jira/browse/HIVE-25371 > Project: Hive > Issue Type: Task >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25371) Add myself to thrift file reviewers
[ https://issues.apache.org/jira/browse/HIVE-25371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reassigned HIVE-25371: > Add myself to thrift file reviewers > --- > > Key: HIVE-25371 > URL: https://issues.apache.org/jira/browse/HIVE-25371 > Project: Hive > Issue Type: Task >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-25359) Changes to metastore API in HIVE-24880 are not backwards compatible
[ https://issues.apache.org/jira/browse/HIVE-25359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384886#comment-17384886 ] Karen Coppage commented on HIVE-25359: -- [~kgyrtkirk] sure thing. I'll add myself to the thrift watchers. > Changes to metastore API in HIVE-24880 are not backwards compatible > --- > > Key: HIVE-25359 > URL: https://issues.apache.org/jira/browse/HIVE-25359 > Project: Hive > Issue Type: Sub-task >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > With HIVE-24880 find_next_compact(String workerId) was changed to > find_next_compact(String workerId, String workerVersion). This isn't > backwards compatible and could break other components > This commit reverts that change, deprecates find_next_compact, adds a new > method: find_next_compact2(FindNextCompactRequest rqst) where > FindNextCompactRequest has fields workerId and workerVersion, and makes Hive > use find_next_compact2. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25359) Changes to metastore API in HIVE-24880 are not backwards compatible
[ https://issues.apache.org/jira/browse/HIVE-25359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-25359: - Parent: HIVE-24824 Issue Type: Sub-task (was: Bug) > Changes to metastore API in HIVE-24880 are not backwards compatible > --- > > Key: HIVE-25359 > URL: https://issues.apache.org/jira/browse/HIVE-25359 > Project: Hive > Issue Type: Sub-task >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > > With HIVE-24880 find_next_compact(String workerId) was changed to > find_next_compact(String workerId, String workerVersion). This isn't > backwards compatible and could break other components > This commit reverts that change, deprecates find_next_compact, adds a new > method: find_next_compact2(FindNextCompactRequest rqst) where > FindNextCompactRequest has fields workerId and workerVersion, and makes Hive > use find_next_compact2. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25359) Changes to metastore API in HIVE-24880 are not backwards compatible
[ https://issues.apache.org/jira/browse/HIVE-25359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reassigned HIVE-25359: > Changes to metastore API in HIVE-24880 are not backwards compatible > --- > > Key: HIVE-25359 > URL: https://issues.apache.org/jira/browse/HIVE-25359 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > > With HIVE-24880 find_next_compact(String workerId) was changed to > find_next_compact(String workerId, String workerVersion). This isn't > backwards compatible and could break other components > This commit reverts that change, deprecates find_next_compact, adds a new > method: find_next_compact2(FindNextCompactRequest rqst) where > FindNextCompactRequest has fields workerId and workerVersion, and makes Hive > use find_next_compact2. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25348) Skip metrics collection about writes to tables with tblproperty no_auto_compaction=true if CTAS
[
https://issues.apache.org/jira/browse/HIVE-25348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage updated HIVE-25348:
-
Parent: HIVE-24824
Issue Type: Sub-task (was: Bug)
> Skip metrics collection about writes to tables with tblproperty
> no_auto_compaction=true if CTAS
> ---
>
> Key: HIVE-25348
> URL: https://issues.apache.org/jira/browse/HIVE-25348
> Project: Hive
> Issue Type: Sub-task
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> We collect metrics about writes to tables with no_auto_compaction=true when
> allocating writeids. In the case of CTAS, if ACID is enabled on the new
> table, a writeid is allocated before the table object is created so we can't
> get tblproperties from it when allocating the writeid.
> In this case we should skip collecting the metric.
> This commit fixes errors like this:
> {code:java}
> 2021-07-16 18:48:04,350 ERROR
> org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-9-thread-72]:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.HMSMetricsListener.onAllocWriteId(HMSMetricsListener.java:104)
> at
> org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.lambda$static$6(MetaStoreListenerNotifier.java:229)
> at
> org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:291)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.allocate_table_write_ids(HiveMetaStore.java:8592)
> at sun.reflect.GeneratedMethodAccessor86.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:160)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:121)
> at com.sun.proxy.$Proxy33.allocate_table_write_ids(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$allocate_table_write_ids.getResult(ThriftHiveMetastore.java:21584)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$allocate_table_write_ids.getResult(ThriftHiveMetastore.java:21568)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-25323) Fix TestVectorCastStatement
[ https://issues.apache.org/jira/browse/HIVE-25323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383937#comment-17383937 ] Karen Coppage commented on HIVE-25323: -- [~adeshrao] great, thank you! It would be good to always keep test coverage at the maximum. > Fix TestVectorCastStatement > --- > > Key: HIVE-25323 > URL: https://issues.apache.org/jira/browse/HIVE-25323 > Project: Hive > Issue Type: Task >Reporter: Karen Coppage >Assignee: Adesh Kumar Rao >Priority: Major > > org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorCastStatement > tests were timing out after 5 hours. > [http://ci.hive.apache.org/job/hive-flaky-check/307/] > First failure: > [http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/master/749/pipeline/242] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25348) Skip metrics collection about writes to tables with tblproperty no_auto_compaction=true if CTAS
[
https://issues.apache.org/jira/browse/HIVE-25348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage reassigned HIVE-25348:
> Skip metrics collection about writes to tables with tblproperty
> no_auto_compaction=true if CTAS
> ---
>
> Key: HIVE-25348
> URL: https://issues.apache.org/jira/browse/HIVE-25348
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
>
> We collect metrics about writes to tables with no_auto_compaction=true when
> allocating writeids. In the case of CTAS, if ACID is enabled on the new
> table, a writeid is allocated before the table object is created so we can't
> get tblproperties from it when allocating the writeid.
> In this case we should skip collecting the metric.
> This commit fixes errors like this:
> {code:java}
> 2021-07-16 18:48:04,350 ERROR
> org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-9-thread-72]:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.HMSMetricsListener.onAllocWriteId(HMSMetricsListener.java:104)
> at
> org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.lambda$static$6(MetaStoreListenerNotifier.java:229)
> at
> org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:291)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.allocate_table_write_ids(HiveMetaStore.java:8592)
> at sun.reflect.GeneratedMethodAccessor86.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:160)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:121)
> at com.sun.proxy.$Proxy33.allocate_table_write_ids(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$allocate_table_write_ids.getResult(ThriftHiveMetastore.java:21584)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$allocate_table_write_ids.getResult(ThriftHiveMetastore.java:21568)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-25323) Fix TestVectorCastStatement
[ https://issues.apache.org/jira/browse/HIVE-25323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381338#comment-17381338 ] Karen Coppage commented on HIVE-25323: -- [~adeshrao] thanks for taking a look! Probably the reason java.sql.Timestamp was left in vectorization is lost to time. Until the fix is done – and I assume it will be a bit of time – we can re-enable the test, convert the timestamps from java.sql.Timestamp -> hive.Timestamp and then compare. By the way do you know why the test was timing out after 5h instead of failing? > Fix TestVectorCastStatement > --- > > Key: HIVE-25323 > URL: https://issues.apache.org/jira/browse/HIVE-25323 > Project: Hive > Issue Type: Task >Reporter: Karen Coppage >Assignee: Adesh Kumar Rao >Priority: Major > > org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorCastStatement > tests were timing out after 5 hours. > [http://ci.hive.apache.org/job/hive-flaky-check/307/] > First failure: > [http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/master/749/pipeline/242] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-25323) Fix TestVectorCastStatement
[ https://issues.apache.org/jira/browse/HIVE-25323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17379948#comment-17379948 ] Karen Coppage commented on HIVE-25323: -- [~adeshrao] Not sure about this but you seem to have committed the only change related to (timestamp) casting just before this test started failing. Would you mind taking a look? [http://ci.hive.apache.org/job/hive-flaky-check/307/] > Fix TestVectorCastStatement > --- > > Key: HIVE-25323 > URL: https://issues.apache.org/jira/browse/HIVE-25323 > Project: Hive > Issue Type: Task >Reporter: Karen Coppage >Priority: Major > > org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorCastStatement > tests were timing out after 5 hours. > [http://ci.hive.apache.org/job/hive-flaky-check/307/] > First failure: > [http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/master/749/pipeline/242] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25323) Fix TestVectorCastStatement
[ https://issues.apache.org/jira/browse/HIVE-25323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-25323: - Description: org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorCastStatement tests were timing out after 5 hours. [http://ci.hive.apache.org/job/hive-flaky-check/307/] First failure: [http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/master/749/pipeline/242] was: org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorCastStatement tests were timing out after 5 hours. [http://ci.hive.apache.org/job/hive-flaky-check/307/] http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/master/749/pipeline/242 > Fix TestVectorCastStatement > --- > > Key: HIVE-25323 > URL: https://issues.apache.org/jira/browse/HIVE-25323 > Project: Hive > Issue Type: Task >Reporter: Karen Coppage >Priority: Major > > org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorCastStatement > tests were timing out after 5 hours. > [http://ci.hive.apache.org/job/hive-flaky-check/307/] > First failure: > [http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/master/749/pipeline/242] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25318) Number of initiator hosts metric should ignore manually initiated compactions
[ https://issues.apache.org/jira/browse/HIVE-25318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-25318. -- Fix Version/s: 4.0.0 Resolution: Fixed > Number of initiator hosts metric should ignore manually initiated compactions > - > > Key: HIVE-25318 > URL: https://issues.apache.org/jira/browse/HIVE-25318 > Project: Hive > Issue Type: Sub-task >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-25318) Number of initiator hosts metric should ignore manually initiated compactions
[ https://issues.apache.org/jira/browse/HIVE-25318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17379156#comment-17379156 ] Karen Coppage commented on HIVE-25318: -- Committed to master on July 11. Thanks for the review [~asinkovits]! > Number of initiator hosts metric should ignore manually initiated compactions > - > > Key: HIVE-25318 > URL: https://issues.apache.org/jira/browse/HIVE-25318 > Project: Hive > Issue Type: Sub-task >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25318) Number of initiator hosts metric should ignore manually initiated compactions
[ https://issues.apache.org/jira/browse/HIVE-25318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reassigned HIVE-25318: > Number of initiator hosts metric should ignore manually initiated compactions > - > > Key: HIVE-25318 > URL: https://issues.apache.org/jira/browse/HIVE-25318 > Project: Hive > Issue Type: Sub-task >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24858) UDFClassLoader leak in Configuration.CACHE_CLASSES
[
https://issues.apache.org/jira/browse/HIVE-24858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage resolved HIVE-24858.
--
Fix Version/s: 4.0.0
Resolution: Fixed
Committed to master on March 10. Thanks for reviewing [~pvary]!
> UDFClassLoader leak in Configuration.CACHE_CLASSES
> --
>
> Key: HIVE-24858
> URL: https://issues.apache.org/jira/browse/HIVE-24858
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> If a UDF jar has been registered in a session and a temporary function
> created from it, when the session is closed its UDFClassLoader is not GC'd as
> it has been leaked to the session's HiveConf object's cache. Since the
> ClassLoader is not GC'd, the UDF jar's classes aren't GC'd from Metaspace.
> This can potentially lead to Metaspace OOM.
> Path to GC root is:
> {code:java}
> Class Name
> | Shallow Heap |
> Retained Heap
> ---
> contextClassLoader org.apache.hive.service.server.ThreadWithGarbageCleanup @
> 0x7164deb50 HiveServer2-Handler-Pool: Thread-72 Thread| 128 |
> 79,072
> referent java.util.WeakHashMap$Entry @ 0x7164e67d0
> | 40 |
>824
> '- [6] java.util.WeakHashMap$Entry[16] @ 0x71581aac0
> | 80 |
> 5,056
>'- table java.util.WeakHashMap @ 0x71580f510
> | 48 |
> 6,920
> '- CACHE_CLASSES class org.apache.hadoop.conf.Configuration @
> 0x71580f3d8 |
> 64 |74,528
> ---
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-25252) All new compaction metrics should be lower case
[ https://issues.apache.org/jira/browse/HIVE-25252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-25252. -- Fix Version/s: 4.0.0 Resolution: Fixed Committed to master branch. Thanks for your contribution [~asinkovits]! > All new compaction metrics should be lower case > --- > > Key: HIVE-25252 > URL: https://issues.apache.org/jira/browse/HIVE-25252 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 50m > Remaining Estimate: 0h > > E.g: > compaction_worker_cycle_MINOR -> compaction_worker_cycle_minor > compaction_worker_cycle_MAJOR -> compaction_worker_cycle_major > compaction_cleaner_cycle_MINOR -> compaction_cleaner_cycle_minor > compaction_cleaner_cycle_MAJOR -> compaction_cleaner_cycle_major -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25259) Tweak delta metrics with custom MBean for Prometheus
[ https://issues.apache.org/jira/browse/HIVE-25259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-25259: - Fix Version/s: 4.0.0 > Tweak delta metrics with custom MBean for Prometheus > > > Key: HIVE-25259 > URL: https://issues.apache.org/jira/browse/HIVE-25259 > Project: Hive > Issue Type: Sub-task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25259) Tweak delta metrics with custom MBean for Prometheus
[ https://issues.apache.org/jira/browse/HIVE-25259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reassigned HIVE-25259: Assignee: Denys Kuzmenko > Tweak delta metrics with custom MBean for Prometheus > > > Key: HIVE-25259 > URL: https://issues.apache.org/jira/browse/HIVE-25259 > Project: Hive > Issue Type: Sub-task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25080) Create metric about oldest entry in "ready for cleaning" state
[ https://issues.apache.org/jira/browse/HIVE-25080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-25080: - Parent: HIVE-24824 Issue Type: Sub-task (was: Bug) > Create metric about oldest entry in "ready for cleaning" state > -- > > Key: HIVE-25080 > URL: https://issues.apache.org/jira/browse/HIVE-25080 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > When a compaction txn commits, COMPACTION_QUEUE.CQ_COMMIT_TIME is updated > with the current time. Then the compaction state is set to "ready for > cleaning". (... and then the Cleaner runs and the state is set to "succeeded" > hopefully) > Based on this we know (roughly) how long a compaction has been in state > "ready for cleaning". > We should create a metric similar to compaction_oldest_enqueue_age_in_sec > that would show that the cleaner is blocked by something i.e. find the > compaction in "ready for cleaning" that has the oldest commit time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25215) tables_with_x_aborted_transactions should count partition/unpartitioned tables
[ https://issues.apache.org/jira/browse/HIVE-25215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-25215. -- Fix Version/s: 4.0.0 Resolution: Fixed Committed to master branch. Thanks for the contribution [~asinkovits]! > tables_with_x_aborted_transactions should count partition/unpartitioned tables > -- > > Key: HIVE-25215 > URL: https://issues.apache.org/jira/browse/HIVE-25215 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Initiator compare's each partition's number of aborts to > hive.compactor.abortedtxn.threshold, so tables_with_x_aborted_transactions > should reflect the number of partitions/unpartitioned tables with >x aborts, > instead of the number of tables with >x aborts. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25081) Put metrics collection behind a feature flag
[ https://issues.apache.org/jira/browse/HIVE-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-25081. -- Fix Version/s: 4.0.0 Resolution: Fixed Committed to master branch. Thanks for the contribution [~asinkovits]! > Put metrics collection behind a feature flag > > > Key: HIVE-25081 > URL: https://issues.apache.org/jira/browse/HIVE-25081 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Most metrics we're creating are collected in AcidMetricsService, which is > behind a feature flag. However there are some metrics that are collected > outside of the service. These should be behind a feature flag in addition to > hive.metastore.metrics.enabled. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-25060) Hive Compactor doesn´t launch cleaner
[ https://issues.apache.org/jira/browse/HIVE-25060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17362841#comment-17362841 ] Karen Coppage commented on HIVE-25060: -- [~FranTCS] could you provide logs please? Otherwise it's impossible to figure out what the issue(s) may be. > Hive Compactor doesn´t launch cleaner > - > > Key: HIVE-25060 > URL: https://issues.apache.org/jira/browse/HIVE-25060 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.1 > Environment: Hive 3.1.0 > Hadoop 3.1.1 >Reporter: Fran Gonzalez >Priority: Major > > Hello, > there are problems with Hive Compactor. We can see in hivemetastore.log this > message "Max block location exceeded for split" and it´s appearing more and > more times. > After that, the "compactor.Cleaner" is not launched. > We observed that after a Hive Metastore restart, the "compactor.Cleaner" has > not been launched nevermore, but logs doesn´t display any message about it. > Could be a degradation of the Hive Compactor when delta files are growing in > the partitions? > Regards. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25080) Create metric about oldest entry in "ready for cleaning" state
[ https://issues.apache.org/jira/browse/HIVE-25080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-25080. -- Fix Version/s: 4.0.0 Resolution: Fixed Committed to master branch. Thanks for your contribution [~asinkovits]! > Create metric about oldest entry in "ready for cleaning" state > -- > > Key: HIVE-25080 > URL: https://issues.apache.org/jira/browse/HIVE-25080 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > When a compaction txn commits, COMPACTION_QUEUE.CQ_COMMIT_TIME is updated > with the current time. Then the compaction state is set to "ready for > cleaning". (... and then the Cleaner runs and the state is set to "succeeded" > hopefully) > Based on this we know (roughly) how long a compaction has been in state > "ready for cleaning". > We should create a metric similar to compaction_oldest_enqueue_age_in_sec > that would show that the cleaner is blocked by something i.e. find the > compaction in "ready for cleaning" that has the oldest commit time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25079) Create new metric about number of writes to tables with manually disabled compaction
[ https://issues.apache.org/jira/browse/HIVE-25079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-25079. -- Fix Version/s: 4.0.0 Resolution: Fixed Committed to master branch. Thanks for your contribution [~asinkovits]! > Create new metric about number of writes to tables with manually disabled > compaction > > > Key: HIVE-25079 > URL: https://issues.apache.org/jira/browse/HIVE-25079 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Create a new metric that measures the number of writes tables that has > compaction turned off manually. It does not matter if the write is committed > or aborted (both are bad...) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25037) Create metric: Number of tables with > x aborts
[ https://issues.apache.org/jira/browse/HIVE-25037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-25037. -- Fix Version/s: 4.0.0 Resolution: Fixed Committed to master branch. Thanks for your contribution [~asinkovits]! > Create metric: Number of tables with > x aborts > --- > > Key: HIVE-25037 > URL: https://issues.apache.org/jira/browse/HIVE-25037 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Create metric about number of tables with > x aborts. > x should be settable and default to 1500. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25115) Compaction queue entries may accumulate in "ready for cleaning" state
[ https://issues.apache.org/jira/browse/HIVE-25115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reassigned HIVE-25115: > Compaction queue entries may accumulate in "ready for cleaning" state > - > > Key: HIVE-25115 > URL: https://issues.apache.org/jira/browse/HIVE-25115 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > > If the Cleaner does not delete any files, the compaction queue entry is > thrown back to the queue and remains in "ready for cleaning" state. > Problem: If 2 compactions run on the same table and enter "ready for > cleaning" state at the same time, only one "cleaning" will remove obsolete > files, the other entry will remain in the queue in "ready for cleaning" state. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25021) Divide oldest_open_txn into oldest replication and non-replication transactions
[ https://issues.apache.org/jira/browse/HIVE-25021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-25021. -- Fix Version/s: 4.0.0 Resolution: Fixed > Divide oldest_open_txn into oldest replication and non-replication > transactions > --- > > Key: HIVE-25021 > URL: https://issues.apache.org/jira/browse/HIVE-25021 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > We should have different metrics (age and txn id) for > oldest replication txn (TXN_TYPE==1) > oldest non-replication txn (TXN_TYPE!=1) > so recommendations can be tailored to the different cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-25021) Divide oldest_open_txn into oldest replication and non-replication transactions
[ https://issues.apache.org/jira/browse/HIVE-25021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17341777#comment-17341777 ] Karen Coppage commented on HIVE-25021: -- Merged to master branch. Thanks for the contribution [~asinkovits]! > Divide oldest_open_txn into oldest replication and non-replication > transactions > --- > > Key: HIVE-25021 > URL: https://issues.apache.org/jira/browse/HIVE-25021 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > We should have different metrics (age and txn id) for > oldest replication txn (TXN_TYPE==1) > oldest non-replication txn (TXN_TYPE!=1) > so recommendations can be tailored to the different cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24980) Add timeout for failed and did not initiate compaction cleanup
[ https://issues.apache.org/jira/browse/HIVE-24980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-24980: - Parent: HIVE-24824 Issue Type: Sub-task (was: Improvement) > Add timeout for failed and did not initiate compaction cleanup > -- > > Key: HIVE-24980 > URL: https://issues.apache.org/jira/browse/HIVE-24980 > Project: Hive > Issue Type: Sub-task >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Clear failed and not initiated compactions from COMPLETED_COMPACTIONS that > are older than a week (configurable) if there already is a newer successful > compaction on the table/partition and either (1) the succeeded compaction is > major or (2) it is minor and the not initiated or failed compaction is also > minor –– so a minor succeeded compaction will not cause the deletion of a > major not initiated or failed compaction from history. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25022) Metric about incomplete compactions
[
https://issues.apache.org/jira/browse/HIVE-25022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage resolved HIVE-25022.
--
Resolution: Won't Fix
Not needed since multiple failures will cause the table to go into "did not
initiate" state long-term.
> Metric about incomplete compactions
> ---
>
> Key: HIVE-25022
> URL: https://issues.apache.org/jira/browse/HIVE-25022
> Project: Hive
> Issue Type: Sub-task
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> "Compactions in a state" metrics (for example compaction_num_working) count
> the sum of tables/partitions where the last compaction is in that state.
> I propose introducing a new metric about incomplete compactions: i.e. the
> number of tables/partitions where the last finished compaction* is
> unsuccessful (failed or "did not initiate"), or where major compaction was
> unsuccessful then minor compaction succeeded (compaction is not "complete"
> since major compaction has not succeeded in the time since it should have
> run).
> Example:
> {code:java}
> These compactions ran on a partition:
> major succeeded
> major working
> major failed
> major initiated
> major working
> major failed
> major initiated
> major working
> The "compactions in a state" metrics will consider the state of this table:
> working.
> The "incomplete compactions" metric will consider this: incomplete, since
> there have been failed compactions since the last succeeded compaction.
> {code}
> Another example:
> {code:java}
> These compactions ran on a partition:
> major succeeded
> major failed
> minor failed
> minor succeeded
> The "compactions in a state" metrics will consider the state of this table:
> succeeded.
> The "incomplete compactions" metric will consider this: incomplete, since
> there hasn't been a major succeeded since major failed.{code}
> Last example:
> {code:java}
> These compactions ran on a partition:
> major succeeded
> minor did not initiate
> The "compactions in a state" metrics will consider the state of this table:
> did not initiate.
> The "incomplete compactions" metric will consider this: incomplete, since the
> last compaction was "did not initiate"{code}
> *finished compaction: state in (succeeded, failed, attempted/did not initiate)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23763) Query based minor compaction produces wrong files when rows with different buckets Ids are processed by the same FileSinkOperator
[
https://issues.apache.org/jira/browse/HIVE-23763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17325760#comment-17325760
]
Karen Coppage commented on HIVE-23763:
--
A similar-looking situation can occur in MR-based minor compaction, in that
case HIVE-17231 can help.
> Query based minor compaction produces wrong files when rows with different
> buckets Ids are processed by the same FileSinkOperator
> -
>
> Key: HIVE-23763
> URL: https://issues.apache.org/jira/browse/HIVE-23763
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> How to reproduce:
> - Create an unbucketed ACID table
> - Insert a bigger amount of data into this table so there would be multiple
> bucket files in the table
> The files in the table should look like this:
> /warehouse/tablespace/managed/hive/bubu_acid/delta_001_001_/bucket_0_0
> /warehouse/tablespace/managed/hive/bubu_acid/delta_001_001_/bucket_1_0
> /warehouse/tablespace/managed/hive/bubu_acid/delta_001_001_/bucket_2_0
> /warehouse/tablespace/managed/hive/bubu_acid/delta_001_001_/bucket_3_0
> /warehouse/tablespace/managed/hive/bubu_acid/delta_001_001_/bucket_4_0
> /warehouse/tablespace/managed/hive/bubu_acid/delta_001_001_/bucket_5_0
> - Do some delete on rows with different bucket Ids
> The files in a delete delta should look like this:
> /warehouse/tablespace/managed/hive/bubu_acid/delete_delta_002_002_/bucket_0
> /warehouse/tablespace/managed/hive/bubu_acid/delete_delta_006_006_/bucket_3
> /warehouse/tablespace/managed/hive/bubu_acid/delete_delta_006_006_/bucket_1
> - Run the query-based minor compaction
> - After the compaction the newly created delete delta containes only 1 bucket
> file. This file contains rows from all buckets and the table becomes unusable
> /warehouse/tablespace/managed/hive/bubu_acid/delete_delta_001_007_v066/bucket_0
> The issue happens only if rows with different bucket Ids are processed by the
> same FileSinkOperator.
> In the FileSinkOperator.process method, the files for the compaction table
> are created like this:
> {noformat}
> if (!bDynParts && !filesCreated) {
> if (lbDirName != null) {
> if (valToPaths.get(lbDirName) == null) {
> createNewPaths(null, lbDirName);
> }
> } else {
> if (conf.isCompactionTable()) {
> int bucketProperty = getBucketProperty(row);
> bucketId =
> BucketCodec.determineVersion(bucketProperty).decodeWriterId(bucketProperty);
> }
> createBucketFiles(fsp);
> }
> }
> {noformat}
> When the first row is processed, the file is created and then the
> filesCreated variable is set to true. Then when the other rows are processed,
> the first if statement will be false, so no new file gets created, but the
> row will be written into the file created for the first row.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24980) Add timeout for failed and did not initiate compaction cleanup
[ https://issues.apache.org/jira/browse/HIVE-24980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-24980: - Summary: Add timeout for failed and did not initiate compaction cleanup (was: Add timeout for failed and "not initiated" compaction cleanup) > Add timeout for failed and did not initiate compaction cleanup > -- > > Key: HIVE-24980 > URL: https://issues.apache.org/jira/browse/HIVE-24980 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Clear failed and not initiated compactions from COMPLETED_COMPACTIONS that > are older than a week (configurable) if there already is a newer successful > compaction on the table/partition and either (1) the succeeded compaction is > major or (2) it is minor and the not initiated or failed compaction is also > minor –– so a minor succeeded compaction will not cause the deletion of a > major not initiated or failed compaction from history. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24980) Add timeout for failed and did not initiate compaction cleanup
[ https://issues.apache.org/jira/browse/HIVE-24980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-24980. -- Fix Version/s: 4.0.0 Resolution: Fixed Committed to master branch. Thanks for reviewing [~dkuzmenko]! > Add timeout for failed and did not initiate compaction cleanup > -- > > Key: HIVE-24980 > URL: https://issues.apache.org/jira/browse/HIVE-24980 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Clear failed and not initiated compactions from COMPLETED_COMPACTIONS that > are older than a week (configurable) if there already is a newer successful > compaction on the table/partition and either (1) the succeeded compaction is > major or (2) it is minor and the not initiated or failed compaction is also > minor –– so a minor succeeded compaction will not cause the deletion of a > major not initiated or failed compaction from history. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25019) Rename metrics that have spaces in the name
[ https://issues.apache.org/jira/browse/HIVE-25019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-25019. -- Fix Version/s: 4.0.0 Resolution: Fixed Committed to master branch. Thanks for your contribution [~asinkovits]! > Rename metrics that have spaces in the name > --- > > Key: HIVE-25019 > URL: https://issues.apache.org/jira/browse/HIVE-25019 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Metrics "num_compactions_ready for cleaning" and "num_compactions_not > initiated" contain spaces. > They should be renamed to "num_compactions_ready_for_cleaning" and > "num_compactions_not_initiated" respectively. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25017) Fix response in GetLatestCommittedCompaction
[ https://issues.apache.org/jira/browse/HIVE-25017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-25017. -- Resolution: Fixed Committed to master branch. Thanks for your contribution [~hsnusonic]! > Fix response in GetLatestCommittedCompaction > > > Key: HIVE-25017 > URL: https://issues.apache.org/jira/browse/HIVE-25017 > Project: Hive > Issue Type: Bug >Reporter: Yu-Wen Lai >Assignee: Yu-Wen Lai >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Dbname and Tablename are required for CompactionInfoStruct but the response > of getLatestCommittedCompactionInfo is not setting them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25017) Fix response in GetLatestCommittedCompaction
[ https://issues.apache.org/jira/browse/HIVE-25017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-25017: - Fix Version/s: 4.0.0 > Fix response in GetLatestCommittedCompaction > > > Key: HIVE-25017 > URL: https://issues.apache.org/jira/browse/HIVE-25017 > Project: Hive > Issue Type: Bug >Reporter: Yu-Wen Lai >Assignee: Yu-Wen Lai >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Dbname and Tablename are required for CompactionInfoStruct but the response > of getLatestCommittedCompactionInfo is not setting them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25022) Metric about incomplete compactions
[
https://issues.apache.org/jira/browse/HIVE-25022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage reassigned HIVE-25022:
> Metric about incomplete compactions
> ---
>
> Key: HIVE-25022
> URL: https://issues.apache.org/jira/browse/HIVE-25022
> Project: Hive
> Issue Type: Sub-task
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
>
> "Compactions in a state" metrics (for example compaction_num_working) count
> the sum of tables/partitions where the last compaction is in that state.
> I propose introducing a new metric about incomplete compactions: i.e. the
> number of tables/partitions where the last finished compaction* is
> unsuccessful (failed or "did not initiate"), or where major compaction was
> unsuccessful then minor compaction succeeded (compaction is not "complete"
> since major compaction has not succeeded in the time since it should have
> run).
> Example:
> {code:java}
> These compactions ran on a partition:
> major succeeded
> major working
> major failed
> major initiated
> major working
> major failed
> major initiated
> major working
> The "compactions in a state" metrics will consider the state of this table:
> working.
> The "incomplete compactions" metric will consider this: incomplete, since
> there have been failed compactions since the last succeeded compaction.
> {code}
> Another example:
> {code:java}
> These compactions ran on a partition:
> major succeeded
> major failed
> minor failed
> minor succeeded
> The "compactions in a state" metrics will consider the state of this table:
> succeeded.
> The "incomplete compactions" metric will consider this: incomplete, since
> there hasn't been a major succeeded since major failed.{code}
> Last example:
> {code:java}
> These compactions ran on a partition:
> major succeeded
> minor did not initiate
> The "compactions in a state" metrics will consider the state of this table:
> did not initiate.
> The "incomplete compactions" metric will consider this: incomplete, since the
> last compaction was "did not initiate"{code}
> *finished compaction: state in (succeeded, failed, attempted/did not initiate)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24828) [HMS] Provide new HMS API to return latest committed compaction record for a given table
[ https://issues.apache.org/jira/browse/HIVE-24828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-24828: - Fix Version/s: 4.0.0 > [HMS] Provide new HMS API to return latest committed compaction record for a > given table > > > Key: HIVE-24828 > URL: https://issues.apache.org/jira/browse/HIVE-24828 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Kishen Das >Assignee: Yu-Wen Lai >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 5h 40m > Remaining Estimate: 0h > > We need a new HMS API to return the latest committed compaction record for a > given table. This can be used by a remote cache to decide whether a given > table's file metadata has been compacted or not, in order to decide whether > file metadata has to be refreshed from the file system before serving or it > can serve the current data from the cache. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24828) [HMS] Provide new HMS API to return latest committed compaction record for a given table
[ https://issues.apache.org/jira/browse/HIVE-24828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-24828. -- Resolution: Fixed Committed to master branch. Thanks for your contribution [~hsnusonic]! > [HMS] Provide new HMS API to return latest committed compaction record for a > given table > > > Key: HIVE-24828 > URL: https://issues.apache.org/jira/browse/HIVE-24828 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Kishen Das >Assignee: Yu-Wen Lai >Priority: Major > Labels: pull-request-available > Time Spent: 5h 40m > Remaining Estimate: 0h > > We need a new HMS API to return the latest committed compaction record for a > given table. This can be used by a remote cache to decide whether a given > table's file metadata has been compacted or not, in order to decide whether > file metadata has to be refreshed from the file system before serving or it > can serve the current data from the cache. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24980) Add timeout for failed and "not initiated" compaction cleanup
[ https://issues.apache.org/jira/browse/HIVE-24980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reassigned HIVE-24980: > Add timeout for failed and "not initiated" compaction cleanup > - > > Key: HIVE-24980 > URL: https://issues.apache.org/jira/browse/HIVE-24980 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > > Clear failed and not initiated compactions from COMPLETED_COMPACTIONS that > are older than a week (configurable) if there already is a newer successful > compaction on the table/partition and either (1) the succeeded compaction is > major or (2) it is minor and the not initiated or failed compaction is also > minor –– so a minor succeeded compaction will not cause the deletion of a > major not initiated or failed compaction from history. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24955) New metrics about aborted transactions
[ https://issues.apache.org/jira/browse/HIVE-24955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-24955. -- Resolution: Fixed Committed to master branch. Thanks [~dkuzmenko] for your review! > New metrics about aborted transactions > -- > > Key: HIVE-24955 > URL: https://issues.apache.org/jira/browse/HIVE-24955 > Project: Hive > Issue Type: Sub-task >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > 5 new metrics: > * Number of aborted transactions in the TXNS table (collected in > AcidMetricsService) > * Oldest aborted transaction (collected in AcidMetricsService) > * Number of aborted write transaction (incremented counter at > abortTransaction) > * Number of committed write transaction (incremented counter at > commitTransaction) > * Number of timed out transactions (cleaner removed them after heartbeat > time out) > The latter 3 will restart as 0 after every HMS restart -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24955) New metrics about aborted transactions
[ https://issues.apache.org/jira/browse/HIVE-24955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reassigned HIVE-24955: > New metrics about aborted transactions > -- > > Key: HIVE-24955 > URL: https://issues.apache.org/jira/browse/HIVE-24955 > Project: Hive > Issue Type: Sub-task >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Fix For: 4.0.0 > > > 5 new metrics: > * Number of aborted transactions in the TXNS table (collected in > AcidMetricsService) > * Oldest aborted transaction (collected in AcidMetricsService) > * Number of aborted write transaction (incremented counter at > abortTransaction) > * Number of committed write transaction (incremented counter at > commitTransaction) > * Number of timed out transactions (cleaner removed them after heartbeat > time out) > The latter 3 will restart as 0 after every HMS restart -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24881) Abort old open replication txns
[ https://issues.apache.org/jira/browse/HIVE-24881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-24881. -- Fix Version/s: 4.0.0 Resolution: Fixed Committed to master branch. Thanks for reviewing [~aasha] and [~dkuzmenko]! > Abort old open replication txns > --- > > Key: HIVE-24881 > URL: https://issues.apache.org/jira/browse/HIVE-24881 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > We should auto-abort/remove open replication txns that are older than a time > threshold (default: 24h). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HIVE-24881) Abort old open replication txns
[ https://issues.apache.org/jira/browse/HIVE-24881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reopened HIVE-24881: -- > Abort old open replication txns > --- > > Key: HIVE-24881 > URL: https://issues.apache.org/jira/browse/HIVE-24881 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > We should auto-abort/remove open replication txns that are older than a time > threshold (default: 24h). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-24881) Abort old open replication txns
[ https://issues.apache.org/jira/browse/HIVE-24881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17302534#comment-17302534 ] Karen Coppage commented on HIVE-24881: -- [~anishek] I didn't see your comment until just now. The use case is that I've seen a user with open txns that are a month+ old. This is blocking the compactor Cleaner thread from cleaning up obsolete files from the FS, which bloats the FS and slows down reads. > Abort old open replication txns > --- > > Key: HIVE-24881 > URL: https://issues.apache.org/jira/browse/HIVE-24881 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > We should auto-abort/remove open replication txns that are older than a time > threshold (default: 24h). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24881) Abort old open replication txns
[ https://issues.apache.org/jira/browse/HIVE-24881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-24881. -- Resolution: Won't Fix Won't do because aborting replication txns will break replication. > Abort old open replication txns > --- > > Key: HIVE-24881 > URL: https://issues.apache.org/jira/browse/HIVE-24881 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > We should auto-abort/remove open replication txns that are older than a time > threshold (default: 24h). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24881) Abort old open replication txns
[ https://issues.apache.org/jira/browse/HIVE-24881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reassigned HIVE-24881: > Abort old open replication txns > --- > > Key: HIVE-24881 > URL: https://issues.apache.org/jira/browse/HIVE-24881 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > > We should auto-abort/remove open replication txns that are older than a time > threshold (default: 24h). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24860) Shut down HeartbeatExecutorService at session close
[ https://issues.apache.org/jira/browse/HIVE-24860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-24860: - Description: If the HeartBeaterExecutorService isn't shut down, UDF classes belonging to the session's UDFClassLoader (which was the session thread's context classloader at HeartBeaterExecutorService creation) could pile up and cause Metaspace OOM. (was: ### What changes were proposed in this pull request? Shut down HeartbeatExecutorService at session close. ### Why are the changes needed? If the HeartBeaterExecutorService isn't shut down, UDF classes belonging to the session's UDFClassLoader (which was the session thread's context classloader at HeartBeaterExecutorService creation) could pile up and cause Metaspace OOM. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Manually) > Shut down HeartbeatExecutorService at session close > --- > > Key: HIVE-24860 > URL: https://issues.apache.org/jira/browse/HIVE-24860 > Project: Hive > Issue Type: Bug >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > If the HeartBeaterExecutorService isn't shut down, UDF classes belonging to > the session's UDFClassLoader (which was the session thread's context > classloader at HeartBeaterExecutorService creation) could pile up and cause > Metaspace OOM. -- This message was sent by Atlassian Jira (v8.3.4#803005)
