[jira] [Commented] (HIVE-19838) simplify & fix ColumnizedDeleteEventRegistry load loop
[ https://issues.apache.org/jira/browse/HIVE-19838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512071#comment-16512071 ] Teddy Choi commented on HIVE-19838: --- +1 LGTM, pending tests. When I try to reproduce the failed tests in the latest patch, they just pass. > simplify & fix ColumnizedDeleteEventRegistry load loop > -- > > Key: HIVE-19838 > URL: https://issues.apache.org/jira/browse/HIVE-19838 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19838.01.patch, HIVE-19838.02.patch, > HIVE-19838.patch > > > Apparently sometimes the delete count in ACID stats doesn't match what merger > actually returns. > It could be due to some deltas having duplicate deletes from parallel queries > (I guess?) that are being squashed by the merger or some other reasons beyond > my mortal comprehension. > The loop assumes the merger will return the exact number of records, so it > fails with array index exception. Also, it could actually be done in a single > loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19838) simplify & fix ColumnizedDeleteEventRegistry load loop
[
https://issues.apache.org/jira/browse/HIVE-19838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512050#comment-16512050
]
Hive QA commented on HIVE-19838:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12927558/HIVE-19838.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14519 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11772/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11772/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11772/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12927558 - PreCommit-HIVE-Build
> simplify & fix ColumnizedDeleteEventRegistry load loop
> --
>
> Key: HIVE-19838
> URL: https://issues.apache.org/jira/browse/HIVE-19838
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19838.01.patch, HIVE-19838.02.patch,
> HIVE-19838.patch
>
>
> Apparently sometimes the delete count in ACID stats doesn't match what merger
> actually returns.
> It could be due to some deltas having duplicate deletes from parallel queries
> (I guess?) that are being squashed by the merger or some other reasons beyond
> my mortal comprehension.
> The loop assumes the merger will return the exact number of records, so it
> fails with array index exception. Also, it could actually be done in a single
> loop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19838) simplify & fix ColumnizedDeleteEventRegistry load loop
[
https://issues.apache.org/jira/browse/HIVE-19838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512021#comment-16512021
]
Hive QA commented on HIVE-19838:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
47s{color} | {color:blue} ql in master has 2279 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
36s{color} | {color:red} ql: The patch generated 1 new + 40 unchanged - 34
fixed = 41 total (was 74) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11772/dev-support/hive-personality.sh
|
| git revision | master / 1bcf403 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11772/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11772/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> simplify & fix ColumnizedDeleteEventRegistry load loop
> --
>
> Key: HIVE-19838
> URL: https://issues.apache.org/jira/browse/HIVE-19838
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19838.01.patch, HIVE-19838.02.patch,
> HIVE-19838.patch
>
>
> Apparently sometimes the delete count in ACID stats doesn't match what merger
> actually returns.
> It could be due to some deltas having duplicate deletes from parallel queries
> (I guess?) that are being squashed by the merger or some other reasons beyond
> my mortal comprehension.
> The loop assumes the merger will return the exact number of records, so it
> fails with array index exception. Also, it could actually be done in a single
> loop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19838) simplify & fix ColumnizedDeleteEventRegistry load loop
[
https://issues.apache.org/jira/browse/HIVE-19838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510408#comment-16510408
]
Sergey Shelukhin commented on HIVE-19838:
-
Hmm, I thought I could repro TestTxnNoBucketsVectorized failure, but this test
fails for me even without this patch, due to rows in the beginning of testCTAS
{noformat}
runStatementOnDriver("create table myctas stored as ORC TBLPROPERTIES
('transactional" +
"'='true', 'transactional_properties'='default') as select a, b from " +
Table.NONACIDORCTBL);
{noformat}
being in reverse order w.r.t. files (rows are the same but the bucket_0 row
is in 1 and vice versa).
This is not the same failure as above by the looks of it.
Retrying the patch.
cc [~ekoifman]
> simplify & fix ColumnizedDeleteEventRegistry load loop
> --
>
> Key: HIVE-19838
> URL: https://issues.apache.org/jira/browse/HIVE-19838
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19838.01.patch, HIVE-19838.02.patch,
> HIVE-19838.patch
>
>
> Apparently sometimes the delete count in ACID stats doesn't match what merger
> actually returns.
> It could be due to some deltas having duplicate deletes from parallel queries
> (I guess?) that are being squashed by the merger or some other reasons beyond
> my mortal comprehension.
> The loop assumes the merger will return the exact number of records, so it
> fails with array index exception. Also, it could actually be done in a single
> loop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19838) simplify & fix ColumnizedDeleteEventRegistry load loop
[
https://issues.apache.org/jira/browse/HIVE-19838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510206#comment-16510206
]
Hive QA commented on HIVE-19838:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12927386/HIVE-19838.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14521 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCtasEmpty (batchId=289)
org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread.testQueueingWithThreads
(batchId=277)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11736/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11736/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11736/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12927386 - PreCommit-HIVE-Build
> simplify & fix ColumnizedDeleteEventRegistry load loop
> --
>
> Key: HIVE-19838
> URL: https://issues.apache.org/jira/browse/HIVE-19838
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19838.01.patch, HIVE-19838.patch
>
>
> Apparently sometimes the delete count in ACID stats doesn't match what merger
> actually returns.
> It could be due to some deltas having duplicate deletes from parallel queries
> (I guess?) that are being squashed by the merger or some other reasons beyond
> my mortal comprehension.
> The loop assumes the merger will return the exact number of records, so it
> fails with array index exception. Also, it could actually be done in a single
> loop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19838) simplify & fix ColumnizedDeleteEventRegistry load loop
[
https://issues.apache.org/jira/browse/HIVE-19838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510153#comment-16510153
]
Hive QA commented on HIVE-19838:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
2s{color} | {color:blue} ql in master has 2279 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 1 new + 40 unchanged - 34
fixed = 41 total (was 74) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11736/dev-support/hive-personality.sh
|
| git revision | master / 230ae36 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11736/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11736/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> simplify & fix ColumnizedDeleteEventRegistry load loop
> --
>
> Key: HIVE-19838
> URL: https://issues.apache.org/jira/browse/HIVE-19838
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19838.01.patch, HIVE-19838.patch
>
>
> Apparently sometimes the delete count in ACID stats doesn't match what merger
> actually returns.
> It could be due to some deltas having duplicate deletes from parallel queries
> (I guess?) that are being squashed by the merger or some other reasons beyond
> my mortal comprehension.
> The loop assumes the merger will return the exact number of records, so it
> fails with array index exception. Also, it could actually be done in a single
> loop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19838) simplify & fix ColumnizedDeleteEventRegistry load loop
[ https://issues.apache.org/jira/browse/HIVE-19838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509003#comment-16509003 ] Eugene Koifman commented on HIVE-19838: --- +1 I left a couple of nits on RB Ignore my previous comment about distinctOwids. It's a poorly named variable - it's really counting the number of distinct (writeid, bucketproperty) pairs and the search on CompressedOwid matches this. Note to self: For unbucketed tables, if multiple bucket files are all loaded, each files has it's own reader in the heap, which means regardless of how delete events are spread among files, the heap sorts all of them by (writeid, bucketprop. rowid) so ColumnizedDeleteEventRegistry.isDeleted() looks ok. > simplify & fix ColumnizedDeleteEventRegistry load loop > -- > > Key: HIVE-19838 > URL: https://issues.apache.org/jira/browse/HIVE-19838 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19838.01.patch, HIVE-19838.patch > > > Apparently sometimes the delete count in ACID stats doesn't match what merger > actually returns. > It could be due to some deltas having duplicate deletes from parallel queries > (I guess?) that are being squashed by the merger or some other reasons beyond > my mortal comprehension. > The loop assumes the merger will return the exact number of records, so it > fails with array index exception. Also, it could actually be done in a single > loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19838) simplify & fix ColumnizedDeleteEventRegistry load loop
[ https://issues.apache.org/jira/browse/HIVE-19838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508926#comment-16508926 ] Sergey Shelukhin commented on HIVE-19838: - Hmm... if you look at the below code in the current master, it does count each bp change as being distinct OWID. So this patch keeps that behavior, that doesn't change. > simplify & fix ColumnizedDeleteEventRegistry load loop > -- > > Key: HIVE-19838 > URL: https://issues.apache.org/jira/browse/HIVE-19838 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19838.01.patch, HIVE-19838.patch > > > Apparently sometimes the delete count in ACID stats doesn't match what merger > actually returns. > It could be due to some deltas having duplicate deletes from parallel queries > (I guess?) that are being squashed by the merger or some other reasons beyond > my mortal comprehension. > The loop assumes the merger will return the exact number of records, so it > fails with array index exception. Also, it could actually be done in a single > loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19838) simplify & fix ColumnizedDeleteEventRegistry load loop
[
https://issues.apache.org/jira/browse/HIVE-19838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508925#comment-16508925
]
Eugene Koifman commented on HIVE-19838:
---
I think one of the ways {{totalDeleteEventCount}} in
{{ColumnizedDeleteEventRegistry}} may be off, is that {{DeleteReaderValue}}
takes a ValidWriteIdList which means that {{next()}} may skip some event
because it belongs to a transaction that was not yet committed when the current
reader locked in the snapshot.
In practice, this would require compaction (at least a minor one) which
includes a txn that is open to the reader's txn, to complete before the
VectorizedOrc reader starts reading - which is possible but not very likely.
Another issue, which I think is eliminated by the current patch is,
{noformat}
if (lastSeenOwid != deleteRecordKey.originalWriteId ||
lastSeenBucketProperty != deleteRecordKey.bucketProperty) {
++distinctOwids;
lastSeenOwid = deleteRecordKey.originalWriteId;
lastSeenBucketProperty = deleteRecordKey.bucketProperty;
}
{noformat}
{{distinctOwids}} is incremented when bucketProperty changes, which seems
invalid even for bucketed tables.
> simplify & fix ColumnizedDeleteEventRegistry load loop
> --
>
> Key: HIVE-19838
> URL: https://issues.apache.org/jira/browse/HIVE-19838
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19838.01.patch, HIVE-19838.patch
>
>
> Apparently sometimes the delete count in ACID stats doesn't match what merger
> actually returns.
> It could be due to some deltas having duplicate deletes from parallel queries
> (I guess?) that are being squashed by the merger or some other reasons beyond
> my mortal comprehension.
> The loop assumes the merger will return the exact number of records, so it
> fails with array index exception. Also, it could actually be done in a single
> loop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19838) simplify & fix ColumnizedDeleteEventRegistry load loop
[ https://issues.apache.org/jira/browse/HIVE-19838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508877#comment-16508877 ] Sergey Shelukhin commented on HIVE-19838: - No, it's a different test. > simplify & fix ColumnizedDeleteEventRegistry load loop > -- > > Key: HIVE-19838 > URL: https://issues.apache.org/jira/browse/HIVE-19838 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19838.01.patch, HIVE-19838.patch > > > Apparently sometimes the delete count in ACID stats doesn't match what merger > actually returns. > It could be due to some deltas having duplicate deletes from parallel queries > (I guess?) that are being squashed by the merger or some other reasons beyond > my mortal comprehension. > The loop assumes the merger will return the exact number of records, so it > fails with array index exception. Also, it could actually be done in a single > loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19838) simplify & fix ColumnizedDeleteEventRegistry load loop
[ https://issues.apache.org/jira/browse/HIVE-19838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508850#comment-16508850 ] Prasanth Jayachandran commented on HIVE-19838: -- Is this related to HIVE-19597 or something new? > simplify & fix ColumnizedDeleteEventRegistry load loop > -- > > Key: HIVE-19838 > URL: https://issues.apache.org/jira/browse/HIVE-19838 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19838.01.patch, HIVE-19838.patch > > > Apparently sometimes the delete count in ACID stats doesn't match what merger > actually returns. > It could be due to some deltas having duplicate deletes from parallel queries > (I guess?) that are being squashed by the merger or some other reasons beyond > my mortal comprehension. > The loop assumes the merger will return the exact number of records, so it > fails with array index exception. Also, it could actually be done in a single > loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19838) simplify & fix ColumnizedDeleteEventRegistry load loop
[ https://issues.apache.org/jira/browse/HIVE-19838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508829#comment-16508829 ] Sergey Shelukhin commented on HIVE-19838: - Looks unrelated, passes for me locally. cc [~prasanth_j] looks like these tests are flaky again, although this time they usually fail all together. > simplify & fix ColumnizedDeleteEventRegistry load loop > -- > > Key: HIVE-19838 > URL: https://issues.apache.org/jira/browse/HIVE-19838 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19838.01.patch, HIVE-19838.patch > > > Apparently sometimes the delete count in ACID stats doesn't match what merger > actually returns. > It could be due to some deltas having duplicate deletes from parallel queries > (I guess?) that are being squashed by the merger or some other reasons beyond > my mortal comprehension. > The loop assumes the merger will return the exact number of records, so it > fails with array index exception. Also, it could actually be done in a single > loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19838) simplify & fix ColumnizedDeleteEventRegistry load loop
[
https://issues.apache.org/jira/browse/HIVE-19838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507242#comment-16507242
]
Hive QA commented on HIVE-19838:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12927140/HIVE-19838.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 14513 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitions
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomReadOps
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerVertexRawInputSplitsNoKill
(batchId=242)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11671/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11671/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11671/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12927140 - PreCommit-HIVE-Build
> simplify & fix ColumnizedDeleteEventRegistry load loop
> --
>
> Key: HIVE-19838
> URL: https://issues.apache.org/jira/browse/HIVE-19838
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19838.patch
>
>
> Apparently sometimes the delete count in ACID stats doesn't match what merger
> actually returns.
> It could be due to some deltas having duplicate deletes from parallel queries
> (I guess?) that are being squashed by the merger or some other reasons beyond
> my mortal comprehension.
> The loop assumes the merger will return the exact number of records, so it
> fails with array index exception. Also, it could actually be done in a single
> loop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19838) simplify & fix ColumnizedDeleteEventRegistry load loop
[
https://issues.apache.org/jira/browse/HIVE-19838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507231#comment-16507231
]
Hive QA commented on HIVE-19838:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
22s{color} | {color:blue} ql in master has 2285 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
34s{color} | {color:red} ql: The patch generated 1 new + 40 unchanged - 34
fixed = 41 total (was 74) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11671/dev-support/hive-personality.sh
|
| git revision | master / 95ea9f5 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11671/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11671/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> simplify & fix ColumnizedDeleteEventRegistry load loop
> --
>
> Key: HIVE-19838
> URL: https://issues.apache.org/jira/browse/HIVE-19838
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19838.patch
>
>
> Apparently sometimes the delete count in ACID stats doesn't match what merger
> actually returns.
> It could be due to some deltas having duplicate deletes from parallel queries
> (I guess?) that are being squashed by the merger or some other reasons beyond
> my mortal comprehension.
> The loop assumes the merger will return the exact number of records, so it
> fails with array index exception. Also, it could actually be done in a single
> loop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19838) simplify & fix ColumnizedDeleteEventRegistry load loop
[ https://issues.apache.org/jira/browse/HIVE-19838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506667#comment-16506667 ] Sergey Shelukhin commented on HIVE-19838: - [~ekoifman] [~teddy.choi] can you take a look? thnx > simplify & fix ColumnizedDeleteEventRegistry load loop > -- > > Key: HIVE-19838 > URL: https://issues.apache.org/jira/browse/HIVE-19838 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19838.patch > > > Apparently sometimes the delete count in ACID stats doesn't match what merger > actually returns. > It could be due to some deltas having duplicate deletes from parallel queries > (I guess?) that are being squashed by the merger or some other reasons beyond > my mortal comprehension. > The loop assumes the merger will return the exact number of records, so it > fails with array index exception. Also, it could actually be done in a single > loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
