subject:"\[jira\] \[Commented\] \(HIVE\-9995\) ACID compaction tries to compact a single file"

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-04-09 Thread Adam Szita (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813150#comment-16813150
 ] 

Adam Szita commented on HIVE-9995:
--

Committed to master. Thanks Denys!

> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch, 
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.05.patch, 
> HIVE-9995.06.patch, HIVE-9995.07.patch, HIVE-9995.08.patch, 
> HIVE-9995.09.patch, HIVE-9995.10.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514 2016-06-09 16:06 
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r--   1 ekoifman staff500 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-04-02 Thread Peter Vary (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807722#comment-16807722
 ] 

Peter Vary commented on HIVE-9995:
--

+1 for patch 10

> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch, 
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.05.patch, 
> HIVE-9995.06.patch, HIVE-9995.07.patch, HIVE-9995.08.patch, 
> HIVE-9995.09.patch, HIVE-9995.10.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514 2016-06-09 16:06 
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r--   1 ekoifman staff500 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-04-01 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807409#comment-16807409
 ] 

Hive QA commented on HIVE-9995:
---



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12964487/HIVE-9995.10.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 15890 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16811/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16811/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16811/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12964487 - PreCommit-HIVE-Build

> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch, 
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.05.patch, 
> HIVE-9995.06.patch, HIVE-9995.07.patch, HIVE-9995.08.patch, 
> HIVE-9995.09.patch, HIVE-9995.10.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514 2016-06-09 16:06 
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r--   1 ekoifman staff500 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-04-01 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807402#comment-16807402
 ] 

Hive QA commented on HIVE-9995:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
 9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
51s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
28s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
48s{color} | {color:green} ql: The patch generated 0 new + 823 unchanged - 11 
fixed = 823 total (was 834) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 16s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16811/dev-support/hive-personality.sh
 |
| git revision | master / 5bf5d14 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16811/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch, 
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.05.patch, 
> HIVE-9995.06.patch, HIVE-9995.07.patch, HIVE-9995.08.patch, 
> HIVE-9995.09.patch, HIVE-9995.10.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-04-01 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807020#comment-16807020
 ] 

Hive QA commented on HIVE-9995:
---



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/1296/HIVE-9995.09.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 15890 tests 
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitions
 (batchId=263)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
 (batchId=263)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomNonExistent
 (batchId=263)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighBytesRead 
(batchId=263)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
 (batchId=263)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerSlowQueryElapsedTime
 (batchId=263)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerSlowQueryExecutionTime
 (batchId=263)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16797/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16797/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16797/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 1296 - PreCommit-HIVE-Build

> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch, 
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.05.patch, 
> HIVE-9995.06.patch, HIVE-9995.07.patch, HIVE-9995.08.patch, 
> HIVE-9995.09.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514 2016-06-09 16:06 
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r--   1 ekoifman staff500 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-04-01 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806979#comment-16806979
 ] 

Hive QA commented on HIVE-9995:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
53s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
39s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
49s{color} | {color:green} ql: The patch generated 0 new + 823 unchanged - 11 
fixed = 823 total (was 834) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 51s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16797/dev-support/hive-personality.sh
 |
| git revision | master / 82feb02 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16797/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch, 
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.05.patch, 
> HIVE-9995.06.patch, HIVE-9995.07.patch, HIVE-9995.08.patch, 
> HIVE-9995.09.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514 2016-06-09 16:06 
>

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-04-01 Thread Denys Kuzmenko (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806755#comment-16806755
 ] 

Denys Kuzmenko commented on HIVE-9995:
--

rebased

> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch, 
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.05.patch, 
> HIVE-9995.06.patch, HIVE-9995.07.patch, HIVE-9995.08.patch, 
> HIVE-9995.09.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514 2016-06-09 16:06 
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r--   1 ekoifman staff500 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-04-01 Thread Peter Vary (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806592#comment-16806592
 ] 

Peter Vary commented on HIVE-9995:
--

[~dkuzmenko]: Rebase please :(

> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch, 
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.05.patch, 
> HIVE-9995.06.patch, HIVE-9995.07.patch, HIVE-9995.08.patch, 
> HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514 2016-06-09 16:06 
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r--   1 ekoifman staff500 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-03-29 Thread Denys Kuzmenko (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805024#comment-16805024
 ] 

Denys Kuzmenko commented on HIVE-9995:
--

[~vgumashta],[~ashutoshc]: Does anyone have any last comments?

> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch, 
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.05.patch, 
> HIVE-9995.06.patch, HIVE-9995.07.patch, HIVE-9995.08.patch, 
> HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514 2016-06-09 16:06 
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r--   1 ekoifman staff500 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-03-29 Thread Peter Vary (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804981#comment-16804981
 ] 

Peter Vary commented on HIVE-9995:
--

LGTM +1 from my side

> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch, 
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.05.patch, 
> HIVE-9995.06.patch, HIVE-9995.07.patch, HIVE-9995.08.patch, 
> HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514 2016-06-09 16:06 
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r--   1 ekoifman staff500 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-03-27 Thread Denys Kuzmenko (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16802743#comment-16802743
 ] 

Denys Kuzmenko commented on HIVE-9995:
--

[~vgumashta] could you please review. Thank you.

> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch, 
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.05.patch, 
> HIVE-9995.06.patch, HIVE-9995.07.patch, HIVE-9995.08.patch, 
> HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514 2016-06-09 16:06 
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r--   1 ekoifman staff500 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-03-23 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799801#comment-16799801
 ] 

Hive QA commented on HIVE-9995:
---



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963514/HIVE-9995.08.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 15836 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16656/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16656/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16656/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12963514 - PreCommit-HIVE-Build

> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch, 
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.05.patch, 
> HIVE-9995.06.patch, HIVE-9995.07.patch, HIVE-9995.08.patch, 
> HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514 2016-06-09 16:06 
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r--   1 ekoifman staff500 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-03-23 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799786#comment-16799786
 ] 

Hive QA commented on HIVE-9995:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
13s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
47s{color} | {color:green} ql: The patch generated 0 new + 702 unchanged - 10 
fixed = 702 total (was 712) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 34s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16656/dev-support/hive-personality.sh
 |
| git revision | master / 2fa22bf |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16656/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch, 
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.05.patch, 
> HIVE-9995.06.patch, HIVE-9995.07.patch, HIVE-9995.08.patch, 
> HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514 2016-06-09 16:06 
>

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-03-23 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799689#comment-16799689
 ] 

Hive QA commented on HIVE-9995:
---



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963503/HIVE-9995.07.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16654/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16654/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16654/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL 
https://issues.apache.org/jira/secure/attachment/12963503/HIVE-9995.07.patch 
was found in seen patch url's cache and a test was probably run already on it. 
Aborting...
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12963503 - PreCommit-HIVE-Build

> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch, 
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.05.patch, 
> HIVE-9995.06.patch, HIVE-9995.07.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514 2016-06-09 16:06 
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r--   1 ekoifman staff500 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-03-23 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799688#comment-16799688
 ] 

Hive QA commented on HIVE-9995:
---



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963503/HIVE-9995.07.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 15836 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16653/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16653/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16653/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12963503 - PreCommit-HIVE-Build

> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch, 
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.05.patch, 
> HIVE-9995.06.patch, HIVE-9995.07.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514 2016-06-09 16:06 
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r--   1 ekoifman staff500 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-03-23 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799677#comment-16799677
 ] 

Hive QA commented on HIVE-9995:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
22s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
45s{color} | {color:red} ql: The patch generated 2 new + 702 unchanged - 10 
fixed = 704 total (was 712) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m  8s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16653/dev-support/hive-personality.sh
 |
| git revision | master / 2fa22bf |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16653/yetus/diff-checkstyle-ql.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16653/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch, 
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.05.patch, 
> HIVE-9995.06.patch, HIVE-9995.07.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-03-22 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799486#comment-16799486
 ] 

Hive QA commented on HIVE-9995:
---



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963436/HIVE-9995.06.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16643/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16643/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16643/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-03-23 01:34:37.391
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-16643/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-03-23 01:34:37.393
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 2fa22bf HIVE-21473: Bumping jackson version to 2.9.8 (Peter 
Vary, reviewed by Marta Kuczora)
+ git clean -f -d
Removing ${project.basedir}/
Removing itests/${project.basedir}/
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 2fa22bf HIVE-21473: Bumping jackson version to 2.9.8 (Peter 
Vary, reviewed by Marta Kuczora)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-03-23 01:34:39.092
+ rm -rf ../yetus_PreCommit-HIVE-Build-16643
+ mkdir ../yetus_PreCommit-HIVE-Build-16643
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-16643
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-16643/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
fatal: corrupt patch at line 113
fatal: corrupt patch at line 113
fatal: corrupt patch at line 113
The patch does not appear to apply with p0, p1, or p2
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-16643
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12963436 - PreCommit-HIVE-Build

> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch, 
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.05.patch, 
> HIVE-9995.06.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-03-22 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799229#comment-16799229
 ] 

Hive QA commented on HIVE-9995:
---



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963410/HIVE-9995.05.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 15836 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_use_ts_stats_for_mapjoin]
 (batchId=191)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
 (batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
 (batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics 
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps 
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse 
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps (batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError 
(batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=230)
org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty 
(batchId=230)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16636/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16636/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16636/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12963410 - PreCommit-HIVE-Build

> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch, 
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.05.patch, 
> HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514 2016-06-09 16:06 
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r--   1 ekoifman staff500 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-03-22 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799207#comment-16799207
 ] 

Hive QA commented on HIVE-9995:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  
8s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
47s{color} | {color:red} ql: The patch generated 1 new + 702 unchanged - 10 
fixed = 703 total (was 712) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
14s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 33s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16636/dev-support/hive-personality.sh
 |
| git revision | master / 2fa22bf |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16636/yetus/diff-checkstyle-ql.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16636/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch, 
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.05.patch, 
> HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514 2016-06-09

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-03-21 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798554#comment-16798554
 ] 

Hive QA commented on HIVE-9995:
---



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963282/HIVE-9995.04.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16619/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16619/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16619/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL 
https://issues.apache.org/jira/secure/attachment/12963282/HIVE-9995.04.patch 
was found in seen patch url's cache and a test was probably run already on it. 
Aborting...
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12963282 - PreCommit-HIVE-Build

> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch, 
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514 2016-06-09 16:06 
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r--   1 ekoifman staff500 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-03-21 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798555#comment-16798555
 ] 

Hive QA commented on HIVE-9995:
---



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963282/HIVE-9995.04.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16620/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16620/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16620/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL 
https://issues.apache.org/jira/secure/attachment/12963282/HIVE-9995.04.patch 
was found in seen patch url's cache and a test was probably run already on it. 
Aborting...
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12963282 - PreCommit-HIVE-Build

> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch, 
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514 2016-06-09 16:06 
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r--   1 ekoifman staff500 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-03-21 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798372#comment-16798372
 ] 

Hive QA commented on HIVE-9995:
---



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963282/HIVE-9995.04.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 15833 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16615/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16615/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16615/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12963282 - PreCommit-HIVE-Build

> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch, 
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514 2016-06-09 16:06 
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r--   1 ekoifman staff500 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-03-21 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798349#comment-16798349
 ] 

Hive QA commented on HIVE-9995:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
47s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m 
17s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
47s{color} | {color:green} ql: The patch generated 0 new + 702 unchanged - 10 
fixed = 702 total (was 712) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 14s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16615/dev-support/hive-personality.sh
 |
| git revision | master / 38682a4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16615/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch, 
> HIVE-9995.03.patch, HIVE-9995.04.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514 2016-06-09 16:06 
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x   - ekoifman staff  0

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-03-21 Thread Denys Kuzmenko (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797975#comment-16797975
 ] 

Denys Kuzmenko commented on HIVE-9995:
--

[~vgumashta] could you please review. Thanks you!

> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch, 
> HIVE-9995.03.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514 2016-06-09 16:06 
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r--   1 ekoifman staff500 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-03-21 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797872#comment-16797872
 ] 

Hive QA commented on HIVE-9995:
---



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963145/HIVE-9995.02.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 15833 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16604/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16604/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16604/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12963145 - PreCommit-HIVE-Build

> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch, 
> HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514 2016-06-09 16:06 
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r--   1 ekoifman staff500 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-03-20 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797852#comment-16797852
 ] 

Hive QA commented on HIVE-9995:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
49s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  4m  
8s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
47s{color} | {color:red} ql: The patch generated 3 new + 680 unchanged - 7 
fixed = 683 total (was 687) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 26s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16604/dev-support/hive-personality.sh
 |
| git revision | master / 25b14be |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16604/yetus/diff-checkstyle-ql.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16604/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.02.patch, 
> HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514 2016-06-09 16:06 
>

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-02-08 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764052#comment-16764052
 ] 

Hive QA commented on HIVE-9995:
---



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12958109/HIVE-9995.01.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 15784 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestTxnCommands2.testEmptyInTblproperties 
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion3 
(batchId=309)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testEmptyInTblproperties
 (batchId=323)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion3
 (batchId=323)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/16004/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16004/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16004/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12958109 - PreCommit-HIVE-Build

> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514 2016-06-09 16:06 
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r--   1 ekoifman staff500 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2019-02-08 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764036#comment-16764036
 ] 

Hive QA commented on HIVE-9995:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
59s{color} | {color:blue} ql in master has 2298 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
40s{color} | {color:red} ql: The patch generated 3 new + 158 unchanged - 7 
fixed = 161 total (was 165) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 27s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16004/dev-support/hive-personality.sh
 |
| git revision | master / 4ddc9de |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16004/yetus/diff-checkstyle-ql.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-16004/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-9995.01.patch, HIVE-9995.WIP.patch
>
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514 2016-06-09 16:06 
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x   -

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

2017-09-20 Thread Eugene Koifman (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174097#comment-16174097
 ] 

Eugene Koifman commented on HIVE-9995:
--

also, if the user is explicitly requesting Minor compaction but there are 
Original files and no base, should we automatically run Major or reject the 
request

also, 
make sure that Initiator request Major if there are Originals but no base



> ACID compaction tries to compact a single file
> --
>
> Key: HIVE-9995
> URL: https://issues.apache.org/jira/browse/HIVE-9995
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> Consider TestWorker.minorWithOpenInMiddle()
> since there is an open txnId=23, this doesn't have any meaningful minor 
> compaction work to do.  The system still tries to compact a single delta file 
> for 21-22 id range, and effectively copies the file onto itself.
> This is 1. inefficient and 2. can potentially affect a reader.
> (from a real cluster)
> Suppose we start with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016
> -rw-r--r--   1 ekoifman staff602 2016-06-09 16:03 
> /user/hive/warehouse/t/base_016/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_017_017_
> -rw-r--r--   1 ekoifman staff514 2016-06-09 16:06 
> /user/hive/warehouse/t/delta_017_017_/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> then do _alter table T compact 'minor';_
> then we end up with 
> {noformat}
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017
> -rw-r--r--   1 ekoifman staff588 2016-06-09 16:07 
> /user/hive/warehouse/t/base_017/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018
> -rw-r--r--   1 ekoifman staff500 2016-06-09 16:11 
> /user/hive/warehouse/t/delta_018_018/bucket_0
> drwxr-xr-x   - ekoifman staff  0 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_
> -rw-r--r--   1 ekoifman staff612 2016-06-09 16:07 
> /user/hive/warehouse/t/delta_018_018_/bucket_0
> {noformat}
> So compaction created a new dir _/user/hive/warehouse/t/delta_018_018_



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

[jira] [Commented] (HIVE-9995) ACID compaction tries to compact a single file

29 matches

Site Navigation

Mail list logo

Footer information