subject:"\[jira\] \[Commented\] \(HIVE\-20699\) Query based compactor for full CRUD Acid tables"

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-02-04 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760377#comment-16760377
 ] 

Hive QA commented on HIVE-20699:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12957558/HIVE-20699.11.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 15729 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/15932/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15932/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15932/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12957558 - PreCommit-HIVE-Build

> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch, 
> HIVE-20699.10.patch, HIVE-20699.11.patch, HIVE-20699.11.patch, 
> HIVE-20699.2.patch, HIVE-20699.3.patch, HIVE-20699.4.patch, 
> HIVE-20699.5.patch, HIVE-20699.6.patch, HIVE-20699.7.patch, 
> HIVE-20699.8.patch, HIVE-20699.9.patch
>
>
> Currently the Acid compactor is implemented as generated MR job 
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition 
> and writes data back to the same partition.  This will merge the deltas and 
> 'apply' the delete events.  The simplest would be to just use Insert 
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new 
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-02-04 Thread Eugene Koifman (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760381#comment-16760381
 ] 

Eugene Koifman commented on HIVE-20699:
---

+1 patch 11

> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch, 
> HIVE-20699.10.patch, HIVE-20699.11.patch, HIVE-20699.11.patch, 
> HIVE-20699.2.patch, HIVE-20699.3.patch, HIVE-20699.4.patch, 
> HIVE-20699.5.patch, HIVE-20699.6.patch, HIVE-20699.7.patch, 
> HIVE-20699.8.patch, HIVE-20699.9.patch
>
>
> Currently the Acid compactor is implemented as generated MR job 
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition 
> and writes data back to the same partition.  This will merge the deltas and 
> 'apply' the delete events.  The simplest would be to just use Insert 
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new 
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-02-04 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760363#comment-16760363
 ] 

Hive QA commented on HIVE-20699:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 8s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
30s{color} | {color:blue} common in master has 65 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
43s{color} | {color:blue} ql in master has 2305 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
42s{color} | {color:red} ql: The patch generated 18 new + 628 unchanged - 3 
fixed = 646 total (was 631) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
16s{color} | {color:red} itests/hive-unit: The patch generated 20 new + 180 
unchanged - 4 fixed = 200 total (was 184) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 29 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  3m 
45s{color} | {color:red} ql generated 2 new + 2305 unchanged - 0 fixed = 2307 
total (was 2305) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
53s{color} | {color:red} ql generated 3 new + 97 unchanged - 3 fixed = 100 
total (was 100) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
12s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 17s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  
org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.buildCrudMajorCompactionQuery(HiveConf,
 Table, Partition, String) concatenates strings using + in a loop  At 
CompactorMR.java:strings using + in a loop  At CompactorMR.java:[line 534] |
|  |  
org.apache.hadoop.hive.ql.udf.generic.GenericUDFValidateAcidSortOrder$WriteIdRowId
 defines compareTo(GenericUDFValidateAcidSortOrder$WriteIdRowId) and uses 
Object.equals()  At GenericUDFValidateAcidSortOrder.java:Object.equals()  At 
GenericUDFValidateAcidSortOrder.java:[lines 88-97] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15932/dev-support/hive-personality.sh
 |
| git revision | master / 4a4b9ca |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-15932/yetus/diff-checkstyle-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-15932/yetus/diff-checkstyle-itests_hive-unit.txt
 |
| whitespace |

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-02-04 Thread Vaibhav Gumashta (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760303#comment-16760303
 ] 

Vaibhav Gumashta commented on HIVE-20699:
-

Patch compiles fine. Trying again

> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch, 
> HIVE-20699.10.patch, HIVE-20699.11.patch, HIVE-20699.2.patch, 
> HIVE-20699.3.patch, HIVE-20699.4.patch, HIVE-20699.5.patch, 
> HIVE-20699.6.patch, HIVE-20699.7.patch, HIVE-20699.8.patch, HIVE-20699.9.patch
>
>
> Currently the Acid compactor is implemented as generated MR job 
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition 
> and writes data back to the same partition.  This will merge the deltas and 
> 'apply' the delete events.  The simplest would be to just use Insert 
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new 
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-02-04 Thread Vaibhav Gumashta (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760289#comment-16760289
 ] 

Vaibhav Gumashta commented on HIVE-20699:
-

[~ekoifman] Thanks, fixed the memory settings for the test case, so I don't see 
the memory estimation returning -ve values now. Also removed redundant imports.

> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch, 
> HIVE-20699.10.patch, HIVE-20699.11.patch, HIVE-20699.2.patch, 
> HIVE-20699.3.patch, HIVE-20699.4.patch, HIVE-20699.5.patch, 
> HIVE-20699.6.patch, HIVE-20699.7.patch, HIVE-20699.8.patch, HIVE-20699.9.patch
>
>
> Currently the Acid compactor is implemented as generated MR job 
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition 
> and writes data back to the same partition.  This will merge the deltas and 
> 'apply' the delete events.  The simplest would be to just use Insert 
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new 
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-02-04 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760292#comment-16760292
 ] 

Hive QA commented on HIVE-20699:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12957554/HIVE-20699.11.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/15930/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15930/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15930/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-02-04 23:01:12.956
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-15930/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-02-04 23:01:12.959
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 4a4b9ca HIVE-21159 Modify Merge statement logic to perform 
Update split early (Eugene Koifman, reviewed by Vaibhav Gumashta)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 4a4b9ca HIVE-21159 Modify Merge statement logic to perform 
Update split early (Eugene Koifman, reviewed by Vaibhav Gumashta)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-02-04 23:01:14.009
+ rm -rf ../yetus_PreCommit-HIVE-Build-15930
+ mkdir ../yetus_PreCommit-HIVE-Build-15930
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-15930
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-15930/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: does not 
exist in index
error: 
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java: 
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java: does 
not exist in index
error: 
a/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveSplitGenerator.java: does 
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/SplitGrouper.java: does 
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java: does not 
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java: 
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java: 
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java: does not 
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java: 
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java: 
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java: 
does not exist in index
error: a/ql/src/test/results/clientpositive/show_functions.q.out: does not 
exist in index
Going to apply patch with: git apply -p1
/data/hiveptest/working/scratch/build.patch:12: trailing whitespace.
SPLIT_GROUPING_MODE("hive.split.grouping.mode", "query", new 
StringSet("query", "compactor"), 
/data/hiveptest/working/scratch/build.patch:47: trailing whitespace.
 
/data/hiveptest/working/scratch/build.patch:98: trailing whitespace.
conf.setBoolean("hive.merge.tezfiles", false); 
/data/hiveptest/working/scratch/build.patch:223: trailing whitespace.
conf.setBoolean("hive.merge.tezfiles", false); 
/data/hiveptest/working/scratch/build.patch:559: trailing whitespace.

warning: squelched 24 whitespace errors
warning: 29 lines add

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-02-04 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760276#comment-16760276
 ] 

Hive QA commented on HIVE-20699:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12957535/HIVE-20699.10.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 15729 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/15927/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15927/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15927/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12957535 - PreCommit-HIVE-Build

> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch, 
> HIVE-20699.10.patch, HIVE-20699.2.patch, HIVE-20699.3.patch, 
> HIVE-20699.4.patch, HIVE-20699.5.patch, HIVE-20699.6.patch, 
> HIVE-20699.7.patch, HIVE-20699.8.patch, HIVE-20699.9.patch
>
>
> Currently the Acid compactor is implemented as generated MR job 
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition 
> and writes data back to the same partition.  This will merge the deltas and 
> 'apply' the delete events.  The simplest would be to just use Insert 
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new 
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-02-04 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760265#comment-16760265
 ] 

Hive QA commented on HIVE-20699:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
30s{color} | {color:blue} common in master has 65 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
31s{color} | {color:blue} ql in master has 2305 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
36s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
40s{color} | {color:red} ql: The patch generated 25 new + 628 unchanged - 3 
fixed = 653 total (was 631) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
16s{color} | {color:red} itests/hive-unit: The patch generated 20 new + 180 
unchanged - 4 fixed = 200 total (was 184) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 27 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  3m 
50s{color} | {color:red} ql generated 2 new + 2305 unchanged - 0 fixed = 2307 
total (was 2305) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
54s{color} | {color:red} ql generated 3 new + 97 unchanged - 3 fixed = 100 
total (was 100) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
13s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 10s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  
org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.buildCrudMajorCompactionQuery(HiveConf,
 Table, Partition, String) concatenates strings using + in a loop  At 
CompactorMR.java:strings using + in a loop  At CompactorMR.java:[line 534] |
|  |  
org.apache.hadoop.hive.ql.udf.generic.GenericUDFValidateAcidSortOrder$WriteIdRowId
 defines compareTo(GenericUDFValidateAcidSortOrder$WriteIdRowId) and uses 
Object.equals()  At GenericUDFValidateAcidSortOrder.java:Object.equals()  At 
GenericUDFValidateAcidSortOrder.java:[lines 88-97] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15927/dev-support/hive-personality.sh
 |
| git revision | master / 4a4b9ca |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-15927/yetus/diff-checkstyle-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-15927/yetus/diff-checkstyle-itests_hive-unit.txt
 |
| whitespace |

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-02-04 Thread Eugene Koifman (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760259#comment-16760259
 ] 

Eugene Koifman commented on HIVE-20699:
---

There are a few unused imports in SplitGrouper
HiveSplitGenerator has unused imports and
{noformat}
 if (HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_IN_TEZ_TEST)) {
taskResource = Math.max(taskResource, 1);
  }
{noformat}
what does this do?


> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch, 
> HIVE-20699.10.patch, HIVE-20699.2.patch, HIVE-20699.3.patch, 
> HIVE-20699.4.patch, HIVE-20699.5.patch, HIVE-20699.6.patch, 
> HIVE-20699.7.patch, HIVE-20699.8.patch, HIVE-20699.9.patch
>
>
> Currently the Acid compactor is implemented as generated MR job 
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition 
> and writes data back to the same partition.  This will merge the deltas and 
> 'apply' the delete events.  The simplest would be to just use Insert 
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new 
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-02-04 Thread Vaibhav Gumashta (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760159#comment-16760159
 ] 

Vaibhav Gumashta commented on HIVE-20699:
-

Test failure is unrelated, but uploading v10 again to get another run.

> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch, 
> HIVE-20699.10.patch, HIVE-20699.2.patch, HIVE-20699.3.patch, 
> HIVE-20699.4.patch, HIVE-20699.5.patch, HIVE-20699.6.patch, 
> HIVE-20699.7.patch, HIVE-20699.8.patch, HIVE-20699.9.patch
>
>
> Currently the Acid compactor is implemented as generated MR job 
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition 
> and writes data back to the same partition.  This will merge the deltas and 
> 'apply' the delete events.  The simplest would be to just use Insert 
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new 
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-02-01 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758911#comment-16758911
 ] 

Hive QA commented on HIVE-20699:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12957337/HIVE-20699.10.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15727 tests 
executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.mapreduce.TestHCatPartitioned.testHCatPartitionedTable[1]
 (batchId=209)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/15902/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15902/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15902/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12957337 - PreCommit-HIVE-Build

> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch, 
> HIVE-20699.10.patch, HIVE-20699.10.patch, HIVE-20699.2.patch, 
> HIVE-20699.3.patch, HIVE-20699.4.patch, HIVE-20699.5.patch, 
> HIVE-20699.6.patch, HIVE-20699.7.patch, HIVE-20699.8.patch, HIVE-20699.9.patch
>
>
> Currently the Acid compactor is implemented as generated MR job 
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition 
> and writes data back to the same partition.  This will merge the deltas and 
> 'apply' the delete events.  The simplest would be to just use Insert 
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new 
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-02-01 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758894#comment-16758894
 ] 

Hive QA commented on HIVE-20699:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
30s{color} | {color:blue} common in master has 65 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
31s{color} | {color:blue} ql in master has 2304 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
40s{color} | {color:red} ql: The patch generated 25 new + 628 unchanged - 3 
fixed = 653 total (was 631) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
16s{color} | {color:red} itests/hive-unit: The patch generated 20 new + 180 
unchanged - 4 fixed = 200 total (was 184) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 27 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  3m 
39s{color} | {color:red} ql generated 2 new + 2304 unchanged - 0 fixed = 2306 
total (was 2304) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
51s{color} | {color:red} ql generated 3 new + 97 unchanged - 3 fixed = 100 
total (was 100) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
12s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 39s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  
org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.buildCrudMajorCompactionQuery(HiveConf,
 Table, Partition, String) concatenates strings using + in a loop  At 
CompactorMR.java:strings using + in a loop  At CompactorMR.java:[line 534] |
|  |  
org.apache.hadoop.hive.ql.udf.generic.GenericUDFValidateAcidSortOrder$WriteIdRowId
 defines compareTo(GenericUDFValidateAcidSortOrder$WriteIdRowId) and uses 
Object.equals()  At GenericUDFValidateAcidSortOrder.java:Object.equals()  At 
GenericUDFValidateAcidSortOrder.java:[lines 88-97] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15902/dev-support/hive-personality.sh
 |
| git revision | master / 80f5e48 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-15902/yetus/diff-checkstyle-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-15902/yetus/diff-checkstyle-itests_hive-unit.txt
 |
| whitespace |

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-02-01 Thread Vaibhav Gumashta (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758745#comment-16758745
 ] 

Vaibhav Gumashta commented on HIVE-20699:
-

Not sure why the last build failed - attaching v10 again. 

> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch, 
> HIVE-20699.10.patch, HIVE-20699.10.patch, HIVE-20699.2.patch, 
> HIVE-20699.3.patch, HIVE-20699.4.patch, HIVE-20699.5.patch, 
> HIVE-20699.6.patch, HIVE-20699.7.patch, HIVE-20699.8.patch, HIVE-20699.9.patch
>
>
> Currently the Acid compactor is implemented as generated MR job 
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition 
> and writes data back to the same partition.  This will merge the deltas and 
> 'apply' the delete events.  The simplest would be to just use Insert 
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new 
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-02-01 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758325#comment-16758325
 ] 

Hive QA commented on HIVE-20699:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12957235/HIVE-20699.10.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/15884/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15884/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15884/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL 
https://issues.apache.org/jira/secure/attachment/12957235/HIVE-20699.10.patch 
was found in seen patch url's cache and a test was probably run already on it. 
Aborting...
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12957235 - PreCommit-HIVE-Build

> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch, 
> HIVE-20699.10.patch, HIVE-20699.2.patch, HIVE-20699.3.patch, 
> HIVE-20699.4.patch, HIVE-20699.5.patch, HIVE-20699.6.patch, 
> HIVE-20699.7.patch, HIVE-20699.8.patch, HIVE-20699.9.patch
>
>
> Currently the Acid compactor is implemented as generated MR job 
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition 
> and writes data back to the same partition.  This will merge the deltas and 
> 'apply' the delete events.  The simplest would be to just use Insert 
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new 
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-02-01 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758223#comment-16758223
 ] 

Hive QA commented on HIVE-20699:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12957235/HIVE-20699.10.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 15726 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/15881/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15881/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15881/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12957235 - PreCommit-HIVE-Build

> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch, 
> HIVE-20699.10.patch, HIVE-20699.2.patch, HIVE-20699.3.patch, 
> HIVE-20699.4.patch, HIVE-20699.5.patch, HIVE-20699.6.patch, 
> HIVE-20699.7.patch, HIVE-20699.8.patch, HIVE-20699.9.patch
>
>
> Currently the Acid compactor is implemented as generated MR job 
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition 
> and writes data back to the same partition.  This will merge the deltas and 
> 'apply' the delete events.  The simplest would be to just use Insert 
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new 
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-02-01 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758189#comment-16758189
 ] 

Hive QA commented on HIVE-20699:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
28s{color} | {color:blue} common in master has 65 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
31s{color} | {color:blue} ql in master has 2304 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
36s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
40s{color} | {color:red} ql: The patch generated 25 new + 628 unchanged - 3 
fixed = 653 total (was 631) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
16s{color} | {color:red} itests/hive-unit: The patch generated 20 new + 180 
unchanged - 4 fixed = 200 total (was 184) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 27 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  3m 
40s{color} | {color:red} ql generated 2 new + 2304 unchanged - 0 fixed = 2306 
total (was 2304) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
52s{color} | {color:red} ql generated 3 new + 97 unchanged - 3 fixed = 100 
total (was 100) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
12s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 34s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  
org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.buildCrudMajorCompactionQuery(HiveConf,
 Table, Partition, String) concatenates strings using + in a loop  At 
CompactorMR.java:strings using + in a loop  At CompactorMR.java:[line 534] |
|  |  
org.apache.hadoop.hive.ql.udf.generic.GenericUDFValidateAcidSortOrder$WriteIdRowId
 defines compareTo(GenericUDFValidateAcidSortOrder$WriteIdRowId) and uses 
Object.equals()  At GenericUDFValidateAcidSortOrder.java:Object.equals()  At 
GenericUDFValidateAcidSortOrder.java:[lines 88-97] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15881/dev-support/hive-personality.sh
 |
| git revision | master / f2e10f2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-15881/yetus/diff-checkstyle-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-15881/yetus/diff-checkstyle-itests_hive-unit.txt
 |
| whitespace |

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-02-01 Thread Vaibhav Gumashta (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758141#comment-16758141
 ] 

Vaibhav Gumashta commented on HIVE-20699:
-

Some issue uploading v10 to review board - will try again in few hours.

> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch, 
> HIVE-20699.10.patch, HIVE-20699.2.patch, HIVE-20699.3.patch, 
> HIVE-20699.4.patch, HIVE-20699.5.patch, HIVE-20699.6.patch, 
> HIVE-20699.7.patch, HIVE-20699.8.patch, HIVE-20699.9.patch
>
>
> Currently the Acid compactor is implemented as generated MR job 
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition 
> and writes data back to the same partition.  This will merge the deltas and 
> 'apply' the delete events.  The simplest would be to just use Insert 
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new 
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-01-28 Thread Eugene Koifman (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754507#comment-16754507
 ] 

Eugene Koifman commented on HIVE-20699:
---

I left a few of minor comments on RB for patch 9.  Overall LGTM, except for 
{{HIVE_TRANSACTIONAL_TABLE_SCAN}} not being visible in {{HiveSplitGenerator}}.  
While not caused by this patch I don't think it can properly work in production 
w/o it being fixed.

> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch, 
> HIVE-20699.2.patch, HIVE-20699.3.patch, HIVE-20699.4.patch, 
> HIVE-20699.5.patch, HIVE-20699.6.patch, HIVE-20699.7.patch, 
> HIVE-20699.8.patch, HIVE-20699.9.patch
>
>
> Currently the Acid compactor is implemented as generated MR job 
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition 
> and writes data back to the same partition.  This will merge the deltas and 
> 'apply' the delete events.  The simplest would be to just use Insert 
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new 
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-01-28 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754369#comment-16754369
 ] 

Hive QA commented on HIVE-20699:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12956608/HIVE-20699.9.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 15725 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/15813/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15813/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15813/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12956608 - PreCommit-HIVE-Build

> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch, 
> HIVE-20699.2.patch, HIVE-20699.3.patch, HIVE-20699.4.patch, 
> HIVE-20699.5.patch, HIVE-20699.6.patch, HIVE-20699.7.patch, 
> HIVE-20699.8.patch, HIVE-20699.9.patch
>
>
> Currently the Acid compactor is implemented as generated MR job 
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition 
> and writes data back to the same partition.  This will merge the deltas and 
> 'apply' the delete events.  The simplest would be to just use Insert 
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new 
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-01-28 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754339#comment-16754339
 ] 

Hive QA commented on HIVE-20699:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
30s{color} | {color:blue} common in master has 65 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
40s{color} | {color:blue} ql in master has 2304 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
41s{color} | {color:red} ql: The patch generated 19 new + 520 unchanged - 1 
fixed = 539 total (was 521) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
16s{color} | {color:red} itests/hive-unit: The patch generated 21 new + 184 
unchanged - 0 fixed = 205 total (was 184) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 25 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  3m 
48s{color} | {color:red} ql generated 2 new + 2304 unchanged - 0 fixed = 2306 
total (was 2304) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
52s{color} | {color:red} ql generated 3 new + 97 unchanged - 3 fixed = 100 
total (was 100) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
12s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 36s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  
org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.buildCrudMajorCompactionQuery(HiveConf,
 Table, Partition, String) concatenates strings using + in a loop  At 
CompactorMR.java:strings using + in a loop  At CompactorMR.java:[line 534] |
|  |  
org.apache.hadoop.hive.ql.udf.generic.GenericUDFValidateAcidSortOrder$WriteIdRowId
 defines compareTo(GenericUDFValidateAcidSortOrder$WriteIdRowId) and uses 
Object.equals()  At GenericUDFValidateAcidSortOrder.java:Object.equals()  At 
GenericUDFValidateAcidSortOrder.java:[lines 88-97] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15813/dev-support/hive-personality.sh
 |
| git revision | master / 698206a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-15813/yetus/diff-checkstyle-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-15813/yetus/diff-checkstyle-itests_hive-unit.txt
 |
| whitespace |

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-01-25 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752915#comment-16752915
 ] 

Hive QA commented on HIVE-20699:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12956411/HIVE-20699.8.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15728 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestAcidOnTez.testCrudMajorCompactionSplitGrouper 
(batchId=241)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/15800/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15800/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15800/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12956411 - PreCommit-HIVE-Build

> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch, 
> HIVE-20699.2.patch, HIVE-20699.3.patch, HIVE-20699.4.patch, 
> HIVE-20699.5.patch, HIVE-20699.6.patch, HIVE-20699.7.patch, HIVE-20699.8.patch
>
>
> Currently the Acid compactor is implemented as generated MR job 
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition 
> and writes data back to the same partition.  This will merge the deltas and 
> 'apply' the delete events.  The simplest would be to just use Insert 
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new 
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-01-25 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752907#comment-16752907
 ] 

Hive QA commented on HIVE-20699:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
30s{color} | {color:blue} common in master has 65 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
34s{color} | {color:blue} ql in master has 2304 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
43s{color} | {color:red} ql: The patch generated 19 new + 520 unchanged - 1 
fixed = 539 total (was 521) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
15s{color} | {color:red} itests/hive-unit: The patch generated 21 new + 184 
unchanged - 0 fixed = 205 total (was 184) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 25 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  3m 
49s{color} | {color:red} ql generated 2 new + 2304 unchanged - 0 fixed = 2306 
total (was 2304) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
53s{color} | {color:red} ql generated 3 new + 97 unchanged - 3 fixed = 100 
total (was 100) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
13s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 43s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  
org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.buildCrudMajorCompactionQuery(HiveConf,
 Table, Partition, String) concatenates strings using + in a loop  At 
CompactorMR.java:strings using + in a loop  At CompactorMR.java:[line 534] |
|  |  
org.apache.hadoop.hive.ql.udf.generic.GenericUDFValidateAcidSortOrder$WriteIdRowId
 defines compareTo(GenericUDFValidateAcidSortOrder$WriteIdRowId) and uses 
Object.equals()  At GenericUDFValidateAcidSortOrder.java:Object.equals()  At 
GenericUDFValidateAcidSortOrder.java:[lines 88-97] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15800/dev-support/hive-personality.sh
 |
| git revision | master / 6d4b19b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-15800/yetus/diff-checkstyle-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-15800/yetus/diff-checkstyle-itests_hive-unit.txt
 |
| whitespace |

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-01-22 Thread Eugene Koifman (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749370#comment-16749370
 ] 

Eugene Koifman commented on HIVE-20699:
---

left some comments on RB

> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch, 
> HIVE-20699.2.patch, HIVE-20699.3.patch, HIVE-20699.4.patch, 
> HIVE-20699.5.patch, HIVE-20699.6.patch, HIVE-20699.7.patch
>
>
> Currently the Acid compactor is implemented as generated MR job 
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition 
> and writes data back to the same partition.  This will merge the deltas and 
> 'apply' the delete events.  The simplest would be to just use Insert 
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new 
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-01-22 Thread Vaibhav Gumashta (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748554#comment-16748554
 ] 

Vaibhav Gumashta commented on HIVE-20699:
-

[~ekoifman] The latest run is clean.

> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch, 
> HIVE-20699.2.patch, HIVE-20699.3.patch, HIVE-20699.4.patch, 
> HIVE-20699.5.patch, HIVE-20699.6.patch, HIVE-20699.7.patch
>
>
> Currently the Acid compactor is implemented as generated MR job 
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition 
> and writes data back to the same partition.  This will merge the deltas and 
> 'apply' the delete events.  The simplest would be to just use Insert 
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new 
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-01-22 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748542#comment-16748542
 ] 

Hive QA commented on HIVE-20699:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12955737/HIVE-20699.7.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 15701 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/15728/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15728/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15728/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12955737 - PreCommit-HIVE-Build

> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch, 
> HIVE-20699.2.patch, HIVE-20699.3.patch, HIVE-20699.4.patch, 
> HIVE-20699.5.patch, HIVE-20699.6.patch, HIVE-20699.7.patch
>
>
> Currently the Acid compactor is implemented as generated MR job 
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition 
> and writes data back to the same partition.  This will merge the deltas and 
> 'apply' the delete events.  The simplest would be to just use Insert 
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new 
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-01-22 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748506#comment-16748506
 ] 

Hive QA commented on HIVE-20699:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
29s{color} | {color:blue} common in master has 65 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
43s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
40s{color} | {color:red} ql: The patch generated 18 new + 520 unchanged - 1 
fixed = 538 total (was 521) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
17s{color} | {color:red} itests/hive-unit: The patch generated 24 new + 184 
unchanged - 0 fixed = 208 total (was 184) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 24 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  3m 
53s{color} | {color:red} ql generated 2 new + 2310 unchanged - 0 fixed = 2312 
total (was 2310) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
54s{color} | {color:red} ql generated 3 new + 97 unchanged - 3 fixed = 100 
total (was 100) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
13s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 42s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  
org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.buildCrudMajorCompactionQuery(HiveConf,
 Table, Partition, String) concatenates strings using + in a loop  At 
CompactorMR.java:strings using + in a loop  At CompactorMR.java:[line 533] |
|  |  
org.apache.hadoop.hive.ql.udf.generic.GenericUDFValidateAcidSortOrder$WriteIdRowId
 defines compareTo(GenericUDFValidateAcidSortOrder$WriteIdRowId) and uses 
Object.equals()  At GenericUDFValidateAcidSortOrder.java:Object.equals()  At 
GenericUDFValidateAcidSortOrder.java:[lines 85-91] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15728/dev-support/hive-personality.sh
 |
| git revision | master / cb74a68 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-15728/yetus/diff-checkstyle-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-15728/yetus/diff-checkstyle-itests_hive-unit.txt
 |
| whitespace |

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-01-18 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746954#comment-16746954
 ] 

Hive QA commented on HIVE-20699:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12955475/HIVE-20699.6.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/15700/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15700/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15700/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL 
https://issues.apache.org/jira/secure/attachment/12955475/HIVE-20699.6.patch 
was found in seen patch url's cache and a test was probably run already on it. 
Aborting...
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12955475 - PreCommit-HIVE-Build

> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch, 
> HIVE-20699.2.patch, HIVE-20699.3.patch, HIVE-20699.4.patch, 
> HIVE-20699.5.patch, HIVE-20699.6.patch
>
>
> Currently the Acid compactor is implemented as generated MR job 
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition 
> and writes data back to the same partition.  This will merge the deltas and 
> 'apply' the delete events.  The simplest would be to just use Insert 
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new 
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-01-18 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746931#comment-16746931
 ] 

Hive QA commented on HIVE-20699:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12955475/HIVE-20699.6.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15700 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.txn.compactor.TestCrudCompactorOnTez.testMajorCompaction
 (batchId=243)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/15698/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15698/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15698/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12955475 - PreCommit-HIVE-Build

> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch, 
> HIVE-20699.2.patch, HIVE-20699.3.patch, HIVE-20699.4.patch, 
> HIVE-20699.5.patch, HIVE-20699.6.patch
>
>
> Currently the Acid compactor is implemented as generated MR job 
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition 
> and writes data back to the same partition.  This will merge the deltas and 
> 'apply' the delete events.  The simplest would be to just use Insert 
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new 
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-01-18 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746928#comment-16746928
 ] 

Hive QA commented on HIVE-20699:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
31s{color} | {color:blue} common in master has 65 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
48s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
39s{color} | {color:red} ql: The patch generated 18 new + 520 unchanged - 1 
fixed = 538 total (was 521) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
17s{color} | {color:red} itests/hive-unit: The patch generated 24 new + 184 
unchanged - 0 fixed = 208 total (was 184) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 24 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  3m 
46s{color} | {color:red} ql generated 2 new + 2310 unchanged - 0 fixed = 2312 
total (was 2310) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
53s{color} | {color:red} ql generated 3 new + 97 unchanged - 3 fixed = 100 
total (was 100) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
12s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 50s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  
org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.buildCrudMajorCompactionQuery(HiveConf,
 Table, Partition, String) concatenates strings using + in a loop  At 
CompactorMR.java:strings using + in a loop  At CompactorMR.java:[line 533] |
|  |  
org.apache.hadoop.hive.ql.udf.generic.GenericUDFValidateAcidSortOrder$WriteIdRowId
 defines compareTo(GenericUDFValidateAcidSortOrder$WriteIdRowId) and uses 
Object.equals()  At GenericUDFValidateAcidSortOrder.java:Object.equals()  At 
GenericUDFValidateAcidSortOrder.java:[lines 85-91] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15698/dev-support/hive-personality.sh
 |
| git revision | master / a712978 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-15698/yetus/diff-checkstyle-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-15698/yetus/diff-checkstyle-itests_hive-unit.txt
 |
| whitespace |

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-01-11 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740981#comment-16740981
 ] 

Hive QA commented on HIVE-20699:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12954634/HIVE-20699.5.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/15602/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15602/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15602/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-01-12 02:57:53.506
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-15602/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-01-12 02:57:53.509
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at a3aa074 HIVE-21113: For HPL/SQL that contains boolean expression 
with NOT, incorrect SQL may be generated (Baoning He, reviewed by Daniel Dai)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at a3aa074 HIVE-21113: For HPL/SQL that contains boolean expression 
with NOT, incorrect SQL may be generated (Baoning He, reviewed by Daniel Dai)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-01-12 02:57:54.161
+ rm -rf ../yetus_PreCommit-HIVE-Build-15602
+ mkdir ../yetus_PreCommit-HIVE-Build-15602
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-15602
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-15602/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: does not 
exist in index
error: 
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java: 
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java: does 
not exist in index
error: 
a/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveSplitGenerator.java: does 
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/SplitGrouper.java: does 
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java: 
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java: 
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java: does not 
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java: 
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java: 
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java: 
does not exist in index
error: a/ql/src/test/results/clientpositive/show_functions.q.out: does not 
exist in index
error: patch failed: 
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java:375
Falling back to three-way merge...
Applied patch to 
'ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java' with 
conflicts.
Going to apply patch with: git apply -p1
/data/hiveptest/working/scratch/build.patch:12: trailing whitespace.
SPLIT_GROUPING_MODE("hive.split.grouping.mode", "query", new 
StringSet("query", "compactor"), 
/data/hiveptest/working/scratch/build.patch:33: trailing whitespace.
 
/data/hiveptest/working/scratch/build.patch:359: trailing whitespace.
  
/data/hiveptest/working/scratch/build.patch:583: trailing whitespace.

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-01-11 Thread Vaibhav Gumashta (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740760#comment-16740760
 ] 

Vaibhav Gumashta commented on HIVE-20699:
-

The failed test is due to the new validate_acid_sort_order udf function not 
showing up in the.out for show functions. Fixing the output.

> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch, 
> HIVE-20699.2.patch, HIVE-20699.3.patch, HIVE-20699.4.patch
>
>
> Currently the Acid compactor is implemented as generated MR job 
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition 
> and writes data back to the same partition.  This will merge the deltas and 
> 'apply' the delete events.  The simplest would be to just use Insert 
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new 
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-01-11 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740747#comment-16740747
 ] 

Hive QA commented on HIVE-20699:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12954613/HIVE-20699.4.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15716 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[show_functions] 
(batchId=79)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/15596/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15596/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15596/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12954613 - PreCommit-HIVE-Build

> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch, 
> HIVE-20699.2.patch, HIVE-20699.3.patch, HIVE-20699.4.patch
>
>
> Currently the Acid compactor is implemented as generated MR job 
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition 
> and writes data back to the same partition.  This will merge the deltas and 
> 'apply' the delete events.  The simplest would be to just use Insert 
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new 
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-01-11 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740714#comment-16740714
 ] 

Hive QA commented on HIVE-20699:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
32s{color} | {color:blue} common in master has 65 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
39s{color} | {color:blue} ql in master has 2309 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
41s{color} | {color:red} ql: The patch generated 18 new + 512 unchanged - 1 
fixed = 530 total (was 513) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
17s{color} | {color:red} itests/hive-unit: The patch generated 24 new + 184 
unchanged - 0 fixed = 208 total (was 184) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 24 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  3m 
57s{color} | {color:red} ql generated 2 new + 2309 unchanged - 0 fixed = 2311 
total (was 2309) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
53s{color} | {color:red} ql generated 3 new + 97 unchanged - 3 fixed = 100 
total (was 100) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
13s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 57s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  
org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.buildCrudMajorCompactionQuery(HiveConf,
 Table, Partition, String) concatenates strings using + in a loop  At 
CompactorMR.java:strings using + in a loop  At CompactorMR.java:[line 530] |
|  |  
org.apache.hadoop.hive.ql.udf.generic.GenericUDFValidateAcidSortOrder$WriteIdRowId
 defines compareTo(GenericUDFValidateAcidSortOrder$WriteIdRowId) and uses 
Object.equals()  At GenericUDFValidateAcidSortOrder.java:Object.equals()  At 
GenericUDFValidateAcidSortOrder.java:[lines 85-91] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15596/dev-support/hive-personality.sh
 |
| git revision | master / f713140 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-15596/yetus/diff-checkstyle-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-15596/yetus/diff-checkstyle-itests_hive-unit.txt
 |
| whitespace |

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2019-01-10 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739856#comment-16739856
 ] 

Hive QA commented on HIVE-20699:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12954471/HIVE-20699.3.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/15582/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15582/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15582/

Messages:
{noformat}
 This message was trimmed, see log for full details 
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/jetty-xml/9.3.25.v20180904/jetty-xml-9.3.25.v20180904.jar(org/eclipse/jetty/xml/XmlConfiguration.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/slf4j/jul-to-slf4j/1.7.10/jul-to-slf4j-1.7.10.jar(org/slf4j/bridge/SLF4JBridgeHandler.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/DispatcherType.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/Filter.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/FilterChain.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/FilterConfig.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/ServletException.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/ServletRequest.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/ServletResponse.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/annotation/WebFilter.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/http/HttpServletRequest.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar(javax/servlet/http/HttpServletResponse.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/classification/target/hive-classification-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/common/classification/InterfaceAudience$LimitedPrivate.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/classification/target/hive-classification-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/common/classification/InterfaceStability$Unstable.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/ByteArrayOutputStream.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/OutputStream.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/Closeable.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/AutoCloseable.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/Flushable.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(javax/xml/bind/annotation/XmlRootElement.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/commons/commons-exec/1.1/commons-exec-1.1.jar(org/apache/commons/exec/ExecuteException.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/security/PrivilegedExceptionAction.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/ExecutionException.class)]]
[loading 
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/TimeoutException.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/fs/FileSystem.class)]]
[loading 
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/HadoopShimsSecure.class)]]
[loading

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2018-11-19 Thread Vaibhav Gumashta (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691600#comment-16691600
 ] 

Vaibhav Gumashta commented on HIVE-20699:
-

Patch with review comments addressed minus the tests. V3 will have all the test 
cases.

> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch, 
> HIVE-20699.2.patch
>
>
> Currently the Acid compactor is implemented as generated MR job 
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition 
> and writes data back to the same partition.  This will merge the deltas and 
> 'apply' the delete events.  The simplest would be to just use Insert 
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new 
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2018-11-14 Thread Vaibhav Gumashta (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686179#comment-16686179
 ] 

Vaibhav Gumashta commented on HIVE-20699:
-

[~ekoifman] This is a partial patch for review. I'll attach another one with 
test cases. This depends on ORC-437. 

> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch
>
>
> Currently the Acid compactor is implemented as generated MR job 
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition 
> and writes data back to the same partition.  This will merge the deltas and 
> 'apply' the delete events.  The simplest would be to just use Insert 
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new 
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables

2018-10-05 Thread Andrew Sherman (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640317#comment-16640317
 ] 

Andrew Sherman commented on HIVE-20699:
---

Hi [~ekoifman] can you add a description please?

> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
>  Issue Type: New Feature
>  Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

37 matches

Mail list logo