[jira] [Commented] (HIVE-15074) Schematool provides a way to detect invalid entries in VERSION table

2016-12-12 Thread Lefty Leverenz (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-15074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15744480#comment-15744480
 ] 

Lefty Leverenz commented on HIVE-15074:
---

Okay, I added a TODOC2.2 label.  Thanks.

> Schematool provides a way to detect invalid entries in VERSION table
> 
>
> Key: HIVE-15074
> URL: https://issues.apache.org/jira/browse/HIVE-15074
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Yongzhi Chen
>Assignee: Chaoyu Tang
>Priority: Minor
>  Labels: TODOC2.2
> Fix For: 2.2.0
>
> Attachments: HIVE-15074.1.patch, HIVE-15074.patch
>
>
> For some unknown reason, we see customer's HMS can not start because there 
> are multiple entries in their HMS VERSION table. Schematool should provide a 
> way to validate the HMS db and provide warning and fix options for this kind 
> of issues. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15074) Schematool provides a way to detect invalid entries in VERSION table

2016-12-12 Thread Aihua Xu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-15074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15742167#comment-15742167
 ] 

Aihua Xu commented on HIVE-15074:
-

Yes. It should get documented in SchemaTool section. I will try to doc it after 
the child tasks are done and doc it along with HIVE-15206. Thanks [~leftylev] 
for reminding.

> Schematool provides a way to detect invalid entries in VERSION table
> 
>
> Key: HIVE-15074
> URL: https://issues.apache.org/jira/browse/HIVE-15074
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Yongzhi Chen
>Assignee: Chaoyu Tang
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-15074.1.patch, HIVE-15074.patch
>
>
> For some unknown reason, we see customer's HMS can not start because there 
> are multiple entries in their HMS VERSION table. Schematool should provide a 
> way to validate the HMS db and provide warning and fix options for this kind 
> of issues. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15074) Schematool provides a way to detect invalid entries in VERSION table

2016-12-10 Thread Lefty Leverenz (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-15074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15737508#comment-15737508
 ] 

Lefty Leverenz commented on HIVE-15074:
---

Does this need to be documented in the wiki?

* [The Hive Schema Tool | 
https://cwiki.apache.org/confluence/display/Hive/Hive+Schema+Tool#HiveSchemaTool-TheHiveSchemaTool]

> Schematool provides a way to detect invalid entries in VERSION table
> 
>
> Key: HIVE-15074
> URL: https://issues.apache.org/jira/browse/HIVE-15074
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Yongzhi Chen
>Assignee: Chaoyu Tang
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-15074.1.patch, HIVE-15074.patch
>
>
> For some unknown reason, we see customer's HMS can not start because there 
> are multiple entries in their HMS VERSION table. Schematool should provide a 
> way to validate the HMS db and provide warning and fix options for this kind 
> of issues. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15074) Schematool provides a way to detect invalid entries in VERSION table

2016-12-01 Thread Aihua Xu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-15074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15713034#comment-15713034
 ] 

Aihua Xu commented on HIVE-15074:
-

The patch looks good. + 1.

> Schematool provides a way to detect invalid entries in VERSION table
> 
>
> Key: HIVE-15074
> URL: https://issues.apache.org/jira/browse/HIVE-15074
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Yongzhi Chen
>Assignee: Chaoyu Tang
>Priority: Minor
> Attachments: HIVE-15074.1.patch, HIVE-15074.patch
>
>
> For some unknown reason, we see customer's HMS can not start because there 
> are multiple entries in their HMS VERSION table. Schematool should provide a 
> way to validate the HMS db and provide warning and fix options for this kind 
> of issues. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15074) Schematool provides a way to detect invalid entries in VERSION table

2016-11-29 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-15074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15706741#comment-15706741
 ] 

Hive QA commented on HIVE-15074:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12840920/HIVE-15074.1.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10747 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
 (batchId=133)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2] 
(batchId=133)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] 
(batchId=91)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2330/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2330/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2330/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12840920 - PreCommit-HIVE-Build

> Schematool provides a way to detect invalid entries in VERSION table
> 
>
> Key: HIVE-15074
> URL: https://issues.apache.org/jira/browse/HIVE-15074
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Yongzhi Chen
>Assignee: Chaoyu Tang
>Priority: Minor
> Attachments: HIVE-15074.1.patch, HIVE-15074.patch
>
>
> For some unknown reason, we see customer's HMS can not start because there 
> are multiple entries in their HMS VERSION table. Schematool should provide a 
> way to validate the HMS db and provide warning and fix options for this kind 
> of issues. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15074) Schematool provides a way to detect invalid entries in VERSION table

2016-11-29 Thread Chaoyu Tang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-15074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15706688#comment-15706688
 ] 

Chaoyu Tang commented on HIVE-15074:


The failed tests are not related to this patch.

> Schematool provides a way to detect invalid entries in VERSION table
> 
>
> Key: HIVE-15074
> URL: https://issues.apache.org/jira/browse/HIVE-15074
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Yongzhi Chen
>Assignee: Chaoyu Tang
>Priority: Minor
> Attachments: HIVE-15074.1.patch, HIVE-15074.patch
>
>
> For some unknown reason, we see customer's HMS can not start because there 
> are multiple entries in their HMS VERSION table. Schematool should provide a 
> way to validate the HMS db and provide warning and fix options for this kind 
> of issues. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15074) Schematool provides a way to detect invalid entries in VERSION table

2016-11-29 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-15074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15706625#comment-15706625
 ] 

Hive QA commented on HIVE-15074:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12840920/HIVE-15074.1.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10747 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2] 
(batchId=133)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2329/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2329/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2329/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12840920 - PreCommit-HIVE-Build

> Schematool provides a way to detect invalid entries in VERSION table
> 
>
> Key: HIVE-15074
> URL: https://issues.apache.org/jira/browse/HIVE-15074
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Yongzhi Chen
>Assignee: Chaoyu Tang
>Priority: Minor
> Attachments: HIVE-15074.1.patch, HIVE-15074.patch
>
>
> For some unknown reason, we see customer's HMS can not start because there 
> are multiple entries in their HMS VERSION table. Schematool should provide a 
> way to validate the HMS db and provide warning and fix options for this kind 
> of issues. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15074) Schematool provides a way to detect invalid entries in VERSION table

2016-11-29 Thread Chaoyu Tang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-15074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15705839#comment-15705839
 ] 

Chaoyu Tang commented on HIVE-15074:


Should not we stop the validations if there are some schema version issues 
(incompatible, duplicated invalid entries, no version etc)? Otherwise, the 
following schema validations, especially those based on the version like 
validateSchemaTables, will become meaningless since the based schema is already 
not right. I can put the validateSchemaVersions as the first validation point. 
Any thought? 

> Schematool provides a way to detect invalid entries in VERSION table
> 
>
> Key: HIVE-15074
> URL: https://issues.apache.org/jira/browse/HIVE-15074
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Yongzhi Chen
>Assignee: Chaoyu Tang
>Priority: Minor
> Attachments: HIVE-15074.patch
>
>
> For some unknown reason, we see customer's HMS can not start because there 
> are multiple entries in their HMS VERSION table. Schematool should provide a 
> way to validate the HMS db and provide warning and fix options for this kind 
> of issues. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15074) Schematool provides a way to detect invalid entries in VERSION table

2016-11-29 Thread Aihua Xu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-15074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15705406#comment-15705406
 ] 

Aihua Xu commented on HIVE-15074:
-

That's right. We are trying to return boolean rather than throwing exceptions 
since we will continue the following functions to detect other issues. 

> Schematool provides a way to detect invalid entries in VERSION table
> 
>
> Key: HIVE-15074
> URL: https://issues.apache.org/jira/browse/HIVE-15074
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Yongzhi Chen
>Assignee: Chaoyu Tang
>Priority: Minor
> Attachments: HIVE-15074.patch
>
>
> For some unknown reason, we see customer's HMS can not start because there 
> are multiple entries in their HMS VERSION table. Schematool should provide a 
> way to validate the HMS db and provide warning and fix options for this kind 
> of issues. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15074) Schematool provides a way to detect invalid entries in VERSION table

2016-11-28 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-15074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15703834#comment-15703834
 ] 

Hive QA commented on HIVE-15074:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12840735/HIVE-15074.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2312/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2312/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2312/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2016-11-29 01:44:23.550
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-2312/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2016-11-29 01:44:23.553
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 63bdfa6 HIVE-15284: Add junit test to test replication scenarios 
(Sushanth Sowmyan reviewed by Vaibhav Gumashta)
+ git clean -f -d
Removing ql/src/test/queries/clientpositive/specialChar.q
Removing ql/src/test/results/clientpositive/specialChar.q.out
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 63bdfa6 HIVE-15284: Add junit test to test replication scenarios 
(Sushanth Sowmyan reviewed by Vaibhav Gumashta)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2016-11-29 01:44:24.497
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: a/beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java: No such 
file or directory
error: 
a/itests/hive-unit/src/test/java/org/apache/hive/beeline/TestSchemaTool.java: 
No such file or directory
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12840735 - PreCommit-HIVE-Build

> Schematool provides a way to detect invalid entries in VERSION table
> 
>
> Key: HIVE-15074
> URL: https://issues.apache.org/jira/browse/HIVE-15074
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Yongzhi Chen
>Assignee: Chaoyu Tang
>Priority: Minor
> Attachments: HIVE-15074.patch
>
>
> For some unknown reason, we see customer's HMS can not start because there 
> are multiple entries in their HMS VERSION table. Schematool should provide a 
> way to validate the HMS db and provide warning and fix options for this kind 
> of issues. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-15074) Schematool provides a way to detect invalid entries in VERSION table

2016-11-28 Thread Naveen Gangam (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-15074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15703362#comment-15703362
 ] 

Naveen Gangam commented on HIVE-15074:
--

Functionally, it looks good to me. 

Just for semantics to make it consistent with other verify* calls, do we want 
to make it return a boolean to indicate success/failure instead of throwing an 
exception? maybe overload {{verifySchemaVersion()}} that catches the exception 
and returns a boolean? I will defer to [~aihuaxu] for the final call on this.

> Schematool provides a way to detect invalid entries in VERSION table
> 
>
> Key: HIVE-15074
> URL: https://issues.apache.org/jira/browse/HIVE-15074
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Yongzhi Chen
>Assignee: Chaoyu Tang
>Priority: Minor
> Attachments: HIVE-15074.patch
>
>
> For some unknown reason, we see customer's HMS can not start because there 
> are multiple entries in their HMS VERSION table. Schematool should provide a 
> way to validate the HMS db and provide warning and fix options for this kind 
> of issues. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)