[jira] [Commented] (HIVE-16045) Print progress bar along with operation log

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885282#comment-15885282
 ] 

Thejas M Nair commented on HIVE-16045:
--

+1 to changes in pull request.
Please upload updated patch here as well.


> Print progress bar along with operation log
> ---
>
> Key: HIVE-16045
> URL: https://issues.apache.org/jira/browse/HIVE-16045
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 2.2.0
>Reporter: anishek
>Assignee: anishek
> Fix For: 2.2.0
>
> Attachments: HIVE-16045.1.patch, HIVE-16045.2.patch
>
>
> allow printing of the operation logs and progress bar such that,
> allow operations logs to output data once -> block it -> start progress bar 
> -> finish progress bar -> unblock the operations log -> finish operations log 
> -> print query results. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16045) Print progress bar along with operation log

[anishek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

anishek updated HIVE-16045:
---
Attachment: HIVE-16045.2.patch

> Print progress bar along with operation log
> ---
>
> Key: HIVE-16045
> URL: https://issues.apache.org/jira/browse/HIVE-16045
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 2.2.0
>Reporter: anishek
>Assignee: anishek
> Fix For: 2.2.0
>
> Attachments: HIVE-16045.1.patch, HIVE-16045.2.patch
>
>
> allow printing of the operation logs and progress bar such that,
> allow operations logs to output data once -> block it -> start progress bar 
> -> finish progress bar -> unblock the operations log -> finish operations log 
> -> print query results. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (HIVE-16045) Print progress bar along with operation log

[anishek (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885246#comment-15885246
 ] 

anishek edited comment on HIVE-16045 at 2/27/17 7:17 AM:
-

sample output from the beeline after patch

{code}
0: jdbc:hive2://localhost:10007/default> select max(age) from t;
INFO  : Compiling 
command(queryId=root_20170227071717_0810070a-71bc-450b-b88d-fbd505e38d4b): 
select max(age) from t
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, 
type:bigint, comment:null)], properties:null)
INFO  : Completed compiling 
command(queryId=root_20170227071717_0810070a-71bc-450b-b88d-fbd505e38d4b); Time 
taken: 3.994 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing 
command(queryId=root_20170227071717_0810070a-71bc-450b-b88d-fbd505e38d4b): 
select max(age) from t
INFO  : Query ID = root_20170227071717_0810070a-71bc-450b-b88d-fbd505e38d4b
INFO  : Total jobs = 1
INFO  : Launching Job 1 out of 1
INFO  : Starting task [Stage-1:MAPRED] in serial mode
INFO  : Session is already open
INFO  : Dag name: select max(age) from t(Stage-1)
INFO  : Status: Running (Executing on YARN cluster with App id 
application_1486985293385_0278)

INFO  : Map 1: 0/1  Reducer 2: 0/1
--
VERTICES  MODESTATUS  TOTAL  COMPLETED  RUNNING  PENDING  
FAILED  KILLED
--
Map 1 .. container SUCCEEDED  1  100
   0   0
Reducer 2 .. container SUCCEEDED  1  100
   0   0
--
VERTICES: 02/02  [==>>] 100%  ELAPSED TIME: 4.48 s
--
INFO  : Map 1: 0(+1)/1  Reducer 2: 0/1
INFO  : Map 1: 1/1  Reducer 2: 0(+1)/1
INFO  : Map 1: 1/1  Reducer 2: 1/1
INFO  : Status: DAG finished successfully in 4.35 seconds
INFO  :
INFO  : Query Execution Summary
INFO  : 
--
INFO  : OPERATIONDURATION
INFO  : 
--
INFO  : Compile Query   3.99s
INFO  : Prepare Plan0.64s
INFO  : Submit Plan 0.43s
INFO  : Start DAG   0.61s
INFO  : Run DAG 4.35s
INFO  : 
--
INFO  :
INFO  : Task Execution Summary
INFO  : 
--
INFO  :   VERTICES  DURATION(ms)   CPU_TIME(ms)GC_TIME(ms)   
INPUT_RECORDS   OUTPUT_RECORDS
INFO  : 
--
INFO  :  Map 1   2061.00  3,920 26  
 51
INFO  :  Reducer 2429.00  1,690  0  
 10
INFO  : 
--
INFO  :
INFO  : LLAP IO Summary
INFO  : 
--
INFO  :   VERTICES ROWGROUPS  META_HIT  META_MISS  DATA_HIT  DATA_MISS  
ALLOCATION USED  TOTAL_IO
INFO  : 
--
INFO  :  Map 1 0 0  00B 0B  
0B   0B 0.00s
INFO  : 
--
INFO  :
INFO  : FileSystem Counters Summary
INFO  :
INFO  : Scheme: FILE
INFO  : 
--
INFO  :   VERTICES  BYTES_READ  READ_OPS LARGE_READ_OPS  
BYTES_WRITTEN WRITE_OPS
INFO  : 
--
INFO  :  Map 1  0B 0  0 
   59B 0
INFO  :  Reducer 2  0B 0  0 
0B 0
INFO  : 
--
INFO  :
INFO  : Scheme: HDFS
INFO  : 

[jira] [Commented] (HIVE-16045) Print progress bar along with operation log

[anishek (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885246#comment-15885246
 ] 

anishek commented on HIVE-16045:


sample output from the beeline after patch

{code}
0: jdbc:hive2://localhost:10007/default> select max(age) from t;
INFO  : Compiling 
command(queryId=root_20170227064051_dab7f2ad-0009-44c2-b59b-e55179bc6eae): 
select max(age) from t
INFO  : We are setting the hadoop caller context from 
HIVE_SSN_ID:5d59ff69-3f21-48af-a688-9855b4052901 to 
root_20170227064051_dab7f2ad-0009-44c2-b59b-e55179bc6eae
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, 
type:bigint, comment:null)], properties:null)
INFO  : Completed compiling 
command(queryId=root_20170227064051_dab7f2ad-0009-44c2-b59b-e55179bc6eae); Time 
taken: 3.78 seconds
INFO  : We are resetting the hadoop caller context to 
HIVE_SSN_ID:5d59ff69-3f21-48af-a688-9855b4052901
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Setting caller context to query id 
root_20170227064051_dab7f2ad-0009-44c2-b59b-e55179bc6eae
INFO  : Executing 
command(queryId=root_20170227064051_dab7f2ad-0009-44c2-b59b-e55179bc6eae): 
select max(age) from t
INFO  : Query ID = root_20170227064051_dab7f2ad-0009-44c2-b59b-e55179bc6eae
INFO  : Total jobs = 1
INFO  : Launching Job 1 out of 1
INFO  : Starting task [Stage-1:MAPRED] in serial mode
INFO  : Session is already open
INFO  : Dag name: select max(age) from t(Stage-1)
INFO  : Status: Running (Executing on YARN cluster with App id 
application_1486985293385_0274)

INFO  : Map 1: 0/1  Reducer 2: 0/1
--
VERTICES  MODESTATUS  TOTAL  COMPLETED  RUNNING  PENDING  
FAILED  KILLED
--
Map 1 .. container SUCCEEDED  1  100
   0   0
Reducer 2 .. container SUCCEEDED  1  100
   0   0
VERTICES: 02/02  [==>>] 100%  ELAPSED TIME: 3.93 s
--
INFO  : Map 1: 0(+1)/1  Reducer 2: 0/1
INFO  : Map 1: 1/1  Reducer 2: 0/1
INFO  : Map 1: 1/1  Reducer 2: 0(+1)/1
INFO  : Map 1: 1/1  Reducer 2: 1/1
INFO  : Status: DAG finished successfully in 3.83 seconds
INFO  :
INFO  : Query Execution Summary
INFO  : 
--
INFO  : OPERATIONDURATION
INFO  : 
--
INFO  : Compile Query   3.78s
INFO  : Prepare Plan0.70s
INFO  : Submit Plan 0.40s
INFO  : Start DAG   0.59s
INFO  : Run DAG 3.83s
INFO  : 
--
INFO  :
INFO  : Task Execution Summary
INFO  : 

INFO  :   VERTICES  TOTAL_TASKS  FAILED_ATTEMPTS  KILLED_TASKS   DURATION(ms)  
CPU_TIME(ms)  GC_TIME(ms)  INPUT_RECORDS  OUTPUT_RECORDS
INFO  : 

INFO  :  Map 110 01983.00   
  3,670   26  5   1
INFO  :  Reducer 210 0 347.00   
  1,7800  1   0
INFO  : 

INFO  :
INFO  : LLAP IO Summary
INFO  : 
--
INFO  :   VERTICES ROWGROUPS  META_HIT  META_MISS  DATA_HIT  DATA_MISS  
ALLOCATION USED  TOTAL_IO
INFO  : 
--
INFO  :  Map 1 0 0  00B 0B  
0B   0B 0.00s
INFO  : 
--
INFO  :
INFO  : FileSystem Counters Summary
INFO  :
INFO  : Scheme: FILE
INFO  : 
--
INFO  :   VERTICES  BYTES_READ  READ_OPS LARGE_READ_OPS  
BYTES_WRITTEN WRITE_OPS
INFO  : 

[jira] [Updated] (HIVE-16045) Print progress bar along with operation log

[anishek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

anishek updated HIVE-16045:
---
Status: Patch Available  (was: Open)

> Print progress bar along with operation log
> ---
>
> Key: HIVE-16045
> URL: https://issues.apache.org/jira/browse/HIVE-16045
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 2.2.0
>Reporter: anishek
>Assignee: anishek
> Fix For: 2.2.0
>
> Attachments: HIVE-16045.1.patch
>
>
> allow printing of the operation logs and progress bar such that,
> allow operations logs to output data once -> block it -> start progress bar 
> -> finish progress bar -> unblock the operations log -> finish operations log 
> -> print query results. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16045) Print progress bar along with operation log

[anishek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

anishek updated HIVE-16045:
---
Attachment: HIVE-16045.1.patch

> Print progress bar along with operation log
> ---
>
> Key: HIVE-16045
> URL: https://issues.apache.org/jira/browse/HIVE-16045
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 2.2.0
>Reporter: anishek
>Assignee: anishek
> Fix For: 2.2.0
>
> Attachments: HIVE-16045.1.patch
>
>
> allow printing of the operation logs and progress bar such that,
> allow operations logs to output data once -> block it -> start progress bar 
> -> finish progress bar -> unblock the operations log -> finish operations log 
> -> print query results. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Work started] (HIVE-16045) Print progress bar along with operation log

[anishek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-16045 started by anishek.
--
> Print progress bar along with operation log
> ---
>
> Key: HIVE-16045
> URL: https://issues.apache.org/jira/browse/HIVE-16045
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 2.2.0
>Reporter: anishek
>Assignee: anishek
> Fix For: 2.2.0
>
> Attachments: HIVE-16045.1.patch
>
>
> allow printing of the operation logs and progress bar such that,
> allow operations logs to output data once -> block it -> start progress bar 
> -> finish progress bar -> unblock the operations log -> finish operations log 
> -> print query results. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Work stopped] (HIVE-16045) Print progress bar along with operation log

[anishek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-16045 stopped by anishek.
--
> Print progress bar along with operation log
> ---
>
> Key: HIVE-16045
> URL: https://issues.apache.org/jira/browse/HIVE-16045
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 2.2.0
>Reporter: anishek
>Assignee: anishek
> Fix For: 2.2.0
>
> Attachments: HIVE-16045.1.patch
>
>
> allow printing of the operation logs and progress bar such that,
> allow operations logs to output data once -> block it -> start progress bar 
> -> finish progress bar -> unblock the operations log -> finish operations log 
> -> print query results. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16045) Print progress bar along with operation log

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885244#comment-15885244
 ] 

ASF GitHub Bot commented on HIVE-16045:
---

GitHub user anishek opened a pull request:

https://github.com/apache/hive/pull/153

HIVE-16045: Print progress bar along with operation log



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/anishek/hive HIVE-16045

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/153.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #153


commit 6d83061adc001ea8a7ec76a5b8b10e16d86da400
Author: Anishek Agarwal 
Date:   2017-02-27T06:36:13Z

HIVE-16045: Print progress bar along with operation log




> Print progress bar along with operation log
> ---
>
> Key: HIVE-16045
> URL: https://issues.apache.org/jira/browse/HIVE-16045
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 2.2.0
>Reporter: anishek
>Assignee: anishek
> Fix For: 2.2.0
>
>
> allow printing of the operation logs and progress bar such that,
> allow operations logs to output data once -> block it -> start progress bar 
> -> finish progress bar -> unblock the operations log -> finish operations log 
> -> print query results. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16045) Print progress bar along with operation log

[anishek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

anishek reassigned HIVE-16045:
--


> Print progress bar along with operation log
> ---
>
> Key: HIVE-16045
> URL: https://issues.apache.org/jira/browse/HIVE-16045
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 2.2.0
>Reporter: anishek
>Assignee: anishek
> Fix For: 2.2.0
>
>
> allow printing of the operation logs and progress bar such that,
> allow operations logs to output data once -> block it -> start progress bar 
> -> finish progress bar -> unblock the operations log -> finish operations log 
> -> print query results. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16042) some sql file with comment encounter ParseException

[KaiXu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885190#comment-15885190
 ] 

KaiXu commented on HIVE-16042:
--

Hi [~pxiong], can you point out where is the problem? if this's the cause, why 
previous Hive(commit: ac68aed) is ok?

> some sql file with comment encounter ParseException
> ---
>
> Key: HIVE-16042
> URL: https://issues.apache.org/jira/browse/HIVE-16042
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 2.2.0
> Environment: Hive2.2 (commit: 2768361)
> TPCx-BB v1.2
>Reporter: KaiXu
> Attachments: q04.sql, q17.sql, q18.sql, q23.sql
>
>
> current Hive upstream(commit: 2768361) failed to parse some 
> queries(q04,q17,q18,q23) in TPCx-BB v1.2, while it's ok with Hive(commit: 
> ac68aed).
> Q04: FAILED: ParseException line 24:0 missing EOF at ';' near 
> 'abandonedShoppingCartsPageCountsPerSession'
> Q17:
> NoViableAltException(350@[])
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.limitClause(HiveParser.java:38898)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:37002)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:36404)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:35722)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:35610)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:2279)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1328)
> at 
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:204)
> at 
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:75)
> at 
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:68)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:468)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
> at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
> at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
> at 
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:474)
> at 
> org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:490)
> at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> FAILED: ParseException line 39:0 cannot recognize input near 'LIMIT' '100' 
> ';' in limit clause
> Q18:
> NoViableAltException(350@[()* loopback of 424:20: ( ( LSQUARE ^ expression 
> RSQUARE !) | ( DOT ^ identifier ) )*])
> at org.antlr.runtime.DFA.noViableAlt(DFA.java:158)
> at org.antlr.runtime.DFA.predict(DFA.java:116)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6665)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6992)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:7048)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7210)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7353)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7496)
> at 
> 

[jira] [Assigned] (HIVE-15939) Make cast expressions comply more to sql2011

[Teddy Choi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Teddy Choi reassigned HIVE-15939:
-

Assignee: Teddy Choi

> Make cast expressions comply more to sql2011
> 
>
> Key: HIVE-15939
> URL: https://issues.apache.org/jira/browse/HIVE-15939
> Project: Hive
>  Issue Type: Sub-task
>  Components: SQL
>Reporter: Zoltan Haindrich
>Assignee: Teddy Choi
>
> in HIVE-6590 Jason have uncovered the fact that UDFToBoolean treats all 
> non-empty strings as true.
> It would be great to have the cast expressions closer to the standard...at 
> least when there is an expected behaviour from the user;
> like {{cast('false' as boolean)}} should be true.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15881) Use new thread count variable name instead of mapred.dfsclient.parallelism.max

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885178#comment-15885178
 ] 

Hive QA commented on HIVE-15881:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12854812/HIVE-15881.6.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10268 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=140)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=223)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=211)
org.apache.hive.jdbc.TestJdbcDriver2.testSelectExecAsync2 (batchId=215)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3803/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3803/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3803/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12854812 - PreCommit-HIVE-Build

> Use new thread count variable name instead of mapred.dfsclient.parallelism.max
> --
>
> Key: HIVE-15881
> URL: https://issues.apache.org/jira/browse/HIVE-15881
> Project: Hive
>  Issue Type: Task
>  Components: Query Planning
>Reporter: Sergio Peña
>Assignee: Sergio Peña
>Priority: Minor
> Attachments: HIVE-15881.1.patch, HIVE-15881.2.patch, 
> HIVE-15881.3.patch, HIVE-15881.4.patch, HIVE-15881.5.patch, HIVE-15881.6.patch
>
>
> The Utilities class has two methods, {{getInputSummary}} and 
> {{getInputPaths}}, that use the variable {{mapred.dfsclient.parallelism.max}} 
> to get the summary of a list of input locations in parallel. These methods 
> are Hive related, but the variable name does not look it is specific for Hive.
> Also, the above variable is not on HiveConf nor used anywhere else. I just 
> found a reference on the Hadoop MR1 code.
> I'd like to propose the deprecation of {{mapred.dfsclient.parallelism.max}}, 
> and use a different variable name, such as 
> {{hive.get.input.listing.num.threads}}, that reflects the intention of the 
> variable. The removal of the old variable might happen on Hive 3.x



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16042) some sql file with comment encounter ParseException

[Pengcheng Xiong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885173#comment-15885173
 ] 

Pengcheng Xiong commented on HIVE-16042:


The comments should begin with "--" without any spaces. For example, this one 
has problem
{code}
INSERT INTO TABLE ${hiveconf:RESULT_TABLE}
SELECT SUM(pagecount) / COUNT(*)
FROM
(
  FROM
  (
SELECT *
FROM ${hiveconf:TEMP_TABLE1} sessions
DISTRIBUTE BY sessionid SORT BY sessionid, tstamp, wp_type --required by 
"abandonment analysis script"
  ) distributedSessions
  REDUCE 
wp_type,
--tstamp, --already sorted by time-stamp
sessionid --but we still need the sessionid within the script to identify 
session boundaries

-- script requires input tuples to be grouped by sessionid and ordered by 
timestamp ascending.
-- output one tuple:  if a session's shopping cart is abandoned, 
else: nothing
USING 'python q4_abandonedShoppingCarts.py'
AS (pagecount BIGINT)
) abandonedShoppingCartsPageCountsPerSession
;
{code}


> some sql file with comment encounter ParseException
> ---
>
> Key: HIVE-16042
> URL: https://issues.apache.org/jira/browse/HIVE-16042
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 2.2.0
> Environment: Hive2.2 (commit: 2768361)
> TPCx-BB v1.2
>Reporter: KaiXu
> Attachments: q04.sql, q17.sql, q18.sql, q23.sql
>
>
> current Hive upstream(commit: 2768361) failed to parse some 
> queries(q04,q17,q18,q23) in TPCx-BB v1.2, while it's ok with Hive(commit: 
> ac68aed).
> Q04: FAILED: ParseException line 24:0 missing EOF at ';' near 
> 'abandonedShoppingCartsPageCountsPerSession'
> Q17:
> NoViableAltException(350@[])
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.limitClause(HiveParser.java:38898)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:37002)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:36404)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:35722)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:35610)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:2279)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1328)
> at 
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:204)
> at 
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:75)
> at 
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:68)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:468)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
> at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
> at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
> at 
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:474)
> at 
> org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:490)
> at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> FAILED: ParseException line 39:0 cannot recognize input near 'LIMIT' '100' 
> ';' in limit clause
> Q18:
> NoViableAltException(350@[()* loopback of 424:20: ( ( LSQUARE ^ expression 
> RSQUARE !) | ( DOT ^ identifier ) )*])
> at org.antlr.runtime.DFA.noViableAlt(DFA.java:158)
> at org.antlr.runtime.DFA.predict(DFA.java:116)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6665)
> at 
> 

[jira] [Updated] (HIVE-16042) some sql file with comment encounter ParseException

[KaiXu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

KaiXu updated HIVE-16042:
-
Attachment: q23.sql
q18.sql
q17.sql
q04.sql

the 4 sql files in TPCx-BB v1.2

> some sql file with comment encounter ParseException
> ---
>
> Key: HIVE-16042
> URL: https://issues.apache.org/jira/browse/HIVE-16042
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 2.2.0
> Environment: Hive2.2 (commit: 2768361)
> TPCx-BB v1.2
>Reporter: KaiXu
> Attachments: q04.sql, q17.sql, q18.sql, q23.sql
>
>
> current Hive upstream(commit: 2768361) failed to parse some 
> queries(q04,q17,q18,q23) in TPCx-BB v1.2, while it's ok with Hive(commit: 
> ac68aed).
> Q04: FAILED: ParseException line 24:0 missing EOF at ';' near 
> 'abandonedShoppingCartsPageCountsPerSession'
> Q17:
> NoViableAltException(350@[])
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.limitClause(HiveParser.java:38898)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:37002)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:36404)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:35722)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:35610)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:2279)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1328)
> at 
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:204)
> at 
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:75)
> at 
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:68)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:468)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
> at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
> at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
> at 
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:474)
> at 
> org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:490)
> at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> FAILED: ParseException line 39:0 cannot recognize input near 'LIMIT' '100' 
> ';' in limit clause
> Q18:
> NoViableAltException(350@[()* loopback of 424:20: ( ( LSQUARE ^ expression 
> RSQUARE !) | ( DOT ^ identifier ) )*])
> at org.antlr.runtime.DFA.noViableAlt(DFA.java:158)
> at org.antlr.runtime.DFA.predict(DFA.java:116)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6665)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6992)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:7048)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7210)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7353)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7496)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceConcatenateExpression(HiveParser_IdentifiersParser.java:7635)
> at 
> 

[jira] [Updated] (HIVE-15859) HoS: Write RPC messages in event loop

[Rui Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Li updated HIVE-15859:
--
   Resolution: Fixed
Fix Version/s: 2.2.0
   Status: Resolved  (was: Patch Available)

Committed to master. Thanks Xuefu for the review.

> HoS: Write RPC messages in event loop
> -
>
> Key: HIVE-15859
> URL: https://issues.apache.org/jira/browse/HIVE-15859
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, Spark
>Affects Versions: 2.1.1
> Environment: hadoop2.7.1
> spark1.6.2
> hive2.2
>Reporter: KaiXu
>Assignee: Rui Li
> Fix For: 2.2.0
>
> Attachments: HIVE-15859.1.patch, HIVE-15859.2.patch, 
> HIVE-15859.3.patch
>
>
> Hive on Spark, failed with error:
> {noformat}
> 2017-02-08 09:50:59,331 Stage-2_0: 1039(+2)/1041 Stage-3_0: 796(+456)/1520 
> Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> 2017-02-08 09:51:00,335 Stage-2_0: 1040(+1)/1041 Stage-3_0: 914(+398)/1520 
> Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> 2017-02-08 09:51:01,338 Stage-2_0: 1041/1041 Finished Stage-3_0: 
> 961(+383)/1520 Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> Failed to monitor Job[ 2] with exception 'java.lang.IllegalStateException(RPC 
> channel is closed.)'
> FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.spark.SparkTask
> {noformat}
> application log shows the driver commanded a shutdown with some unknown 
> reason, but hive's log shows Driver could not get RPC header( Expected RPC 
> header, got org.apache.hive.spark.client.rpc.Rpc$NullMessage instead).
> {noformat}
> 17/02/08 09:51:04 INFO exec.Utilities: PLAN PATH = 
> hdfs://hsx-node1:8020/tmp/hive/root/b723c85d-2a7b-469e-bab1-9c165b25e656/hive_2017-02-08_09-49-37_890_6267025825539539056-1/-mr-10006/71a9dacb-a463-40ef-9e86-78d3b8e3738d/map.xml
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 1169.0 in 
> stage 3.0 (TID 2519)
> 17/02/08 09:51:04 INFO executor.CoarseGrainedExecutorBackend: Driver 
> commanded a shutdown
> 17/02/08 09:51:04 INFO storage.MemoryStore: MemoryStore cleared
> 17/02/08 09:51:04 INFO storage.BlockManager: BlockManager stopped
> 17/02/08 09:51:04 INFO exec.Utilities: PLAN PATH = 
> hdfs://hsx-node1:8020/tmp/hive/root/b723c85d-2a7b-469e-bab1-9c165b25e656/hive_2017-02-08_09-49-37_890_6267025825539539056-1/-mr-10006/71a9dacb-a463-40ef-9e86-78d3b8e3738d/map.xml
> 17/02/08 09:51:04 WARN executor.CoarseGrainedExecutorBackend: An unknown 
> (hsx-node1:42777) driver disconnected.
> 17/02/08 09:51:04 ERROR executor.CoarseGrainedExecutorBackend: Driver 
> 192.168.1.1:42777 disassociated! Shutting down.
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 1105.0 in 
> stage 3.0 (TID 2511)
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Shutdown hook called
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator: 
> Shutting down remote daemon.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk6/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-71da1dfc-99bd-4687-bc2f-33452db8de3d
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk2/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-7f134d81-e77e-4b92-bd99-0a51d0962c14
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk5/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-77a90d63-fb05-4bc6-8d5e-1562cc502e6c
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator: 
> Remote daemon shut down; proceeding with flushing remote transports.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk4/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-91f8b91a-114d-4340-8560-d3cd085c1cd4
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk1/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-a3c24f9e-8609-48f0-9d37-0de7ae06682a
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator: 
> Remoting shut down.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk7/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-f6120a43-2158-4780-927c-c5786b78f53e
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk3/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-e17931ad-9e8a-45da-86f8-9a0fdca0fad1
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk8/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-4de34175-f871-4c28-8ec0-d2fc0020c5c3
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 1137.0 in 
> stage 3.0 (TID 2515)
> 17/02/08 09:51:04 INFO 

[jira] [Updated] (HIVE-15859) HoS: Write RPC messages in event loop

[Rui Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Li updated HIVE-15859:
--
Affects Version/s: (was: 2.2.0)
   2.1.1

> HoS: Write RPC messages in event loop
> -
>
> Key: HIVE-15859
> URL: https://issues.apache.org/jira/browse/HIVE-15859
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, Spark
>Affects Versions: 2.1.1
> Environment: hadoop2.7.1
> spark1.6.2
> hive2.2
>Reporter: KaiXu
>Assignee: Rui Li
> Fix For: 2.2.0
>
> Attachments: HIVE-15859.1.patch, HIVE-15859.2.patch, 
> HIVE-15859.3.patch
>
>
> Hive on Spark, failed with error:
> {noformat}
> 2017-02-08 09:50:59,331 Stage-2_0: 1039(+2)/1041 Stage-3_0: 796(+456)/1520 
> Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> 2017-02-08 09:51:00,335 Stage-2_0: 1040(+1)/1041 Stage-3_0: 914(+398)/1520 
> Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> 2017-02-08 09:51:01,338 Stage-2_0: 1041/1041 Finished Stage-3_0: 
> 961(+383)/1520 Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> Failed to monitor Job[ 2] with exception 'java.lang.IllegalStateException(RPC 
> channel is closed.)'
> FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.spark.SparkTask
> {noformat}
> application log shows the driver commanded a shutdown with some unknown 
> reason, but hive's log shows Driver could not get RPC header( Expected RPC 
> header, got org.apache.hive.spark.client.rpc.Rpc$NullMessage instead).
> {noformat}
> 17/02/08 09:51:04 INFO exec.Utilities: PLAN PATH = 
> hdfs://hsx-node1:8020/tmp/hive/root/b723c85d-2a7b-469e-bab1-9c165b25e656/hive_2017-02-08_09-49-37_890_6267025825539539056-1/-mr-10006/71a9dacb-a463-40ef-9e86-78d3b8e3738d/map.xml
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 1169.0 in 
> stage 3.0 (TID 2519)
> 17/02/08 09:51:04 INFO executor.CoarseGrainedExecutorBackend: Driver 
> commanded a shutdown
> 17/02/08 09:51:04 INFO storage.MemoryStore: MemoryStore cleared
> 17/02/08 09:51:04 INFO storage.BlockManager: BlockManager stopped
> 17/02/08 09:51:04 INFO exec.Utilities: PLAN PATH = 
> hdfs://hsx-node1:8020/tmp/hive/root/b723c85d-2a7b-469e-bab1-9c165b25e656/hive_2017-02-08_09-49-37_890_6267025825539539056-1/-mr-10006/71a9dacb-a463-40ef-9e86-78d3b8e3738d/map.xml
> 17/02/08 09:51:04 WARN executor.CoarseGrainedExecutorBackend: An unknown 
> (hsx-node1:42777) driver disconnected.
> 17/02/08 09:51:04 ERROR executor.CoarseGrainedExecutorBackend: Driver 
> 192.168.1.1:42777 disassociated! Shutting down.
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 1105.0 in 
> stage 3.0 (TID 2511)
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Shutdown hook called
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator: 
> Shutting down remote daemon.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk6/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-71da1dfc-99bd-4687-bc2f-33452db8de3d
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk2/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-7f134d81-e77e-4b92-bd99-0a51d0962c14
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk5/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-77a90d63-fb05-4bc6-8d5e-1562cc502e6c
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator: 
> Remote daemon shut down; proceeding with flushing remote transports.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk4/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-91f8b91a-114d-4340-8560-d3cd085c1cd4
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk1/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-a3c24f9e-8609-48f0-9d37-0de7ae06682a
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator: 
> Remoting shut down.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk7/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-f6120a43-2158-4780-927c-c5786b78f53e
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk3/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-e17931ad-9e8a-45da-86f8-9a0fdca0fad1
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk8/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-4de34175-f871-4c28-8ec0-d2fc0020c5c3
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 1137.0 in 
> stage 3.0 (TID 2515)
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 897.0 in stage 
> 3.0 (TID 2417)
> 17/02/08 09:51:04 INFO 

[jira] [Updated] (HIVE-15859) HoS: Write RPC messages in event loop

[Rui Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Li updated HIVE-15859:
--
Summary: HoS: Write RPC messages in event loop  (was: Hive client side 
shows Spark Driver disconnected while Spark Driver side could not get RPC 
header )

> HoS: Write RPC messages in event loop
> -
>
> Key: HIVE-15859
> URL: https://issues.apache.org/jira/browse/HIVE-15859
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, Spark
>Affects Versions: 2.2.0
> Environment: hadoop2.7.1
> spark1.6.2
> hive2.2
>Reporter: KaiXu
>Assignee: Rui Li
> Attachments: HIVE-15859.1.patch, HIVE-15859.2.patch, 
> HIVE-15859.3.patch
>
>
> Hive on Spark, failed with error:
> {noformat}
> 2017-02-08 09:50:59,331 Stage-2_0: 1039(+2)/1041 Stage-3_0: 796(+456)/1520 
> Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> 2017-02-08 09:51:00,335 Stage-2_0: 1040(+1)/1041 Stage-3_0: 914(+398)/1520 
> Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> 2017-02-08 09:51:01,338 Stage-2_0: 1041/1041 Finished Stage-3_0: 
> 961(+383)/1520 Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> Failed to monitor Job[ 2] with exception 'java.lang.IllegalStateException(RPC 
> channel is closed.)'
> FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.spark.SparkTask
> {noformat}
> application log shows the driver commanded a shutdown with some unknown 
> reason, but hive's log shows Driver could not get RPC header( Expected RPC 
> header, got org.apache.hive.spark.client.rpc.Rpc$NullMessage instead).
> {noformat}
> 17/02/08 09:51:04 INFO exec.Utilities: PLAN PATH = 
> hdfs://hsx-node1:8020/tmp/hive/root/b723c85d-2a7b-469e-bab1-9c165b25e656/hive_2017-02-08_09-49-37_890_6267025825539539056-1/-mr-10006/71a9dacb-a463-40ef-9e86-78d3b8e3738d/map.xml
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 1169.0 in 
> stage 3.0 (TID 2519)
> 17/02/08 09:51:04 INFO executor.CoarseGrainedExecutorBackend: Driver 
> commanded a shutdown
> 17/02/08 09:51:04 INFO storage.MemoryStore: MemoryStore cleared
> 17/02/08 09:51:04 INFO storage.BlockManager: BlockManager stopped
> 17/02/08 09:51:04 INFO exec.Utilities: PLAN PATH = 
> hdfs://hsx-node1:8020/tmp/hive/root/b723c85d-2a7b-469e-bab1-9c165b25e656/hive_2017-02-08_09-49-37_890_6267025825539539056-1/-mr-10006/71a9dacb-a463-40ef-9e86-78d3b8e3738d/map.xml
> 17/02/08 09:51:04 WARN executor.CoarseGrainedExecutorBackend: An unknown 
> (hsx-node1:42777) driver disconnected.
> 17/02/08 09:51:04 ERROR executor.CoarseGrainedExecutorBackend: Driver 
> 192.168.1.1:42777 disassociated! Shutting down.
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 1105.0 in 
> stage 3.0 (TID 2511)
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Shutdown hook called
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator: 
> Shutting down remote daemon.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk6/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-71da1dfc-99bd-4687-bc2f-33452db8de3d
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk2/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-7f134d81-e77e-4b92-bd99-0a51d0962c14
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk5/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-77a90d63-fb05-4bc6-8d5e-1562cc502e6c
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator: 
> Remote daemon shut down; proceeding with flushing remote transports.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk4/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-91f8b91a-114d-4340-8560-d3cd085c1cd4
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk1/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-a3c24f9e-8609-48f0-9d37-0de7ae06682a
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator: 
> Remoting shut down.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk7/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-f6120a43-2158-4780-927c-c5786b78f53e
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk3/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-e17931ad-9e8a-45da-86f8-9a0fdca0fad1
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk8/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-4de34175-f871-4c28-8ec0-d2fc0020c5c3
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 1137.0 in 
> stage 3.0 (TID 2515)
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed 

[jira] [Assigned] (HIVE-16044) LLAP: Shuffle Handler keep-alive connections are closed from the server side

[Rajesh Balamohan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan reassigned HIVE-16044:
---


> LLAP: Shuffle Handler keep-alive connections are closed from the server side
> 
>
> Key: HIVE-16044
> URL: https://issues.apache.org/jira/browse/HIVE-16044
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>
> LLAP's shufflehandler could be closing the keep-alive connections after 
> output is served. This could break the connection from server side. JDK http 
> logs may not be revealing this. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-12274) Increase width of columns used for general configuration in the metastore.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885137#comment-15885137
 ] 

Hive QA commented on HIVE-12274:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12854811/HIVE-12274.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 84 failed/errored test(s), 10266 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query12] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query13] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query15] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query16] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query17] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query18] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query19] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query1] (batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query20] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query21] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query22] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query25] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query26] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query27] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query28] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query29] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query30] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query31] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query32] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query33] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query34] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query36] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query37] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query38] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query39] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query3] (batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query40] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query42] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query43] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query46] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query48] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query50] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query51] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query52] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query54] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query55] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query56] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query58] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query5] (batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query60] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query64] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query65] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query66] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query67] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query68] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query69] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query6] (batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query70] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query71] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query72] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query73] 
(batchId=223)

[jira] [Commented] (HIVE-16042) some sql file with comment encounter ParseException

[Pengcheng Xiong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885132#comment-15885132
 ] 

Pengcheng Xiong commented on HIVE-16042:


Could u paste the sql file in the JIRA?

> some sql file with comment encounter ParseException
> ---
>
> Key: HIVE-16042
> URL: https://issues.apache.org/jira/browse/HIVE-16042
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 2.2.0
> Environment: Hive2.2 (commit: 2768361)
> TPCx-BB v1.2
>Reporter: KaiXu
>
> current Hive upstream(commit: 2768361) failed to parse some 
> queries(q04,q17,q18,q23) in TPCx-BB v1.2, while it's ok with Hive(commit: 
> ac68aed).
> Q04: FAILED: ParseException line 24:0 missing EOF at ';' near 
> 'abandonedShoppingCartsPageCountsPerSession'
> Q17:
> NoViableAltException(350@[])
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.limitClause(HiveParser.java:38898)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:37002)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:36404)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:35722)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:35610)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:2279)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1328)
> at 
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:204)
> at 
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:75)
> at 
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:68)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:468)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
> at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
> at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
> at 
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:474)
> at 
> org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:490)
> at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> FAILED: ParseException line 39:0 cannot recognize input near 'LIMIT' '100' 
> ';' in limit clause
> Q18:
> NoViableAltException(350@[()* loopback of 424:20: ( ( LSQUARE ^ expression 
> RSQUARE !) | ( DOT ^ identifier ) )*])
> at org.antlr.runtime.DFA.noViableAlt(DFA.java:158)
> at org.antlr.runtime.DFA.predict(DFA.java:116)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6665)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6992)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:7048)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7210)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7353)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7496)
> at 
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceConcatenateExpression(HiveParser_IdentifiersParser.java:7635)
> at 
> 

[jira] [Issue Comment Deleted] (HIVE-12274) Increase width of columns used for general configuration in the metastore.

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-12274:
---
Comment: was deleted

(was: unsubscribe


)

> Increase width of columns used for general configuration in the metastore.
> --
>
> Key: HIVE-12274
> URL: https://issues.apache.org/jira/browse/HIVE-12274
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Elliot West
>Assignee: Naveen Gangam
>  Labels: metastore
> Attachments: HIVE-12274.2.patch, HIVE-12274.example.ddl.hql, 
> HIVE-12274.patch, HIVE-12274.patch, HIVE-12274.patch
>
>
> h2. Overview
> This issue is very similar in principle to HIVE-1364. We are hitting a limit 
> when processing JSON data that has a large nested schema. The struct 
> definition is truncated when inserted into the metastore database column 
> {{COLUMNS_V2.YPE_NAME}} as it is greater than 4000 characters in length.
> Given that the purpose of these columns is to hold very loosely defined 
> configuration values it seems rather limiting to impose such a relatively low 
> length bound. One can imagine that valid use cases will arise where 
> reasonable parameter/property values exceed the current limit. 
> h2. Context
> These limitations were in by the [patch 
> attributed|https://github.com/apache/hive/commit/c21a526b0a752df2a51d20a2729cc8493c228799]
>  to HIVE-1364 which mentions the _"max length on Oracle 9i/10g/11g"_ as the 
> reason. However, nowadays the limit can be increased because:
> * Oracle DB's {{varchar2}} supports 32767 bytes now, by setting the 
> configuration parameter {{MAX_STRING_SIZE}} to {{EXTENDED}}. 
> ([source|http://docs.oracle.com/database/121/SQLRF/sql_elements001.htm#SQLRF55623])
> * Postgres supports a max of 1GB for {{character}} datatype. 
> ([source|http://www.postgresql.org/docs/8.3/static/datatype-character.html])
> * MySQL can support upto 65535 bytes for the entire row. So long as the 
> {{PARAM_KEY}} value + {{PARAM_VALUE}} is less than 65535, we should be good. 
> ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * SQL Server's {{varchar}} max length is 8000 and can go beyond using 
> "varchar(max)" with the same limitation as MySQL being 65535 bytes for the 
> entire row. ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * Derby's {{varchar}} can be upto 32672 bytes. 
> ([source|https://db.apache.org/derby/docs/10.7/ref/rrefsqlj41207.html])
> h2. Proposal
> Can these columns not use CLOB-like types as for example as used by 
> {{TBLS.VIEW_EXPANDED_TEXT}}? It would seem that suitable type equivalents 
> exist for all targeted database platforms:
> * MySQL: {{mediumtext}}
> * Postgres: {{text}}
> * Oracle: {{CLOB}}
> * Derby: {{LONG VARCHAR}}
> I'd suggest that the candidates for type change are:
> * {{COLUMNS_V2.TYPE_NAME}}
> * {{TABLE_PARAMS.PARAM_VALUE}}
> * {{SERDE_PARAMS.PARAM_VALUE}}
> * {{SD_PARAMS.PARAM_VALUE}}
> After updating the maximum length the metastore database needs to be 
> configured and restarted with the new settings. Altering {{MAX_STRING_SIZE}} 
> will update database objects and possibly invalidate them, as follows:
> * Tables with virtual columns will be updated with new data type metadata for 
> virtual columns of {{VARCHAR2(4000)}}, 4000-byte {{NVARCHAR2}}, or 
> {{RAW(2000)}} type.
> * Functional indexes will become unusable if a change to their associated 
> virtual columns causes the index key to exceed index key length limits. 
> Attempts to rebuild such indexes will fail with {{ORA-01450: maximum key 
> length exceeded}}.
> * Views will be invalidated if they contain {{VARCHAR2(4000)}}, 4000-byte 
> {{NVARCHAR2}}, or {{RAW(2000)}} typed expression columns.
> * Materialized views will be updated with new metadata {{VARCHAR2(4000)}}, 
> 4000-byte {{NVARCHAR2}}, and {{RAW(2000)}} typed expression columns
> * So the limitation could be raised to 32672 bytes, with the caveat that 
> MySQL and SQL Server limit the row length to 65535 bytes, so that should also 
> be validated to provide consistency.
> Finally, will this limitation persist in the work resulting from HIVE-9452?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885126#comment-15885126
 ] 

Xuefu Zhang commented on HIVE-15859:


+1

> Hive client side shows Spark Driver disconnected while Spark Driver side 
> could not get RPC header 
> --
>
> Key: HIVE-15859
> URL: https://issues.apache.org/jira/browse/HIVE-15859
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, Spark
>Affects Versions: 2.2.0
> Environment: hadoop2.7.1
> spark1.6.2
> hive2.2
>Reporter: KaiXu
>Assignee: Rui Li
> Attachments: HIVE-15859.1.patch, HIVE-15859.2.patch, 
> HIVE-15859.3.patch
>
>
> Hive on Spark, failed with error:
> {noformat}
> 2017-02-08 09:50:59,331 Stage-2_0: 1039(+2)/1041 Stage-3_0: 796(+456)/1520 
> Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> 2017-02-08 09:51:00,335 Stage-2_0: 1040(+1)/1041 Stage-3_0: 914(+398)/1520 
> Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> 2017-02-08 09:51:01,338 Stage-2_0: 1041/1041 Finished Stage-3_0: 
> 961(+383)/1520 Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> Failed to monitor Job[ 2] with exception 'java.lang.IllegalStateException(RPC 
> channel is closed.)'
> FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.spark.SparkTask
> {noformat}
> application log shows the driver commanded a shutdown with some unknown 
> reason, but hive's log shows Driver could not get RPC header( Expected RPC 
> header, got org.apache.hive.spark.client.rpc.Rpc$NullMessage instead).
> {noformat}
> 17/02/08 09:51:04 INFO exec.Utilities: PLAN PATH = 
> hdfs://hsx-node1:8020/tmp/hive/root/b723c85d-2a7b-469e-bab1-9c165b25e656/hive_2017-02-08_09-49-37_890_6267025825539539056-1/-mr-10006/71a9dacb-a463-40ef-9e86-78d3b8e3738d/map.xml
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 1169.0 in 
> stage 3.0 (TID 2519)
> 17/02/08 09:51:04 INFO executor.CoarseGrainedExecutorBackend: Driver 
> commanded a shutdown
> 17/02/08 09:51:04 INFO storage.MemoryStore: MemoryStore cleared
> 17/02/08 09:51:04 INFO storage.BlockManager: BlockManager stopped
> 17/02/08 09:51:04 INFO exec.Utilities: PLAN PATH = 
> hdfs://hsx-node1:8020/tmp/hive/root/b723c85d-2a7b-469e-bab1-9c165b25e656/hive_2017-02-08_09-49-37_890_6267025825539539056-1/-mr-10006/71a9dacb-a463-40ef-9e86-78d3b8e3738d/map.xml
> 17/02/08 09:51:04 WARN executor.CoarseGrainedExecutorBackend: An unknown 
> (hsx-node1:42777) driver disconnected.
> 17/02/08 09:51:04 ERROR executor.CoarseGrainedExecutorBackend: Driver 
> 192.168.1.1:42777 disassociated! Shutting down.
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 1105.0 in 
> stage 3.0 (TID 2511)
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Shutdown hook called
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator: 
> Shutting down remote daemon.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk6/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-71da1dfc-99bd-4687-bc2f-33452db8de3d
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk2/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-7f134d81-e77e-4b92-bd99-0a51d0962c14
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk5/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-77a90d63-fb05-4bc6-8d5e-1562cc502e6c
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator: 
> Remote daemon shut down; proceeding with flushing remote transports.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk4/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-91f8b91a-114d-4340-8560-d3cd085c1cd4
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk1/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-a3c24f9e-8609-48f0-9d37-0de7ae06682a
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator: 
> Remoting shut down.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk7/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-f6120a43-2158-4780-927c-c5786b78f53e
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk3/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-e17931ad-9e8a-45da-86f8-9a0fdca0fad1
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk8/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-4de34175-f871-4c28-8ec0-d2fc0020c5c3
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 1137.0 in 
> stage 3.0 (TID 2515)
> 17/02/08 09:51:04 INFO executor.Executor: 

[jira] [Updated] (HIVE-16043) TezConfiguration.TEZ_QUEUE_NAME instead of tez.queue.name

[Fei Hui (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fei Hui updated HIVE-16043:
---
Status: Patch Available  (was: Open)

> TezConfiguration.TEZ_QUEUE_NAME instead of tez.queue.name
> -
>
> Key: HIVE-16043
> URL: https://issues.apache.org/jira/browse/HIVE-16043
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: 2.2.0
>Reporter: Fei Hui
>Assignee: Fei Hui
> Attachments: HIVE-16043.1.patch
>
>
> I see the following source in hive
> {code:title=TezSessionPoolManager.java|borderStyle=solid}
>private TezSessionState getSession(HiveConf conf, boolean doOpen)
>throws Exception {
>  String queueName = conf.get("tez.queue.name");
>  ...
>}
>   private TezSessionState getNewSessionState(HiveConf conf,
>   String queueName, boolean doOpen) throws Exception {
> TezSessionPoolSession retTezSessionState = 
> createAndInitSession(queueName, false);
> if (queueName != null) {
>   conf.set(TezConfiguration.TEZ_QUEUE_NAME, queueName);
> }
>   ...
>   }
> {code}
> TezConfiguration.TEZ_QUEUE_NAME is the same as tez.queue.name , i think we 
> should consistently use it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16043) TezConfiguration.TEZ_QUEUE_NAME instead of tez.queue.name

[Fei Hui (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885124#comment-15885124
 ] 

Fei Hui commented on HIVE-16043:


CC [~sershe] .I see HIVE-16010 , and find that maybe it is better like this for 
consistent usage.
How do you think, could you please give any suggestions?

> TezConfiguration.TEZ_QUEUE_NAME instead of tez.queue.name
> -
>
> Key: HIVE-16043
> URL: https://issues.apache.org/jira/browse/HIVE-16043
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: 2.2.0
>Reporter: Fei Hui
>Assignee: Fei Hui
> Attachments: HIVE-16043.1.patch
>
>
> I see the following source in hive
> {code:title=TezSessionPoolManager.java|borderStyle=solid}
>private TezSessionState getSession(HiveConf conf, boolean doOpen)
>throws Exception {
>  String queueName = conf.get("tez.queue.name");
>  ...
>}
>   private TezSessionState getNewSessionState(HiveConf conf,
>   String queueName, boolean doOpen) throws Exception {
> TezSessionPoolSession retTezSessionState = 
> createAndInitSession(queueName, false);
> if (queueName != null) {
>   conf.set(TezConfiguration.TEZ_QUEUE_NAME, queueName);
> }
>   ...
>   }
> {code}
> TezConfiguration.TEZ_QUEUE_NAME is the same as tez.queue.name , i think we 
> should consistently use it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16043) TezConfiguration.TEZ_QUEUE_NAME instead of tez.queue.name

[Fei Hui (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fei Hui updated HIVE-16043:
---
Attachment: HIVE-16043.1.patch

patch uploaded

> TezConfiguration.TEZ_QUEUE_NAME instead of tez.queue.name
> -
>
> Key: HIVE-16043
> URL: https://issues.apache.org/jira/browse/HIVE-16043
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: 2.2.0
>Reporter: Fei Hui
> Attachments: HIVE-16043.1.patch
>
>
> I see the following source in hive
> {code:title=TezSessionPoolManager.java|borderStyle=solid}
>private TezSessionState getSession(HiveConf conf, boolean doOpen)
>throws Exception {
>  String queueName = conf.get("tez.queue.name");
>  ...
>}
>   private TezSessionState getNewSessionState(HiveConf conf,
>   String queueName, boolean doOpen) throws Exception {
> TezSessionPoolSession retTezSessionState = 
> createAndInitSession(queueName, false);
> if (queueName != null) {
>   conf.set(TezConfiguration.TEZ_QUEUE_NAME, queueName);
> }
>   ...
>   }
> {code}
> TezConfiguration.TEZ_QUEUE_NAME is the same as tez.queue.name , i think we 
> should consistently use it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16043) TezConfiguration.TEZ_QUEUE_NAME instead of tez.queue.name

[Fei Hui (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fei Hui reassigned HIVE-16043:
--

Assignee: Fei Hui

> TezConfiguration.TEZ_QUEUE_NAME instead of tez.queue.name
> -
>
> Key: HIVE-16043
> URL: https://issues.apache.org/jira/browse/HIVE-16043
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: 2.2.0
>Reporter: Fei Hui
>Assignee: Fei Hui
> Attachments: HIVE-16043.1.patch
>
>
> I see the following source in hive
> {code:title=TezSessionPoolManager.java|borderStyle=solid}
>private TezSessionState getSession(HiveConf conf, boolean doOpen)
>throws Exception {
>  String queueName = conf.get("tez.queue.name");
>  ...
>}
>   private TezSessionState getNewSessionState(HiveConf conf,
>   String queueName, boolean doOpen) throws Exception {
> TezSessionPoolSession retTezSessionState = 
> createAndInitSession(queueName, false);
> if (queueName != null) {
>   conf.set(TezConfiguration.TEZ_QUEUE_NAME, queueName);
> }
>   ...
>   }
> {code}
> TezConfiguration.TEZ_QUEUE_NAME is the same as tez.queue.name , i think we 
> should consistently use it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15881) Use new thread count variable name instead of mapred.dfsclient.parallelism.max

[JIRA]

 [ 
https://issues.apache.org/jira/browse/HIVE-15881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergio Peña updated HIVE-15881:
---
Attachment: HIVE-15881.6.patch

Patch rebased.

> Use new thread count variable name instead of mapred.dfsclient.parallelism.max
> --
>
> Key: HIVE-15881
> URL: https://issues.apache.org/jira/browse/HIVE-15881
> Project: Hive
>  Issue Type: Task
>  Components: Query Planning
>Reporter: Sergio Peña
>Assignee: Sergio Peña
>Priority: Minor
> Attachments: HIVE-15881.1.patch, HIVE-15881.2.patch, 
> HIVE-15881.3.patch, HIVE-15881.4.patch, HIVE-15881.5.patch, HIVE-15881.6.patch
>
>
> The Utilities class has two methods, {{getInputSummary}} and 
> {{getInputPaths}}, that use the variable {{mapred.dfsclient.parallelism.max}} 
> to get the summary of a list of input locations in parallel. These methods 
> are Hive related, but the variable name does not look it is specific for Hive.
> Also, the above variable is not on HiveConf nor used anywhere else. I just 
> found a reference on the Hadoop MR1 code.
> I'd like to propose the deprecation of {{mapred.dfsclient.parallelism.max}}, 
> and use a different variable name, such as 
> {{hive.get.input.listing.num.threads}}, that reflects the intention of the 
> variable. The removal of the old variable might happen on Hive 3.x



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15430) Change SchemaTool table validator to test based on the dbType

[Naveen Gangam (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885109#comment-15885109
 ] 

Naveen Gangam commented on HIVE-15430:
--

The test failures do not appear related to the proposed patch. 

> Change SchemaTool table validator to test based on the dbType
> -
>
> Key: HIVE-15430
> URL: https://issues.apache.org/jira/browse/HIVE-15430
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Affects Versions: 2.2.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Minor
> Attachments: HIVE-15430.1.patch, HIVE-15430.2.patch, 
> HIVE-15430.3.patch, HIVE-15430.4.patch
>
>
> Currently the validator parses the "oracle" schema file to determine what 
> tables are expected in the database.  (mostly because of ease of parsing the 
> schema file compared to other syntax). We have learnt from HIVE-15118, that 
> not all schema files have the same amount of tables. For example, derby has 
> an old table that is never used that other DBs do not contain).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-12274) Increase width of columns used for general configuration in the metastore.

[Scott Steimle (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Steimle updated HIVE-12274:
-

unsubscribe




> Increase width of columns used for general configuration in the metastore.
> --
>
> Key: HIVE-12274
> URL: https://issues.apache.org/jira/browse/HIVE-12274
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Elliot West
>Assignee: Naveen Gangam
>  Labels: metastore
> Attachments: HIVE-12274.2.patch, HIVE-12274.example.ddl.hql, 
> HIVE-12274.patch, HIVE-12274.patch, HIVE-12274.patch
>
>
> h2. Overview
> This issue is very similar in principle to HIVE-1364. We are hitting a limit 
> when processing JSON data that has a large nested schema. The struct 
> definition is truncated when inserted into the metastore database column 
> {{COLUMNS_V2.YPE_NAME}} as it is greater than 4000 characters in length.
> Given that the purpose of these columns is to hold very loosely defined 
> configuration values it seems rather limiting to impose such a relatively low 
> length bound. One can imagine that valid use cases will arise where 
> reasonable parameter/property values exceed the current limit. 
> h2. Context
> These limitations were in by the [patch 
> attributed|https://github.com/apache/hive/commit/c21a526b0a752df2a51d20a2729cc8493c228799]
>  to HIVE-1364 which mentions the _"max length on Oracle 9i/10g/11g"_ as the 
> reason. However, nowadays the limit can be increased because:
> * Oracle DB's {{varchar2}} supports 32767 bytes now, by setting the 
> configuration parameter {{MAX_STRING_SIZE}} to {{EXTENDED}}. 
> ([source|http://docs.oracle.com/database/121/SQLRF/sql_elements001.htm#SQLRF55623])
> * Postgres supports a max of 1GB for {{character}} datatype. 
> ([source|http://www.postgresql.org/docs/8.3/static/datatype-character.html])
> * MySQL can support upto 65535 bytes for the entire row. So long as the 
> {{PARAM_KEY}} value + {{PARAM_VALUE}} is less than 65535, we should be good. 
> ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * SQL Server's {{varchar}} max length is 8000 and can go beyond using 
> "varchar(max)" with the same limitation as MySQL being 65535 bytes for the 
> entire row. ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * Derby's {{varchar}} can be upto 32672 bytes. 
> ([source|https://db.apache.org/derby/docs/10.7/ref/rrefsqlj41207.html])
> h2. Proposal
> Can these columns not use CLOB-like types as for example as used by 
> {{TBLS.VIEW_EXPANDED_TEXT}}? It would seem that suitable type equivalents 
> exist for all targeted database platforms:
> * MySQL: {{mediumtext}}
> * Postgres: {{text}}
> * Oracle: {{CLOB}}
> * Derby: {{LONG VARCHAR}}
> I'd suggest that the candidates for type change are:
> * {{COLUMNS_V2.TYPE_NAME}}
> * {{TABLE_PARAMS.PARAM_VALUE}}
> * {{SERDE_PARAMS.PARAM_VALUE}}
> * {{SD_PARAMS.PARAM_VALUE}}
> After updating the maximum length the metastore database needs to be 
> configured and restarted with the new settings. Altering {{MAX_STRING_SIZE}} 
> will update database objects and possibly invalidate them, as follows:
> * Tables with virtual columns will be updated with new data type metadata for 
> virtual columns of {{VARCHAR2(4000)}}, 4000-byte {{NVARCHAR2}}, or 
> {{RAW(2000)}} type.
> * Functional indexes will become unusable if a change to their associated 
> virtual columns causes the index key to exceed index key length limits. 
> Attempts to rebuild such indexes will fail with {{ORA-01450: maximum key 
> length exceeded}}.
> * Views will be invalidated if they contain {{VARCHAR2(4000)}}, 4000-byte 
> {{NVARCHAR2}}, or {{RAW(2000)}} typed expression columns.
> * Materialized views will be updated with new metadata {{VARCHAR2(4000)}}, 
> 4000-byte {{NVARCHAR2}}, and {{RAW(2000)}} typed expression columns
> * So the limitation could be raised to 32672 bytes, with the caveat that 
> MySQL and SQL Server limit the row length to 65535 bytes, so that should also 
> be validated to provide consistency.
> Finally, will this limitation persist in the work resulting from HIVE-9452?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-12274) Increase width of columns used for general configuration in the metastore.

[Naveen Gangam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-12274:
-
Attachment: HIVE-12274.2.patch

> Increase width of columns used for general configuration in the metastore.
> --
>
> Key: HIVE-12274
> URL: https://issues.apache.org/jira/browse/HIVE-12274
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Elliot West
>Assignee: Naveen Gangam
>  Labels: metastore
> Attachments: HIVE-12274.2.patch, HIVE-12274.example.ddl.hql, 
> HIVE-12274.patch, HIVE-12274.patch, HIVE-12274.patch
>
>
> h2. Overview
> This issue is very similar in principle to HIVE-1364. We are hitting a limit 
> when processing JSON data that has a large nested schema. The struct 
> definition is truncated when inserted into the metastore database column 
> {{COLUMNS_V2.YPE_NAME}} as it is greater than 4000 characters in length.
> Given that the purpose of these columns is to hold very loosely defined 
> configuration values it seems rather limiting to impose such a relatively low 
> length bound. One can imagine that valid use cases will arise where 
> reasonable parameter/property values exceed the current limit. 
> h2. Context
> These limitations were in by the [patch 
> attributed|https://github.com/apache/hive/commit/c21a526b0a752df2a51d20a2729cc8493c228799]
>  to HIVE-1364 which mentions the _"max length on Oracle 9i/10g/11g"_ as the 
> reason. However, nowadays the limit can be increased because:
> * Oracle DB's {{varchar2}} supports 32767 bytes now, by setting the 
> configuration parameter {{MAX_STRING_SIZE}} to {{EXTENDED}}. 
> ([source|http://docs.oracle.com/database/121/SQLRF/sql_elements001.htm#SQLRF55623])
> * Postgres supports a max of 1GB for {{character}} datatype. 
> ([source|http://www.postgresql.org/docs/8.3/static/datatype-character.html])
> * MySQL can support upto 65535 bytes for the entire row. So long as the 
> {{PARAM_KEY}} value + {{PARAM_VALUE}} is less than 65535, we should be good. 
> ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * SQL Server's {{varchar}} max length is 8000 and can go beyond using 
> "varchar(max)" with the same limitation as MySQL being 65535 bytes for the 
> entire row. ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * Derby's {{varchar}} can be upto 32672 bytes. 
> ([source|https://db.apache.org/derby/docs/10.7/ref/rrefsqlj41207.html])
> h2. Proposal
> Can these columns not use CLOB-like types as for example as used by 
> {{TBLS.VIEW_EXPANDED_TEXT}}? It would seem that suitable type equivalents 
> exist for all targeted database platforms:
> * MySQL: {{mediumtext}}
> * Postgres: {{text}}
> * Oracle: {{CLOB}}
> * Derby: {{LONG VARCHAR}}
> I'd suggest that the candidates for type change are:
> * {{COLUMNS_V2.TYPE_NAME}}
> * {{TABLE_PARAMS.PARAM_VALUE}}
> * {{SERDE_PARAMS.PARAM_VALUE}}
> * {{SD_PARAMS.PARAM_VALUE}}
> After updating the maximum length the metastore database needs to be 
> configured and restarted with the new settings. Altering {{MAX_STRING_SIZE}} 
> will update database objects and possibly invalidate them, as follows:
> * Tables with virtual columns will be updated with new data type metadata for 
> virtual columns of {{VARCHAR2(4000)}}, 4000-byte {{NVARCHAR2}}, or 
> {{RAW(2000)}} type.
> * Functional indexes will become unusable if a change to their associated 
> virtual columns causes the index key to exceed index key length limits. 
> Attempts to rebuild such indexes will fail with {{ORA-01450: maximum key 
> length exceeded}}.
> * Views will be invalidated if they contain {{VARCHAR2(4000)}}, 4000-byte 
> {{NVARCHAR2}}, or {{RAW(2000)}} typed expression columns.
> * Materialized views will be updated with new metadata {{VARCHAR2(4000)}}, 
> 4000-byte {{NVARCHAR2}}, and {{RAW(2000)}} typed expression columns
> * So the limitation could be raised to 32672 bytes, with the caveat that 
> MySQL and SQL Server limit the row length to 65535 bytes, so that should also 
> be validated to provide consistency.
> Finally, will this limitation persist in the work resulting from HIVE-9452?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-12274) Increase width of columns used for general configuration in the metastore.

[Naveen Gangam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-12274:
-
Status: Patch Available  (was: Open)

> Increase width of columns used for general configuration in the metastore.
> --
>
> Key: HIVE-12274
> URL: https://issues.apache.org/jira/browse/HIVE-12274
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Elliot West
>Assignee: Naveen Gangam
>  Labels: metastore
> Attachments: HIVE-12274.2.patch, HIVE-12274.example.ddl.hql, 
> HIVE-12274.patch, HIVE-12274.patch, HIVE-12274.patch
>
>
> h2. Overview
> This issue is very similar in principle to HIVE-1364. We are hitting a limit 
> when processing JSON data that has a large nested schema. The struct 
> definition is truncated when inserted into the metastore database column 
> {{COLUMNS_V2.YPE_NAME}} as it is greater than 4000 characters in length.
> Given that the purpose of these columns is to hold very loosely defined 
> configuration values it seems rather limiting to impose such a relatively low 
> length bound. One can imagine that valid use cases will arise where 
> reasonable parameter/property values exceed the current limit. 
> h2. Context
> These limitations were in by the [patch 
> attributed|https://github.com/apache/hive/commit/c21a526b0a752df2a51d20a2729cc8493c228799]
>  to HIVE-1364 which mentions the _"max length on Oracle 9i/10g/11g"_ as the 
> reason. However, nowadays the limit can be increased because:
> * Oracle DB's {{varchar2}} supports 32767 bytes now, by setting the 
> configuration parameter {{MAX_STRING_SIZE}} to {{EXTENDED}}. 
> ([source|http://docs.oracle.com/database/121/SQLRF/sql_elements001.htm#SQLRF55623])
> * Postgres supports a max of 1GB for {{character}} datatype. 
> ([source|http://www.postgresql.org/docs/8.3/static/datatype-character.html])
> * MySQL can support upto 65535 bytes for the entire row. So long as the 
> {{PARAM_KEY}} value + {{PARAM_VALUE}} is less than 65535, we should be good. 
> ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * SQL Server's {{varchar}} max length is 8000 and can go beyond using 
> "varchar(max)" with the same limitation as MySQL being 65535 bytes for the 
> entire row. ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * Derby's {{varchar}} can be upto 32672 bytes. 
> ([source|https://db.apache.org/derby/docs/10.7/ref/rrefsqlj41207.html])
> h2. Proposal
> Can these columns not use CLOB-like types as for example as used by 
> {{TBLS.VIEW_EXPANDED_TEXT}}? It would seem that suitable type equivalents 
> exist for all targeted database platforms:
> * MySQL: {{mediumtext}}
> * Postgres: {{text}}
> * Oracle: {{CLOB}}
> * Derby: {{LONG VARCHAR}}
> I'd suggest that the candidates for type change are:
> * {{COLUMNS_V2.TYPE_NAME}}
> * {{TABLE_PARAMS.PARAM_VALUE}}
> * {{SERDE_PARAMS.PARAM_VALUE}}
> * {{SD_PARAMS.PARAM_VALUE}}
> After updating the maximum length the metastore database needs to be 
> configured and restarted with the new settings. Altering {{MAX_STRING_SIZE}} 
> will update database objects and possibly invalidate them, as follows:
> * Tables with virtual columns will be updated with new data type metadata for 
> virtual columns of {{VARCHAR2(4000)}}, 4000-byte {{NVARCHAR2}}, or 
> {{RAW(2000)}} type.
> * Functional indexes will become unusable if a change to their associated 
> virtual columns causes the index key to exceed index key length limits. 
> Attempts to rebuild such indexes will fail with {{ORA-01450: maximum key 
> length exceeded}}.
> * Views will be invalidated if they contain {{VARCHAR2(4000)}}, 4000-byte 
> {{NVARCHAR2}}, or {{RAW(2000)}} typed expression columns.
> * Materialized views will be updated with new metadata {{VARCHAR2(4000)}}, 
> 4000-byte {{NVARCHAR2}}, and {{RAW(2000)}} typed expression columns
> * So the limitation could be raised to 32672 bytes, with the caveat that 
> MySQL and SQL Server limit the row length to 65535 bytes, so that should also 
> be validated to provide consistency.
> Finally, will this limitation persist in the work resulting from HIVE-9452?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-12274) Increase width of columns used for general configuration in the metastore.

[Naveen Gangam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-12274:
-
Status: Open  (was: Patch Available)

Test failures from DirectSQL implementation unable to convert SQL Clob to 
Strings.

> Increase width of columns used for general configuration in the metastore.
> --
>
> Key: HIVE-12274
> URL: https://issues.apache.org/jira/browse/HIVE-12274
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Elliot West
>Assignee: Naveen Gangam
>  Labels: metastore
> Attachments: HIVE-12274.example.ddl.hql, HIVE-12274.patch, 
> HIVE-12274.patch, HIVE-12274.patch
>
>
> h2. Overview
> This issue is very similar in principle to HIVE-1364. We are hitting a limit 
> when processing JSON data that has a large nested schema. The struct 
> definition is truncated when inserted into the metastore database column 
> {{COLUMNS_V2.YPE_NAME}} as it is greater than 4000 characters in length.
> Given that the purpose of these columns is to hold very loosely defined 
> configuration values it seems rather limiting to impose such a relatively low 
> length bound. One can imagine that valid use cases will arise where 
> reasonable parameter/property values exceed the current limit. 
> h2. Context
> These limitations were in by the [patch 
> attributed|https://github.com/apache/hive/commit/c21a526b0a752df2a51d20a2729cc8493c228799]
>  to HIVE-1364 which mentions the _"max length on Oracle 9i/10g/11g"_ as the 
> reason. However, nowadays the limit can be increased because:
> * Oracle DB's {{varchar2}} supports 32767 bytes now, by setting the 
> configuration parameter {{MAX_STRING_SIZE}} to {{EXTENDED}}. 
> ([source|http://docs.oracle.com/database/121/SQLRF/sql_elements001.htm#SQLRF55623])
> * Postgres supports a max of 1GB for {{character}} datatype. 
> ([source|http://www.postgresql.org/docs/8.3/static/datatype-character.html])
> * MySQL can support upto 65535 bytes for the entire row. So long as the 
> {{PARAM_KEY}} value + {{PARAM_VALUE}} is less than 65535, we should be good. 
> ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * SQL Server's {{varchar}} max length is 8000 and can go beyond using 
> "varchar(max)" with the same limitation as MySQL being 65535 bytes for the 
> entire row. ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * Derby's {{varchar}} can be upto 32672 bytes. 
> ([source|https://db.apache.org/derby/docs/10.7/ref/rrefsqlj41207.html])
> h2. Proposal
> Can these columns not use CLOB-like types as for example as used by 
> {{TBLS.VIEW_EXPANDED_TEXT}}? It would seem that suitable type equivalents 
> exist for all targeted database platforms:
> * MySQL: {{mediumtext}}
> * Postgres: {{text}}
> * Oracle: {{CLOB}}
> * Derby: {{LONG VARCHAR}}
> I'd suggest that the candidates for type change are:
> * {{COLUMNS_V2.TYPE_NAME}}
> * {{TABLE_PARAMS.PARAM_VALUE}}
> * {{SERDE_PARAMS.PARAM_VALUE}}
> * {{SD_PARAMS.PARAM_VALUE}}
> After updating the maximum length the metastore database needs to be 
> configured and restarted with the new settings. Altering {{MAX_STRING_SIZE}} 
> will update database objects and possibly invalidate them, as follows:
> * Tables with virtual columns will be updated with new data type metadata for 
> virtual columns of {{VARCHAR2(4000)}}, 4000-byte {{NVARCHAR2}}, or 
> {{RAW(2000)}} type.
> * Functional indexes will become unusable if a change to their associated 
> virtual columns causes the index key to exceed index key length limits. 
> Attempts to rebuild such indexes will fail with {{ORA-01450: maximum key 
> length exceeded}}.
> * Views will be invalidated if they contain {{VARCHAR2(4000)}}, 4000-byte 
> {{NVARCHAR2}}, or {{RAW(2000)}} typed expression columns.
> * Materialized views will be updated with new metadata {{VARCHAR2(4000)}}, 
> 4000-byte {{NVARCHAR2}}, and {{RAW(2000)}} typed expression columns
> * So the limitation could be raised to 32672 bytes, with the caveat that 
> MySQL and SQL Server limit the row length to 65535 bytes, so that should also 
> be validated to provide consistency.
> Finally, will this limitation persist in the work resulting from HIVE-9452?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-6535) JDBC: provide an async API to execute query and fetch results

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-6535:

Release Note: 
Introduced new method HiveStatement.executeAsync - Starts the query execution 
asynchronously on the server, and immediately returns to the client. The client 
subsequently blocks on ResultSet#next or Statement#getUpdateCount, depending on 
the query type. Users should call ResultSet.next or Statement#getUpdateCount 
(depending on whether query returns results) to ensure that query completes 
successfully. Calling another execute method, or close before query completion 
would result in the async query getting killed if it is not already finished.
Note: This method is an API for limited usage outside of Hive by applications 
like Apache Ambari, although it is not part of the interface java.sql.Statement.

  was:
Starts the query execution asynchronously on the server, and immediately 
returns to the client. The client subsequently blocks on ResultSet#next or 
Statement#getUpdateCount, depending on the query type. Users should call 
ResultSet.next or Statement#getUpdateCount (depending on whether query returns 
results) to ensure that query completes successfully. Calling another execute 
method, or close before query completion would result in the async query 
getting killed if it is not already finished.
Note: This method is an API for limited usage outside of Hive by applications 
like Apache Ambari, although it is not part of the interface java.sql.Statement.


> JDBC: provide an async API to execute query and fetch results
> -
>
> Key: HIVE-6535
> URL: https://issues.apache.org/jira/browse/HIVE-6535
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2, JDBC
>Affects Versions: 0.14.0, 1.2.1, 2.0.0
>Reporter: Thejas M Nair
>Assignee: Vaibhav Gumashta
> Fix For: 2.1.0
>
> Attachments: HIVE-6535.1.patch, HIVE-6535.2.patch, HIVE-6535.3.patch, 
> HIVE-6535.4.patch, HIVE-6535.5.patch, HIVE-6535.6.patch
>
>
> The hive jdbc client waits query completion during execute() call. It would 
> be better to block in the jdbc for completion when the results are being 
> fetched.
> This way the application using hive jdbc driver can do other tasks while 
> asynchronous query execution is happening, until it needs to fetch the result 
> set.
>  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885092#comment-15885092
 ] 

Hive QA commented on HIVE-15859:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12854804/HIVE-15859.3.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10266 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=223)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3801/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3801/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3801/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12854804 - PreCommit-HIVE-Build

> Hive client side shows Spark Driver disconnected while Spark Driver side 
> could not get RPC header 
> --
>
> Key: HIVE-15859
> URL: https://issues.apache.org/jira/browse/HIVE-15859
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, Spark
>Affects Versions: 2.2.0
> Environment: hadoop2.7.1
> spark1.6.2
> hive2.2
>Reporter: KaiXu
>Assignee: Rui Li
> Attachments: HIVE-15859.1.patch, HIVE-15859.2.patch, 
> HIVE-15859.3.patch
>
>
> Hive on Spark, failed with error:
> {noformat}
> 2017-02-08 09:50:59,331 Stage-2_0: 1039(+2)/1041 Stage-3_0: 796(+456)/1520 
> Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> 2017-02-08 09:51:00,335 Stage-2_0: 1040(+1)/1041 Stage-3_0: 914(+398)/1520 
> Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> 2017-02-08 09:51:01,338 Stage-2_0: 1041/1041 Finished Stage-3_0: 
> 961(+383)/1520 Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> Failed to monitor Job[ 2] with exception 'java.lang.IllegalStateException(RPC 
> channel is closed.)'
> FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.spark.SparkTask
> {noformat}
> application log shows the driver commanded a shutdown with some unknown 
> reason, but hive's log shows Driver could not get RPC header( Expected RPC 
> header, got org.apache.hive.spark.client.rpc.Rpc$NullMessage instead).
> {noformat}
> 17/02/08 09:51:04 INFO exec.Utilities: PLAN PATH = 
> hdfs://hsx-node1:8020/tmp/hive/root/b723c85d-2a7b-469e-bab1-9c165b25e656/hive_2017-02-08_09-49-37_890_6267025825539539056-1/-mr-10006/71a9dacb-a463-40ef-9e86-78d3b8e3738d/map.xml
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 1169.0 in 
> stage 3.0 (TID 2519)
> 17/02/08 09:51:04 INFO executor.CoarseGrainedExecutorBackend: Driver 
> commanded a shutdown
> 17/02/08 09:51:04 INFO storage.MemoryStore: MemoryStore cleared
> 17/02/08 09:51:04 INFO storage.BlockManager: BlockManager stopped
> 17/02/08 09:51:04 INFO exec.Utilities: PLAN PATH = 
> hdfs://hsx-node1:8020/tmp/hive/root/b723c85d-2a7b-469e-bab1-9c165b25e656/hive_2017-02-08_09-49-37_890_6267025825539539056-1/-mr-10006/71a9dacb-a463-40ef-9e86-78d3b8e3738d/map.xml
> 17/02/08 09:51:04 WARN executor.CoarseGrainedExecutorBackend: An unknown 
> (hsx-node1:42777) driver disconnected.
> 17/02/08 09:51:04 ERROR executor.CoarseGrainedExecutorBackend: Driver 
> 192.168.1.1:42777 disassociated! Shutting down.
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 1105.0 in 
> stage 3.0 (TID 2511)
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Shutdown hook called
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator: 
> Shutting down remote daemon.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk6/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-71da1dfc-99bd-4687-bc2f-33452db8de3d
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk2/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-7f134d81-e77e-4b92-bd99-0a51d0962c14
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk5/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-77a90d63-fb05-4bc6-8d5e-1562cc502e6c
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator: 
> Remote daemon shut down; proceeding with flushing remote transports.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> 

[jira] [Commented] (HIVE-16041) HCatalog doesn't delete temp _SCRATCH dir when job failed

[yunfei liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885070#comment-15885070
 ] 

yunfei liu commented on HIVE-16041:
---

The unit test failures seems not relevant to this patch.
[~mithun] would you mind to help me review the patch

> HCatalog doesn't delete temp  _SCRATCH dir when job failed
> --
>
> Key: HIVE-16041
> URL: https://issues.apache.org/jira/browse/HIVE-16041
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 2.2.0
>Reporter: yunfei liu
>Assignee: yunfei liu
> Fix For: 2.2.0
>
> Attachments: HIVE-16041.1.patch, HIVE-16041.2.patch
>
>
> when we use HCatOutputFormat to write to an external partitioned table,  a 
> tmp dir (which starts with "_SCRATCH" ) will appear under table path if the 
> job failed. 
> {quote}
> drwxr-xr-x   - yun hdfs  0 2017-02-27 01:45 
> /tmp/hive/_SCRATCH0.31946356159329714
> drwxr-xr-x   - yun hdfs  0 2017-02-27 01:51 
> /tmp/hive/_SCRATCH0.31946356159329714/c1=1
> drwxr-xr-x   - yun hdfs  0 2017-02-27 00:57 /tmp/hive/c1=1
> drwxr-xr-x   - yun hdfs  0 2017-02-27 01:28 /tmp/hive/c1=1/c2=2
> -rw-r--r--   3 yun hdfs 12 2017-02-27 00:57 
> /tmp/hive/c1=1/c2=2/part-r-0
> -rw-r--r--   3 yun hdfs 12 2017-02-27 01:28 
> /tmp/hive/c1=1/c2=2/part-r-0_a_1
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16041) HCatalog doesn't delete temp _SCRATCH dir when job failed

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885058#comment-15885058
 ] 

Hive QA commented on HIVE-16041:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12854802/HIVE-16041.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10266 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] 
(batchId=223)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3800/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3800/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3800/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12854802 - PreCommit-HIVE-Build

> HCatalog doesn't delete temp  _SCRATCH dir when job failed
> --
>
> Key: HIVE-16041
> URL: https://issues.apache.org/jira/browse/HIVE-16041
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 2.2.0
>Reporter: yunfei liu
>Assignee: yunfei liu
> Fix For: 2.2.0
>
> Attachments: HIVE-16041.1.patch, HIVE-16041.2.patch
>
>
> when we use HCatOutputFormat to write to an external partitioned table,  a 
> tmp dir (which starts with "_SCRATCH" ) will appear under table path if the 
> job failed. 
> {quote}
> drwxr-xr-x   - yun hdfs  0 2017-02-27 01:45 
> /tmp/hive/_SCRATCH0.31946356159329714
> drwxr-xr-x   - yun hdfs  0 2017-02-27 01:51 
> /tmp/hive/_SCRATCH0.31946356159329714/c1=1
> drwxr-xr-x   - yun hdfs  0 2017-02-27 00:57 /tmp/hive/c1=1
> drwxr-xr-x   - yun hdfs  0 2017-02-27 01:28 /tmp/hive/c1=1/c2=2
> -rw-r--r--   3 yun hdfs 12 2017-02-27 00:57 
> /tmp/hive/c1=1/c2=2/part-r-0
> -rw-r--r--   3 yun hdfs 12 2017-02-27 01:28 
> /tmp/hive/c1=1/c2=2/part-r-0_a_1
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

[Rui Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Li updated HIVE-15859:
--
Attachment: HIVE-15859.3.patch

Thanks [~KaiXu] for the clarifications. Update patch v3 to make sure we log the 
exception caught in the pipeline.
[~xuefuz], [~vanzin] could you please take a look? Thanks!

> Hive client side shows Spark Driver disconnected while Spark Driver side 
> could not get RPC header 
> --
>
> Key: HIVE-15859
> URL: https://issues.apache.org/jira/browse/HIVE-15859
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, Spark
>Affects Versions: 2.2.0
> Environment: hadoop2.7.1
> spark1.6.2
> hive2.2
>Reporter: KaiXu
>Assignee: Rui Li
> Attachments: HIVE-15859.1.patch, HIVE-15859.2.patch, 
> HIVE-15859.3.patch
>
>
> Hive on Spark, failed with error:
> {noformat}
> 2017-02-08 09:50:59,331 Stage-2_0: 1039(+2)/1041 Stage-3_0: 796(+456)/1520 
> Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> 2017-02-08 09:51:00,335 Stage-2_0: 1040(+1)/1041 Stage-3_0: 914(+398)/1520 
> Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> 2017-02-08 09:51:01,338 Stage-2_0: 1041/1041 Finished Stage-3_0: 
> 961(+383)/1520 Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> Failed to monitor Job[ 2] with exception 'java.lang.IllegalStateException(RPC 
> channel is closed.)'
> FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.spark.SparkTask
> {noformat}
> application log shows the driver commanded a shutdown with some unknown 
> reason, but hive's log shows Driver could not get RPC header( Expected RPC 
> header, got org.apache.hive.spark.client.rpc.Rpc$NullMessage instead).
> {noformat}
> 17/02/08 09:51:04 INFO exec.Utilities: PLAN PATH = 
> hdfs://hsx-node1:8020/tmp/hive/root/b723c85d-2a7b-469e-bab1-9c165b25e656/hive_2017-02-08_09-49-37_890_6267025825539539056-1/-mr-10006/71a9dacb-a463-40ef-9e86-78d3b8e3738d/map.xml
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 1169.0 in 
> stage 3.0 (TID 2519)
> 17/02/08 09:51:04 INFO executor.CoarseGrainedExecutorBackend: Driver 
> commanded a shutdown
> 17/02/08 09:51:04 INFO storage.MemoryStore: MemoryStore cleared
> 17/02/08 09:51:04 INFO storage.BlockManager: BlockManager stopped
> 17/02/08 09:51:04 INFO exec.Utilities: PLAN PATH = 
> hdfs://hsx-node1:8020/tmp/hive/root/b723c85d-2a7b-469e-bab1-9c165b25e656/hive_2017-02-08_09-49-37_890_6267025825539539056-1/-mr-10006/71a9dacb-a463-40ef-9e86-78d3b8e3738d/map.xml
> 17/02/08 09:51:04 WARN executor.CoarseGrainedExecutorBackend: An unknown 
> (hsx-node1:42777) driver disconnected.
> 17/02/08 09:51:04 ERROR executor.CoarseGrainedExecutorBackend: Driver 
> 192.168.1.1:42777 disassociated! Shutting down.
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 1105.0 in 
> stage 3.0 (TID 2511)
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Shutdown hook called
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator: 
> Shutting down remote daemon.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk6/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-71da1dfc-99bd-4687-bc2f-33452db8de3d
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk2/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-7f134d81-e77e-4b92-bd99-0a51d0962c14
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk5/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-77a90d63-fb05-4bc6-8d5e-1562cc502e6c
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator: 
> Remote daemon shut down; proceeding with flushing remote transports.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk4/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-91f8b91a-114d-4340-8560-d3cd085c1cd4
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk1/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-a3c24f9e-8609-48f0-9d37-0de7ae06682a
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator: 
> Remoting shut down.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk7/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-f6120a43-2158-4780-927c-c5786b78f53e
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk3/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-e17931ad-9e8a-45da-86f8-9a0fdca0fad1
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> 

[jira] [Commented] (HIVE-15848) count or sum distinct incorrect when hive.optimize.reducededuplication set to true

[Biao Wu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885037#comment-15885037
 ] 

Biao Wu commented on HIVE-15848:


Thanks  [~kgyrtkirk], I think  childDistinctColumnIndices should less than 2, 
the optimization is  enabled when childDistinctColumnIndices only have one key.
PR:  
https://github.com/apache/hive/pull/150/commits/a4fc3af4c77beafe11e3e4188571177862d64e4e


> count or sum distinct incorrect when hive.optimize.reducededuplication set to 
> true
> --
>
> Key: HIVE-15848
> URL: https://issues.apache.org/jira/browse/HIVE-15848
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Biao Wu
>Assignee: Zoltan Haindrich
>Priority: Critical
> Attachments: HIVE-15848.1.patch
>
>
> Test Table:
> {code:sql}
> create table test(id int,key int,name int);
> {code}
> Data：
> ||id||key||name||
> |1|1  |2
> |1|2  |3
> |1|3  |2
> |1|4  |2
> |1|5  |3
> Test SQL1:
> {code:sql}
> select id,count(Distinct key),count(Distinct name)
> from (select id,key,name from count_distinct_test group by id,key,name)m
> group by id;
> {code}
> result：
> |1|5|4
> expect:
> |1|5|2
> Test SQL2:
> {code:sql}
> select id,count(Distinct name),count(Distinct key)
> from (select id,key,name from count_distinct_test group by id,name,key)m
> group by id;
> {code}
> result:
> |1|2|5



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16041) HCatalog doesn't delete temp _SCRATCH dir when job failed

[yunfei liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yunfei liu updated HIVE-16041:
--
Attachment: HIVE-16041.2.patch

> HCatalog doesn't delete temp  _SCRATCH dir when job failed
> --
>
> Key: HIVE-16041
> URL: https://issues.apache.org/jira/browse/HIVE-16041
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 2.2.0
>Reporter: yunfei liu
>Assignee: yunfei liu
> Fix For: 2.2.0
>
> Attachments: HIVE-16041.1.patch, HIVE-16041.2.patch
>
>
> when we use HCatOutputFormat to write to an external partitioned table,  a 
> tmp dir (which starts with "_SCRATCH" ) will appear under table path if the 
> job failed. 
> {quote}
> drwxr-xr-x   - yun hdfs  0 2017-02-27 01:45 
> /tmp/hive/_SCRATCH0.31946356159329714
> drwxr-xr-x   - yun hdfs  0 2017-02-27 01:51 
> /tmp/hive/_SCRATCH0.31946356159329714/c1=1
> drwxr-xr-x   - yun hdfs  0 2017-02-27 00:57 /tmp/hive/c1=1
> drwxr-xr-x   - yun hdfs  0 2017-02-27 01:28 /tmp/hive/c1=1/c2=2
> -rw-r--r--   3 yun hdfs 12 2017-02-27 00:57 
> /tmp/hive/c1=1/c2=2/part-r-0
> -rw-r--r--   3 yun hdfs 12 2017-02-27 01:28 
> /tmp/hive/c1=1/c2=2/part-r-0_a_1
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15859) Hive client side shows Spark Driver disconnected while Spark Driver side could not get RPC header

[KaiXu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885007#comment-15885007
 ] 

KaiXu commented on HIVE-15859:
--

Hi [~xuefuz] and [~lirui], I have tried to run 3 times with the patch, 
currently the issue not occurred any more, though it's random previously, but 
can frequently reproduce. So I think the patch solved the issue, Thanks for all 
your efforts!

> Hive client side shows Spark Driver disconnected while Spark Driver side 
> could not get RPC header 
> --
>
> Key: HIVE-15859
> URL: https://issues.apache.org/jira/browse/HIVE-15859
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, Spark
>Affects Versions: 2.2.0
> Environment: hadoop2.7.1
> spark1.6.2
> hive2.2
>Reporter: KaiXu
>Assignee: Rui Li
> Attachments: HIVE-15859.1.patch, HIVE-15859.2.patch
>
>
> Hive on Spark, failed with error:
> {noformat}
> 2017-02-08 09:50:59,331 Stage-2_0: 1039(+2)/1041 Stage-3_0: 796(+456)/1520 
> Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> 2017-02-08 09:51:00,335 Stage-2_0: 1040(+1)/1041 Stage-3_0: 914(+398)/1520 
> Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> 2017-02-08 09:51:01,338 Stage-2_0: 1041/1041 Finished Stage-3_0: 
> 961(+383)/1520 Stage-4_0: 0/2021 Stage-5_0: 0/1009 Stage-6_0: 0/1
> Failed to monitor Job[ 2] with exception 'java.lang.IllegalStateException(RPC 
> channel is closed.)'
> FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.spark.SparkTask
> {noformat}
> application log shows the driver commanded a shutdown with some unknown 
> reason, but hive's log shows Driver could not get RPC header( Expected RPC 
> header, got org.apache.hive.spark.client.rpc.Rpc$NullMessage instead).
> {noformat}
> 17/02/08 09:51:04 INFO exec.Utilities: PLAN PATH = 
> hdfs://hsx-node1:8020/tmp/hive/root/b723c85d-2a7b-469e-bab1-9c165b25e656/hive_2017-02-08_09-49-37_890_6267025825539539056-1/-mr-10006/71a9dacb-a463-40ef-9e86-78d3b8e3738d/map.xml
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 1169.0 in 
> stage 3.0 (TID 2519)
> 17/02/08 09:51:04 INFO executor.CoarseGrainedExecutorBackend: Driver 
> commanded a shutdown
> 17/02/08 09:51:04 INFO storage.MemoryStore: MemoryStore cleared
> 17/02/08 09:51:04 INFO storage.BlockManager: BlockManager stopped
> 17/02/08 09:51:04 INFO exec.Utilities: PLAN PATH = 
> hdfs://hsx-node1:8020/tmp/hive/root/b723c85d-2a7b-469e-bab1-9c165b25e656/hive_2017-02-08_09-49-37_890_6267025825539539056-1/-mr-10006/71a9dacb-a463-40ef-9e86-78d3b8e3738d/map.xml
> 17/02/08 09:51:04 WARN executor.CoarseGrainedExecutorBackend: An unknown 
> (hsx-node1:42777) driver disconnected.
> 17/02/08 09:51:04 ERROR executor.CoarseGrainedExecutorBackend: Driver 
> 192.168.1.1:42777 disassociated! Shutting down.
> 17/02/08 09:51:04 INFO executor.Executor: Executor killed task 1105.0 in 
> stage 3.0 (TID 2511)
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Shutdown hook called
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator: 
> Shutting down remote daemon.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk6/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-71da1dfc-99bd-4687-bc2f-33452db8de3d
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk2/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-7f134d81-e77e-4b92-bd99-0a51d0962c14
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk5/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-77a90d63-fb05-4bc6-8d5e-1562cc502e6c
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator: 
> Remote daemon shut down; proceeding with flushing remote transports.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk4/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-91f8b91a-114d-4340-8560-d3cd085c1cd4
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk1/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-a3c24f9e-8609-48f0-9d37-0de7ae06682a
> 17/02/08 09:51:04 INFO remote.RemoteActorRefProvider$RemotingTerminator: 
> Remoting shut down.
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk7/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-f6120a43-2158-4780-927c-c5786b78f53e
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> /mnt/disk3/yarn/nm/usercache/root/appcache/application_1486453422616_0150/spark-e17931ad-9e8a-45da-86f8-9a0fdca0fad1
> 17/02/08 09:51:04 INFO util.ShutdownHookManager: Deleting directory 
> 

[jira] [Commented] (HIVE-12274) Increase width of columns used for general configuration in the metastore.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884970#comment-15884970
 ] 

Hive QA commented on HIVE-12274:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12854796/HIVE-12274.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 274 failed/errored test(s), 6267 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.org.apache.hadoop.hive.cli.TestBlobstoreCliDriver
 (batchId=231)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_dynamic_partitions]
 (batchId=231)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_table]
 (batchId=231)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_directory]
 (batchId=231)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions]
 (batchId=231)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_table]
 (batchId=231)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[write_final_output_blobstore]
 (batchId=231)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
 (batchId=43)

[jira] [Commented] (HIVE-15430) Change SchemaTool table validator to test based on the dbType

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884943#comment-15884943
 ] 

Hive QA commented on HIVE-15430:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12854789/HIVE-15430.4.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10266 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] 
(batchId=223)
org.apache.hive.hcatalog.pig.TestTextFileHCatStorer.testWriteDecimal 
(batchId=173)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3798/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3798/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3798/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: NonZeroExitCodeException
Command 'cd /data/hiveptest/logs/PreCommit-HIVE-Build-3798/ && tar -zvcf 
test-results.tar.gz test-results/' failed with exit status 2 and output 
'test-results/
test-results/TEST-261_UTBatch_ql_10_tests-TEST-org.apache.hadoop.hive.ql.exec.vector.expressions.gen.TestColumnScalarOperationVectorExpressionEvaluation.xml
test-results/TEST-276_UTBatch_serde_20_tests-TEST-org.apache.hadoop.hive.serde2.avro.TestThatEvolvedSchemasActAsWeWant.xml
test-results/TEST-271_UTBatch_ql_10_tests-TEST-org.apache.hadoop.hive.ql.lockmgr.zookeeper.TestZookeeperLockManager.xml
test-results/TEST-195_UTBatch_itests__hive-unit_9_tests-TEST-org.apache.hadoop.hive.metastore.TestMetaStoreConnectionUrlHook.xml
test-results/TEST-180_UTBatch_hcatalog__core_9_tests-TEST-org.apache.hive.hcatalog.mapreduce.TestHCatInputFormat.xml
test-results/TEST-182_UTBatch_hcatalog__streaming_16_tests-TEST-org.apache.hive.hcatalog.streaming.mutate.worker.TestRecordInspectorImpl.xml
test-results/TEST-252_UTBatch_ql_10_tests-TEST-org.apache.hadoop.hive.ql.parse.TestSplitSample.xml
test-results/TEST-262_UTBatch_ql_10_tests-TEST-org.apache.hadoop.hive.ql.exec.vector.TestVectorSerDeRow.xml
test-results/TEST-126-TestSparkCliDriver-ptf_seqfile.q-union_remove_23.q-parallel_join0.q-and-12-more-TEST-org.apache.hadoop.hive.cli.TestSparkCliDriver.xml
test-results/TEST-173_UTBatch_hcatalog__hcatalog-pig-adapter_20_tests-TEST-org.apache.hive.hcatalog.pig.TestUtil.xml
test-results/TEST-168_UTBatch_beeline_13_tests-TEST-org.apache.hive.beeline.TestClientCommandHookFactory.xml
test-results/TEST-241_UTBatch_ql_10_tests-TEST-org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFBridge.xml
test-results/TEST-173_UTBatch_hcatalog__hcatalog-pig-adapter_20_tests-TEST-org.apache.hive.hcatalog.pig.TestHCatLoaderStorer.xml
test-results/TEST-203_UTBatch_itests__hive-unit_9_tests-TEST-org.apache.hadoop.hive.metastore.TestHiveMetaStoreTxns.xml
test-results/TEST-169_UTBatch_storage-api_13_tests-TEST-org.apache.hadoop.hive.ql.exec.vector.TestBytesColumnVector.xml
test-results/TEST-182_UTBatch_hcatalog__streaming_16_tests-TEST-org.apache.hive.hcatalog.streaming.mutate.worker.TestMutatorCoordinator.xml
test-results/TEST-213_UTBatch_itests__hive-unit_9_tests-TEST-org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.xml
test-results/TEST-87-TestNegativeMinimrCliDriver-udf_local_resource.q-mapreduce_stack_trace.q-minimr_broken_pipe.q-and-5-more-TEST-org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.xml
test-results/TEST-250_UTBatch_ql_10_tests-TEST-org.apache.hadoop.hive.ql.io.orc.TestOrcSplitElimination.xml
test-results/TEST-210_UTBatch_itests__hive-unit_9_tests-TEST-org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.xml
test-results/TEST-14-TestCliDriver-authorization_create_temp_table.q-skewjoinopt16.q-drop_partitions_filter3.q-and-27-more-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml
test-results/TEST-185_UTBatch_service_8_tests-TEST-org.apache.hive.service.auth.TestLdapAtnProviderWithMiniDS.xml
test-results/TEST-275_UTBatch_serde_20_tests-TEST-org.apache.hadoop.hive.serde2.objectinspector.TestStandardObjectInspectors.xml
test-results/TEST-73-TestCliDriver-tez_union_group_by.q-llap_acid.q-truncate_table.q-and-27-more-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml
test-results/TEST-199_UTBatch_itests__hive-unit_9_tests-TEST-org.apache.hadoop.hive.metastore.hbase.TestHBaseSchemaTool2.xml
test-results/TEST-277_UTBatch_serde_6_tests-TEST-org.apache.hadoop.hive.serde2.columnar.TestLazyBinaryColumnarSerDe.xml

[jira] [Updated] (HIVE-12274) Increase width of columns used for general configuration in the metastore.

[Naveen Gangam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-12274:
-
Attachment: HIVE-12274.patch

> Increase width of columns used for general configuration in the metastore.
> --
>
> Key: HIVE-12274
> URL: https://issues.apache.org/jira/browse/HIVE-12274
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Elliot West
>Assignee: Naveen Gangam
>  Labels: metastore
> Attachments: HIVE-12274.example.ddl.hql, HIVE-12274.patch, 
> HIVE-12274.patch, HIVE-12274.patch
>
>
> h2. Overview
> This issue is very similar in principle to HIVE-1364. We are hitting a limit 
> when processing JSON data that has a large nested schema. The struct 
> definition is truncated when inserted into the metastore database column 
> {{COLUMNS_V2.YPE_NAME}} as it is greater than 4000 characters in length.
> Given that the purpose of these columns is to hold very loosely defined 
> configuration values it seems rather limiting to impose such a relatively low 
> length bound. One can imagine that valid use cases will arise where 
> reasonable parameter/property values exceed the current limit. 
> h2. Context
> These limitations were in by the [patch 
> attributed|https://github.com/apache/hive/commit/c21a526b0a752df2a51d20a2729cc8493c228799]
>  to HIVE-1364 which mentions the _"max length on Oracle 9i/10g/11g"_ as the 
> reason. However, nowadays the limit can be increased because:
> * Oracle DB's {{varchar2}} supports 32767 bytes now, by setting the 
> configuration parameter {{MAX_STRING_SIZE}} to {{EXTENDED}}. 
> ([source|http://docs.oracle.com/database/121/SQLRF/sql_elements001.htm#SQLRF55623])
> * Postgres supports a max of 1GB for {{character}} datatype. 
> ([source|http://www.postgresql.org/docs/8.3/static/datatype-character.html])
> * MySQL can support upto 65535 bytes for the entire row. So long as the 
> {{PARAM_KEY}} value + {{PARAM_VALUE}} is less than 65535, we should be good. 
> ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * SQL Server's {{varchar}} max length is 8000 and can go beyond using 
> "varchar(max)" with the same limitation as MySQL being 65535 bytes for the 
> entire row. ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * Derby's {{varchar}} can be upto 32672 bytes. 
> ([source|https://db.apache.org/derby/docs/10.7/ref/rrefsqlj41207.html])
> h2. Proposal
> Can these columns not use CLOB-like types as for example as used by 
> {{TBLS.VIEW_EXPANDED_TEXT}}? It would seem that suitable type equivalents 
> exist for all targeted database platforms:
> * MySQL: {{mediumtext}}
> * Postgres: {{text}}
> * Oracle: {{CLOB}}
> * Derby: {{LONG VARCHAR}}
> I'd suggest that the candidates for type change are:
> * {{COLUMNS_V2.TYPE_NAME}}
> * {{TABLE_PARAMS.PARAM_VALUE}}
> * {{SERDE_PARAMS.PARAM_VALUE}}
> * {{SD_PARAMS.PARAM_VALUE}}
> After updating the maximum length the metastore database needs to be 
> configured and restarted with the new settings. Altering {{MAX_STRING_SIZE}} 
> will update database objects and possibly invalidate them, as follows:
> * Tables with virtual columns will be updated with new data type metadata for 
> virtual columns of {{VARCHAR2(4000)}}, 4000-byte {{NVARCHAR2}}, or 
> {{RAW(2000)}} type.
> * Functional indexes will become unusable if a change to their associated 
> virtual columns causes the index key to exceed index key length limits. 
> Attempts to rebuild such indexes will fail with {{ORA-01450: maximum key 
> length exceeded}}.
> * Views will be invalidated if they contain {{VARCHAR2(4000)}}, 4000-byte 
> {{NVARCHAR2}}, or {{RAW(2000)}} typed expression columns.
> * Materialized views will be updated with new metadata {{VARCHAR2(4000)}}, 
> 4000-byte {{NVARCHAR2}}, and {{RAW(2000)}} typed expression columns
> * So the limitation could be raised to 32672 bytes, with the caveat that 
> MySQL and SQL Server limit the row length to 65535 bytes, so that should also 
> be validated to provide consistency.
> Finally, will this limitation persist in the work resulting from HIVE-9452?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-12274) Increase width of columns used for general configuration in the metastore.

[Naveen Gangam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-12274:
-
Status: Patch Available  (was: Open)

pre-commit still fails with some execution failure. Retrying

> Increase width of columns used for general configuration in the metastore.
> --
>
> Key: HIVE-12274
> URL: https://issues.apache.org/jira/browse/HIVE-12274
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Elliot West
>Assignee: Naveen Gangam
>  Labels: metastore
> Attachments: HIVE-12274.example.ddl.hql, HIVE-12274.patch, 
> HIVE-12274.patch, HIVE-12274.patch
>
>
> h2. Overview
> This issue is very similar in principle to HIVE-1364. We are hitting a limit 
> when processing JSON data that has a large nested schema. The struct 
> definition is truncated when inserted into the metastore database column 
> {{COLUMNS_V2.YPE_NAME}} as it is greater than 4000 characters in length.
> Given that the purpose of these columns is to hold very loosely defined 
> configuration values it seems rather limiting to impose such a relatively low 
> length bound. One can imagine that valid use cases will arise where 
> reasonable parameter/property values exceed the current limit. 
> h2. Context
> These limitations were in by the [patch 
> attributed|https://github.com/apache/hive/commit/c21a526b0a752df2a51d20a2729cc8493c228799]
>  to HIVE-1364 which mentions the _"max length on Oracle 9i/10g/11g"_ as the 
> reason. However, nowadays the limit can be increased because:
> * Oracle DB's {{varchar2}} supports 32767 bytes now, by setting the 
> configuration parameter {{MAX_STRING_SIZE}} to {{EXTENDED}}. 
> ([source|http://docs.oracle.com/database/121/SQLRF/sql_elements001.htm#SQLRF55623])
> * Postgres supports a max of 1GB for {{character}} datatype. 
> ([source|http://www.postgresql.org/docs/8.3/static/datatype-character.html])
> * MySQL can support upto 65535 bytes for the entire row. So long as the 
> {{PARAM_KEY}} value + {{PARAM_VALUE}} is less than 65535, we should be good. 
> ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * SQL Server's {{varchar}} max length is 8000 and can go beyond using 
> "varchar(max)" with the same limitation as MySQL being 65535 bytes for the 
> entire row. ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * Derby's {{varchar}} can be upto 32672 bytes. 
> ([source|https://db.apache.org/derby/docs/10.7/ref/rrefsqlj41207.html])
> h2. Proposal
> Can these columns not use CLOB-like types as for example as used by 
> {{TBLS.VIEW_EXPANDED_TEXT}}? It would seem that suitable type equivalents 
> exist for all targeted database platforms:
> * MySQL: {{mediumtext}}
> * Postgres: {{text}}
> * Oracle: {{CLOB}}
> * Derby: {{LONG VARCHAR}}
> I'd suggest that the candidates for type change are:
> * {{COLUMNS_V2.TYPE_NAME}}
> * {{TABLE_PARAMS.PARAM_VALUE}}
> * {{SERDE_PARAMS.PARAM_VALUE}}
> * {{SD_PARAMS.PARAM_VALUE}}
> After updating the maximum length the metastore database needs to be 
> configured and restarted with the new settings. Altering {{MAX_STRING_SIZE}} 
> will update database objects and possibly invalidate them, as follows:
> * Tables with virtual columns will be updated with new data type metadata for 
> virtual columns of {{VARCHAR2(4000)}}, 4000-byte {{NVARCHAR2}}, or 
> {{RAW(2000)}} type.
> * Functional indexes will become unusable if a change to their associated 
> virtual columns causes the index key to exceed index key length limits. 
> Attempts to rebuild such indexes will fail with {{ORA-01450: maximum key 
> length exceeded}}.
> * Views will be invalidated if they contain {{VARCHAR2(4000)}}, 4000-byte 
> {{NVARCHAR2}}, or {{RAW(2000)}} typed expression columns.
> * Materialized views will be updated with new metadata {{VARCHAR2(4000)}}, 
> 4000-byte {{NVARCHAR2}}, and {{RAW(2000)}} typed expression columns
> * So the limitation could be raised to 32672 bytes, with the caveat that 
> MySQL and SQL Server limit the row length to 65535 bytes, so that should also 
> be validated to provide consistency.
> Finally, will this limitation persist in the work resulting from HIVE-9452?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-12274) Increase width of columns used for general configuration in the metastore.

[Naveen Gangam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-12274:
-
Status: Open  (was: Patch Available)

> Increase width of columns used for general configuration in the metastore.
> --
>
> Key: HIVE-12274
> URL: https://issues.apache.org/jira/browse/HIVE-12274
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Elliot West
>Assignee: Naveen Gangam
>  Labels: metastore
> Attachments: HIVE-12274.example.ddl.hql, HIVE-12274.patch, 
> HIVE-12274.patch
>
>
> h2. Overview
> This issue is very similar in principle to HIVE-1364. We are hitting a limit 
> when processing JSON data that has a large nested schema. The struct 
> definition is truncated when inserted into the metastore database column 
> {{COLUMNS_V2.YPE_NAME}} as it is greater than 4000 characters in length.
> Given that the purpose of these columns is to hold very loosely defined 
> configuration values it seems rather limiting to impose such a relatively low 
> length bound. One can imagine that valid use cases will arise where 
> reasonable parameter/property values exceed the current limit. 
> h2. Context
> These limitations were in by the [patch 
> attributed|https://github.com/apache/hive/commit/c21a526b0a752df2a51d20a2729cc8493c228799]
>  to HIVE-1364 which mentions the _"max length on Oracle 9i/10g/11g"_ as the 
> reason. However, nowadays the limit can be increased because:
> * Oracle DB's {{varchar2}} supports 32767 bytes now, by setting the 
> configuration parameter {{MAX_STRING_SIZE}} to {{EXTENDED}}. 
> ([source|http://docs.oracle.com/database/121/SQLRF/sql_elements001.htm#SQLRF55623])
> * Postgres supports a max of 1GB for {{character}} datatype. 
> ([source|http://www.postgresql.org/docs/8.3/static/datatype-character.html])
> * MySQL can support upto 65535 bytes for the entire row. So long as the 
> {{PARAM_KEY}} value + {{PARAM_VALUE}} is less than 65535, we should be good. 
> ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * SQL Server's {{varchar}} max length is 8000 and can go beyond using 
> "varchar(max)" with the same limitation as MySQL being 65535 bytes for the 
> entire row. ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * Derby's {{varchar}} can be upto 32672 bytes. 
> ([source|https://db.apache.org/derby/docs/10.7/ref/rrefsqlj41207.html])
> h2. Proposal
> Can these columns not use CLOB-like types as for example as used by 
> {{TBLS.VIEW_EXPANDED_TEXT}}? It would seem that suitable type equivalents 
> exist for all targeted database platforms:
> * MySQL: {{mediumtext}}
> * Postgres: {{text}}
> * Oracle: {{CLOB}}
> * Derby: {{LONG VARCHAR}}
> I'd suggest that the candidates for type change are:
> * {{COLUMNS_V2.TYPE_NAME}}
> * {{TABLE_PARAMS.PARAM_VALUE}}
> * {{SERDE_PARAMS.PARAM_VALUE}}
> * {{SD_PARAMS.PARAM_VALUE}}
> After updating the maximum length the metastore database needs to be 
> configured and restarted with the new settings. Altering {{MAX_STRING_SIZE}} 
> will update database objects and possibly invalidate them, as follows:
> * Tables with virtual columns will be updated with new data type metadata for 
> virtual columns of {{VARCHAR2(4000)}}, 4000-byte {{NVARCHAR2}}, or 
> {{RAW(2000)}} type.
> * Functional indexes will become unusable if a change to their associated 
> virtual columns causes the index key to exceed index key length limits. 
> Attempts to rebuild such indexes will fail with {{ORA-01450: maximum key 
> length exceeded}}.
> * Views will be invalidated if they contain {{VARCHAR2(4000)}}, 4000-byte 
> {{NVARCHAR2}}, or {{RAW(2000)}} typed expression columns.
> * Materialized views will be updated with new metadata {{VARCHAR2(4000)}}, 
> 4000-byte {{NVARCHAR2}}, and {{RAW(2000)}} typed expression columns
> * So the limitation could be raised to 32672 bytes, with the caveat that 
> MySQL and SQL Server limit the row length to 65535 bytes, so that should also 
> be validated to provide consistency.
> Finally, will this limitation persist in the work resulting from HIVE-9452?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15430) Change SchemaTool table validator to test based on the dbType

[Naveen Gangam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-15430:
-
Status: Patch Available  (was: Open)

> Change SchemaTool table validator to test based on the dbType
> -
>
> Key: HIVE-15430
> URL: https://issues.apache.org/jira/browse/HIVE-15430
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Affects Versions: 2.2.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Minor
> Attachments: HIVE-15430.1.patch, HIVE-15430.2.patch, 
> HIVE-15430.3.patch, HIVE-15430.4.patch
>
>
> Currently the validator parses the "oracle" schema file to determine what 
> tables are expected in the database.  (mostly because of ease of parsing the 
> schema file compared to other syntax). We have learnt from HIVE-15118, that 
> not all schema files have the same amount of tables. For example, derby has 
> an old table that is never used that other DBs do not contain).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15430) Change SchemaTool table validator to test based on the dbType

[Naveen Gangam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-15430:
-
Status: Open  (was: Patch Available)

Got some additional feedback. Will attach patch based on the new feedback.

> Change SchemaTool table validator to test based on the dbType
> -
>
> Key: HIVE-15430
> URL: https://issues.apache.org/jira/browse/HIVE-15430
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Affects Versions: 2.2.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Minor
> Attachments: HIVE-15430.1.patch, HIVE-15430.2.patch, 
> HIVE-15430.3.patch, HIVE-15430.4.patch
>
>
> Currently the validator parses the "oracle" schema file to determine what 
> tables are expected in the database.  (mostly because of ease of parsing the 
> schema file compared to other syntax). We have learnt from HIVE-15118, that 
> not all schema files have the same amount of tables. For example, derby has 
> an old table that is never used that other DBs do not contain).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15430) Change SchemaTool table validator to test based on the dbType

[Naveen Gangam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-15430:
-
Attachment: HIVE-15430.4.patch

> Change SchemaTool table validator to test based on the dbType
> -
>
> Key: HIVE-15430
> URL: https://issues.apache.org/jira/browse/HIVE-15430
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Affects Versions: 2.2.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Minor
> Attachments: HIVE-15430.1.patch, HIVE-15430.2.patch, 
> HIVE-15430.3.patch, HIVE-15430.4.patch
>
>
> Currently the validator parses the "oracle" schema file to determine what 
> tables are expected in the database.  (mostly because of ease of parsing the 
> schema file compared to other syntax). We have learnt from HIVE-15118, that 
> not all schema files have the same amount of tables. For example, derby has 
> an old table that is never used that other DBs do not contain).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16041) HCatalog doesn't delete temp _SCRATCH dir when job failed

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884799#comment-15884799
 ] 

Hive QA commented on HIVE-16041:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12854774/HIVE-16041.1.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3797/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3797/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3797/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Tests exited with: ExecutionException: java.util.concurrent.ExecutionException: 
org.apache.hive.ptest.execution.ssh.SSHExecutionException: RSyncResult 
[localFile=/data/hiveptest/logs/PreCommit-HIVE-Build-3797/succeeded/268_TestTxnCommands2WithSplitUpdate,
 remoteFile=/home/hiveptest/104.154.183.241-hiveptest-0/logs/, 
getExitCode()=11, getException()=null, getUser()=hiveptest, 
getHost()=104.154.183.241, getInstance()=0]: 'Warning: Permanently added 
'104.154.183.241' (ECDSA) to the list of known hosts.
receiving incremental file list
./
TEST-268_TestTxnCommands2WithSplitUpdate-TEST-org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdate.xml

  0   0%0.00kB/s0:00:00  
 10,167 100%9.70MB/s0:00:00 (xfr#1, to-chk=5/7)
maven-test.txt

  0   0%0.00kB/s0:00:00  
  5,930 100%5.66MB/s0:00:00 (xfr#2, to-chk=4/7)
logs/
logs/derby.log

  0   0%0.00kB/s0:00:00  
978 100%  955.08kB/s0:00:00 (xfr#3, to-chk=1/7)
logs/hive.log

  0   0%0.00kB/s0:00:00  
rsync: write failed on 
"/data/hiveptest/logs/PreCommit-HIVE-Build-3797/succeeded/268_TestTxnCommands2WithSplitUpdate/logs/hive.log":
 No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(393) [receiver=3.1.1]
Warning: Permanently added '104.154.183.241' (ECDSA) to the list of known hosts.
receiving incremental file list
logs/
logs/hive.log

  0   0%0.00kB/s0:00:00  
rsync: write failed on 
"/data/hiveptest/logs/PreCommit-HIVE-Build-3797/succeeded/268_TestTxnCommands2WithSplitUpdate/logs/hive.log":
 No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(393) [receiver=3.1.1]
Warning: Permanently added '104.154.183.241' (ECDSA) to the list of known hosts.
receiving incremental file list
logs/
logs/hive.log

  0   0%0.00kB/s0:00:00  
rsync: write failed on 
"/data/hiveptest/logs/PreCommit-HIVE-Build-3797/succeeded/268_TestTxnCommands2WithSplitUpdate/logs/hive.log":
 No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(393) [receiver=3.1.1]
Warning: Permanently added '104.154.183.241' (ECDSA) to the list of known hosts.
receiving incremental file list
logs/
logs/hive.log

  0   0%0.00kB/s0:00:00  
rsync: write failed on 
"/data/hiveptest/logs/PreCommit-HIVE-Build-3797/succeeded/268_TestTxnCommands2WithSplitUpdate/logs/hive.log":
 No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(393) [receiver=3.1.1]
Warning: Permanently added '104.154.183.241' (ECDSA) to the list of known hosts.
receiving incremental file list
logs/
logs/hive.log

  0   0%0.00kB/s0:00:00  
rsync: write failed on 
"/data/hiveptest/logs/PreCommit-HIVE-Build-3797/succeeded/268_TestTxnCommands2WithSplitUpdate/logs/hive.log":
 No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(393) [receiver=3.1.1]
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12854774 - PreCommit-HIVE-Build

> HCatalog doesn't delete temp  _SCRATCH dir when job failed
> --
>
> Key: HIVE-16041
> URL: https://issues.apache.org/jira/browse/HIVE-16041
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 2.2.0
>Reporter: yunfei liu
>Assignee: yunfei liu
> Fix For: 2.2.0
>
> Attachments: HIVE-16041.1.patch
>
>
> when we use HCatOutputFormat to write to an external partitioned table,  a 
> tmp dir (which starts with "_SCRATCH" ) will appear under table path if the 
> job failed. 
> {quote}
> drwxr-xr-x   - yun hdfs  0 2017-02-27 01:45 
> /tmp/hive/_SCRATCH0.31946356159329714
> drwxr-xr-x   - yun hdfs  0 2017-02-27 01:51 
> /tmp/hive/_SCRATCH0.31946356159329714/c1=1
> drwxr-xr-x   - yun hdfs  0 2017-02-27 00:57 /tmp/hive/c1=1
> drwxr-xr-x   - yun hdfs  0 2017-02-27 01:28 /tmp/hive/c1=1/c2=2
> -rw-r--r--   3 yun hdfs 12 

[jira] [Updated] (HIVE-16041) HCatalog doesn't delete temp _SCRATCH dir when job failed

[yunfei liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yunfei liu updated HIVE-16041:
--
Fix Version/s: 2.2.0
   Status: Patch Available  (was: Open)

> HCatalog doesn't delete temp  _SCRATCH dir when job failed
> --
>
> Key: HIVE-16041
> URL: https://issues.apache.org/jira/browse/HIVE-16041
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 2.2.0
>Reporter: yunfei liu
>Assignee: yunfei liu
> Fix For: 2.2.0
>
> Attachments: HIVE-16041.1.patch
>
>
> when we use HCatOutputFormat to write to an external partitioned table,  a 
> tmp dir (which starts with "_SCRATCH" ) will appear under table path if the 
> job failed. 
> {quote}
> drwxr-xr-x   - yun hdfs  0 2017-02-27 01:45 
> /tmp/hive/_SCRATCH0.31946356159329714
> drwxr-xr-x   - yun hdfs  0 2017-02-27 01:51 
> /tmp/hive/_SCRATCH0.31946356159329714/c1=1
> drwxr-xr-x   - yun hdfs  0 2017-02-27 00:57 /tmp/hive/c1=1
> drwxr-xr-x   - yun hdfs  0 2017-02-27 01:28 /tmp/hive/c1=1/c2=2
> -rw-r--r--   3 yun hdfs 12 2017-02-27 00:57 
> /tmp/hive/c1=1/c2=2/part-r-0
> -rw-r--r--   3 yun hdfs 12 2017-02-27 01:28 
> /tmp/hive/c1=1/c2=2/part-r-0_a_1
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16041) HCatalog doesn't delete temp _SCRATCH dir when job failed

[yunfei liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yunfei liu updated HIVE-16041:
--
Attachment: HIVE-16041.1.patch

> HCatalog doesn't delete temp  _SCRATCH dir when job failed
> --
>
> Key: HIVE-16041
> URL: https://issues.apache.org/jira/browse/HIVE-16041
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 2.2.0
>Reporter: yunfei liu
>Assignee: yunfei liu
> Attachments: HIVE-16041.1.patch
>
>
> when we use HCatOutputFormat to write to an external partitioned table,  a 
> tmp dir (which starts with "_SCRATCH" ) will appear under table path if the 
> job failed. 
> {quote}
> drwxr-xr-x   - yun hdfs  0 2017-02-27 01:45 
> /tmp/hive/_SCRATCH0.31946356159329714
> drwxr-xr-x   - yun hdfs  0 2017-02-27 01:51 
> /tmp/hive/_SCRATCH0.31946356159329714/c1=1
> drwxr-xr-x   - yun hdfs  0 2017-02-27 00:57 /tmp/hive/c1=1
> drwxr-xr-x   - yun hdfs  0 2017-02-27 01:28 /tmp/hive/c1=1/c2=2
> -rw-r--r--   3 yun hdfs 12 2017-02-27 00:57 
> /tmp/hive/c1=1/c2=2/part-r-0
> -rw-r--r--   3 yun hdfs 12 2017-02-27 01:28 
> /tmp/hive/c1=1/c2=2/part-r-0_a_1
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16041) HCatalog doesn't delete temp _SCRATCH dir when job failed

[yunfei liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yunfei liu updated HIVE-16041:
--
Description: 
when we use HCatOutputFormat to write to an external partitioned table,  a tmp 
dir (which starts with "_SCRATCH" ) will appear under table path if the job 
failed. 

{quote}
drwxr-xr-x   - yun hdfs  0 2017-02-27 01:45 
/tmp/hive/_SCRATCH0.31946356159329714
drwxr-xr-x   - yun hdfs  0 2017-02-27 01:51 
/tmp/hive/_SCRATCH0.31946356159329714/c1=1
drwxr-xr-x   - yun hdfs  0 2017-02-27 00:57 /tmp/hive/c1=1
drwxr-xr-x   - yun hdfs  0 2017-02-27 01:28 /tmp/hive/c1=1/c2=2
-rw-r--r--   3 yun hdfs 12 2017-02-27 00:57 
/tmp/hive/c1=1/c2=2/part-r-0
-rw-r--r--   3 yun hdfs 12 2017-02-27 01:28 
/tmp/hive/c1=1/c2=2/part-r-0_a_1
{quote}


  was:
when we use HCatOutputFormat to write to an external partitioned table,  a tmp 
dir (which starts with "_SCRATCH" ) will appear under table path if the job 
failed. 



> HCatalog doesn't delete temp  _SCRATCH dir when job failed
> --
>
> Key: HIVE-16041
> URL: https://issues.apache.org/jira/browse/HIVE-16041
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 2.2.0
>Reporter: yunfei liu
>Assignee: yunfei liu
>
> when we use HCatOutputFormat to write to an external partitioned table,  a 
> tmp dir (which starts with "_SCRATCH" ) will appear under table path if the 
> job failed. 
> {quote}
> drwxr-xr-x   - yun hdfs  0 2017-02-27 01:45 
> /tmp/hive/_SCRATCH0.31946356159329714
> drwxr-xr-x   - yun hdfs  0 2017-02-27 01:51 
> /tmp/hive/_SCRATCH0.31946356159329714/c1=1
> drwxr-xr-x   - yun hdfs  0 2017-02-27 00:57 /tmp/hive/c1=1
> drwxr-xr-x   - yun hdfs  0 2017-02-27 01:28 /tmp/hive/c1=1/c2=2
> -rw-r--r--   3 yun hdfs 12 2017-02-27 00:57 
> /tmp/hive/c1=1/c2=2/part-r-0
> -rw-r--r--   3 yun hdfs 12 2017-02-27 01:28 
> /tmp/hive/c1=1/c2=2/part-r-0_a_1
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16041) HCatalog doesn't delete temp _SCRATCH dir when job failed

[yunfei liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yunfei liu reassigned HIVE-16041:
-


> HCatalog doesn't delete temp  _SCRATCH dir when job failed
> --
>
> Key: HIVE-16041
> URL: https://issues.apache.org/jira/browse/HIVE-16041
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 2.2.0
>Reporter: yunfei liu
>Assignee: yunfei liu
>
> when we use HCatOutputFormat to write to an external partitioned table,  a 
> tmp dir (which starts with "_SCRATCH" ) will appear under table path if the 
> job failed. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-12274) Increase width of columns used for general configuration in the metastore.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884752#comment-15884752
 ] 

Hive QA commented on HIVE-12274:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12854765/HIVE-12274.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3796/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3796/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3796/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Tests exited with: ExecutionException: java.util.concurrent.ExecutionException: 
java.io.IOException: Error writing to 
/data/hiveptest/working/scratch/hiveptest-29-TestCliDriver-udf_bitwise_and.q-mapjoin_emit_interval.q-date_udf.q-and-27-more.sh
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12854765 - PreCommit-HIVE-Build

> Increase width of columns used for general configuration in the metastore.
> --
>
> Key: HIVE-12274
> URL: https://issues.apache.org/jira/browse/HIVE-12274
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Elliot West
>Assignee: Naveen Gangam
>  Labels: metastore
> Attachments: HIVE-12274.example.ddl.hql, HIVE-12274.patch, 
> HIVE-12274.patch
>
>
> h2. Overview
> This issue is very similar in principle to HIVE-1364. We are hitting a limit 
> when processing JSON data that has a large nested schema. The struct 
> definition is truncated when inserted into the metastore database column 
> {{COLUMNS_V2.YPE_NAME}} as it is greater than 4000 characters in length.
> Given that the purpose of these columns is to hold very loosely defined 
> configuration values it seems rather limiting to impose such a relatively low 
> length bound. One can imagine that valid use cases will arise where 
> reasonable parameter/property values exceed the current limit. 
> h2. Context
> These limitations were in by the [patch 
> attributed|https://github.com/apache/hive/commit/c21a526b0a752df2a51d20a2729cc8493c228799]
>  to HIVE-1364 which mentions the _"max length on Oracle 9i/10g/11g"_ as the 
> reason. However, nowadays the limit can be increased because:
> * Oracle DB's {{varchar2}} supports 32767 bytes now, by setting the 
> configuration parameter {{MAX_STRING_SIZE}} to {{EXTENDED}}. 
> ([source|http://docs.oracle.com/database/121/SQLRF/sql_elements001.htm#SQLRF55623])
> * Postgres supports a max of 1GB for {{character}} datatype. 
> ([source|http://www.postgresql.org/docs/8.3/static/datatype-character.html])
> * MySQL can support upto 65535 bytes for the entire row. So long as the 
> {{PARAM_KEY}} value + {{PARAM_VALUE}} is less than 65535, we should be good. 
> ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * SQL Server's {{varchar}} max length is 8000 and can go beyond using 
> "varchar(max)" with the same limitation as MySQL being 65535 bytes for the 
> entire row. ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * Derby's {{varchar}} can be upto 32672 bytes. 
> ([source|https://db.apache.org/derby/docs/10.7/ref/rrefsqlj41207.html])
> h2. Proposal
> Can these columns not use CLOB-like types as for example as used by 
> {{TBLS.VIEW_EXPANDED_TEXT}}? It would seem that suitable type equivalents 
> exist for all targeted database platforms:
> * MySQL: {{mediumtext}}
> * Postgres: {{text}}
> * Oracle: {{CLOB}}
> * Derby: {{LONG VARCHAR}}
> I'd suggest that the candidates for type change are:
> * {{COLUMNS_V2.TYPE_NAME}}
> * {{TABLE_PARAMS.PARAM_VALUE}}
> * {{SERDE_PARAMS.PARAM_VALUE}}
> * {{SD_PARAMS.PARAM_VALUE}}
> After updating the maximum length the metastore database needs to be 
> configured and restarted with the new settings. Altering {{MAX_STRING_SIZE}} 
> will update database objects and possibly invalidate them, as follows:
> * Tables with virtual columns will be updated with new data type metadata for 
> virtual columns of {{VARCHAR2(4000)}}, 4000-byte {{NVARCHAR2}}, or 
> {{RAW(2000)}} type.
> * Functional indexes will become unusable if a change to their associated 
> virtual columns causes the index key to exceed index key length limits. 
> Attempts to rebuild such indexes will fail with {{ORA-01450: maximum key 
> length exceeded}}.
> * Views will be invalidated if they contain {{VARCHAR2(4000)}}, 4000-byte 
> {{NVARCHAR2}}, or {{RAW(2000)}} typed expression columns.
> * Materialized views will be updated with new metadata {{VARCHAR2(4000)}}, 
> 4000-byte {{NVARCHAR2}}, and {{RAW(2000)}} 

[jira] [Commented] (HIVE-15979) Support character_length and octet_length

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884749#comment-15884749
 ] 

Hive QA commented on HIVE-15979:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12854763/HIVE-15979.1.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10254 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] 
(batchId=223)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
 (batchId=101)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3795/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3795/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3795/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12854763 - PreCommit-HIVE-Build

> Support character_length and octet_length
> -
>
> Key: HIVE-15979
> URL: https://issues.apache.org/jira/browse/HIVE-15979
> Project: Hive
>  Issue Type: Sub-task
>  Components: SQL
>Reporter: Carter Shanklin
>Assignee: Teddy Choi
> Attachments: HIVE-15979.1.patch
>
>
> SQL defines standard ways to get number of characters and octets. SQL 
> reference: section 6.28. Example:
> vagrant=# select character_length('欲速则不达');
>  character_length
> --
> 5
> (1 row)
> vagrant=# select octet_length('欲速则不达');
>  octet_length
> --
>15
> (1 row)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-15977) Support standard ARRAY_AGG

[Teddy Choi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Teddy Choi reassigned HIVE-15977:
-

Assignee: Teddy Choi

> Support standard ARRAY_AGG
> --
>
> Key: HIVE-15977
> URL: https://issues.apache.org/jira/browse/HIVE-15977
> Project: Hive
>  Issue Type: Sub-task
>  Components: SQL
>Reporter: Carter Shanklin
>Assignee: Teddy Choi
>
> SQL defines array_agg as a function that returns an array given a value 
> expression in Section 10.9. Hive has something similar called collect_list. 
> The SQL standard array_agg additionally allows sorting by a value expression. 
> It would be good to move to the standard and as a first step it might be 
> worthwhile to have array_agg as an alias and then to add support for sort 
> specifications.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-12274) Increase width of columns used for general configuration in the metastore.

[Naveen Gangam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-12274:
-
Status: Open  (was: Patch Available)

The precommit build failed with some ptest error. It did not run the tests at 
all. Will re-submit the patch to kick off the build

> Increase width of columns used for general configuration in the metastore.
> --
>
> Key: HIVE-12274
> URL: https://issues.apache.org/jira/browse/HIVE-12274
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Elliot West
>Assignee: Naveen Gangam
>  Labels: metastore
> Attachments: HIVE-12274.example.ddl.hql, HIVE-12274.patch, 
> HIVE-12274.patch
>
>
> h2. Overview
> This issue is very similar in principle to HIVE-1364. We are hitting a limit 
> when processing JSON data that has a large nested schema. The struct 
> definition is truncated when inserted into the metastore database column 
> {{COLUMNS_V2.YPE_NAME}} as it is greater than 4000 characters in length.
> Given that the purpose of these columns is to hold very loosely defined 
> configuration values it seems rather limiting to impose such a relatively low 
> length bound. One can imagine that valid use cases will arise where 
> reasonable parameter/property values exceed the current limit. 
> h2. Context
> These limitations were in by the [patch 
> attributed|https://github.com/apache/hive/commit/c21a526b0a752df2a51d20a2729cc8493c228799]
>  to HIVE-1364 which mentions the _"max length on Oracle 9i/10g/11g"_ as the 
> reason. However, nowadays the limit can be increased because:
> * Oracle DB's {{varchar2}} supports 32767 bytes now, by setting the 
> configuration parameter {{MAX_STRING_SIZE}} to {{EXTENDED}}. 
> ([source|http://docs.oracle.com/database/121/SQLRF/sql_elements001.htm#SQLRF55623])
> * Postgres supports a max of 1GB for {{character}} datatype. 
> ([source|http://www.postgresql.org/docs/8.3/static/datatype-character.html])
> * MySQL can support upto 65535 bytes for the entire row. So long as the 
> {{PARAM_KEY}} value + {{PARAM_VALUE}} is less than 65535, we should be good. 
> ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * SQL Server's {{varchar}} max length is 8000 and can go beyond using 
> "varchar(max)" with the same limitation as MySQL being 65535 bytes for the 
> entire row. ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * Derby's {{varchar}} can be upto 32672 bytes. 
> ([source|https://db.apache.org/derby/docs/10.7/ref/rrefsqlj41207.html])
> h2. Proposal
> Can these columns not use CLOB-like types as for example as used by 
> {{TBLS.VIEW_EXPANDED_TEXT}}? It would seem that suitable type equivalents 
> exist for all targeted database platforms:
> * MySQL: {{mediumtext}}
> * Postgres: {{text}}
> * Oracle: {{CLOB}}
> * Derby: {{LONG VARCHAR}}
> I'd suggest that the candidates for type change are:
> * {{COLUMNS_V2.TYPE_NAME}}
> * {{TABLE_PARAMS.PARAM_VALUE}}
> * {{SERDE_PARAMS.PARAM_VALUE}}
> * {{SD_PARAMS.PARAM_VALUE}}
> After updating the maximum length the metastore database needs to be 
> configured and restarted with the new settings. Altering {{MAX_STRING_SIZE}} 
> will update database objects and possibly invalidate them, as follows:
> * Tables with virtual columns will be updated with new data type metadata for 
> virtual columns of {{VARCHAR2(4000)}}, 4000-byte {{NVARCHAR2}}, or 
> {{RAW(2000)}} type.
> * Functional indexes will become unusable if a change to their associated 
> virtual columns causes the index key to exceed index key length limits. 
> Attempts to rebuild such indexes will fail with {{ORA-01450: maximum key 
> length exceeded}}.
> * Views will be invalidated if they contain {{VARCHAR2(4000)}}, 4000-byte 
> {{NVARCHAR2}}, or {{RAW(2000)}} typed expression columns.
> * Materialized views will be updated with new metadata {{VARCHAR2(4000)}}, 
> 4000-byte {{NVARCHAR2}}, and {{RAW(2000)}} typed expression columns
> * So the limitation could be raised to 32672 bytes, with the caveat that 
> MySQL and SQL Server limit the row length to 65535 bytes, so that should also 
> be validated to provide consistency.
> Finally, will this limitation persist in the work resulting from HIVE-9452?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-12274) Increase width of columns used for general configuration in the metastore.

[Naveen Gangam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-12274:
-
Status: Patch Available  (was: Open)

> Increase width of columns used for general configuration in the metastore.
> --
>
> Key: HIVE-12274
> URL: https://issues.apache.org/jira/browse/HIVE-12274
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Elliot West
>Assignee: Naveen Gangam
>  Labels: metastore
> Attachments: HIVE-12274.example.ddl.hql, HIVE-12274.patch, 
> HIVE-12274.patch
>
>
> h2. Overview
> This issue is very similar in principle to HIVE-1364. We are hitting a limit 
> when processing JSON data that has a large nested schema. The struct 
> definition is truncated when inserted into the metastore database column 
> {{COLUMNS_V2.YPE_NAME}} as it is greater than 4000 characters in length.
> Given that the purpose of these columns is to hold very loosely defined 
> configuration values it seems rather limiting to impose such a relatively low 
> length bound. One can imagine that valid use cases will arise where 
> reasonable parameter/property values exceed the current limit. 
> h2. Context
> These limitations were in by the [patch 
> attributed|https://github.com/apache/hive/commit/c21a526b0a752df2a51d20a2729cc8493c228799]
>  to HIVE-1364 which mentions the _"max length on Oracle 9i/10g/11g"_ as the 
> reason. However, nowadays the limit can be increased because:
> * Oracle DB's {{varchar2}} supports 32767 bytes now, by setting the 
> configuration parameter {{MAX_STRING_SIZE}} to {{EXTENDED}}. 
> ([source|http://docs.oracle.com/database/121/SQLRF/sql_elements001.htm#SQLRF55623])
> * Postgres supports a max of 1GB for {{character}} datatype. 
> ([source|http://www.postgresql.org/docs/8.3/static/datatype-character.html])
> * MySQL can support upto 65535 bytes for the entire row. So long as the 
> {{PARAM_KEY}} value + {{PARAM_VALUE}} is less than 65535, we should be good. 
> ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * SQL Server's {{varchar}} max length is 8000 and can go beyond using 
> "varchar(max)" with the same limitation as MySQL being 65535 bytes for the 
> entire row. ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * Derby's {{varchar}} can be upto 32672 bytes. 
> ([source|https://db.apache.org/derby/docs/10.7/ref/rrefsqlj41207.html])
> h2. Proposal
> Can these columns not use CLOB-like types as for example as used by 
> {{TBLS.VIEW_EXPANDED_TEXT}}? It would seem that suitable type equivalents 
> exist for all targeted database platforms:
> * MySQL: {{mediumtext}}
> * Postgres: {{text}}
> * Oracle: {{CLOB}}
> * Derby: {{LONG VARCHAR}}
> I'd suggest that the candidates for type change are:
> * {{COLUMNS_V2.TYPE_NAME}}
> * {{TABLE_PARAMS.PARAM_VALUE}}
> * {{SERDE_PARAMS.PARAM_VALUE}}
> * {{SD_PARAMS.PARAM_VALUE}}
> After updating the maximum length the metastore database needs to be 
> configured and restarted with the new settings. Altering {{MAX_STRING_SIZE}} 
> will update database objects and possibly invalidate them, as follows:
> * Tables with virtual columns will be updated with new data type metadata for 
> virtual columns of {{VARCHAR2(4000)}}, 4000-byte {{NVARCHAR2}}, or 
> {{RAW(2000)}} type.
> * Functional indexes will become unusable if a change to their associated 
> virtual columns causes the index key to exceed index key length limits. 
> Attempts to rebuild such indexes will fail with {{ORA-01450: maximum key 
> length exceeded}}.
> * Views will be invalidated if they contain {{VARCHAR2(4000)}}, 4000-byte 
> {{NVARCHAR2}}, or {{RAW(2000)}} typed expression columns.
> * Materialized views will be updated with new metadata {{VARCHAR2(4000)}}, 
> 4000-byte {{NVARCHAR2}}, and {{RAW(2000)}} typed expression columns
> * So the limitation could be raised to 32672 bytes, with the caveat that 
> MySQL and SQL Server limit the row length to 65535 bytes, so that should also 
> be validated to provide consistency.
> Finally, will this limitation persist in the work resulting from HIVE-9452?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-12274) Increase width of columns used for general configuration in the metastore.

[Naveen Gangam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-12274:
-
Attachment: HIVE-12274.patch

> Increase width of columns used for general configuration in the metastore.
> --
>
> Key: HIVE-12274
> URL: https://issues.apache.org/jira/browse/HIVE-12274
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Elliot West
>Assignee: Naveen Gangam
>  Labels: metastore
> Attachments: HIVE-12274.example.ddl.hql, HIVE-12274.patch, 
> HIVE-12274.patch
>
>
> h2. Overview
> This issue is very similar in principle to HIVE-1364. We are hitting a limit 
> when processing JSON data that has a large nested schema. The struct 
> definition is truncated when inserted into the metastore database column 
> {{COLUMNS_V2.YPE_NAME}} as it is greater than 4000 characters in length.
> Given that the purpose of these columns is to hold very loosely defined 
> configuration values it seems rather limiting to impose such a relatively low 
> length bound. One can imagine that valid use cases will arise where 
> reasonable parameter/property values exceed the current limit. 
> h2. Context
> These limitations were in by the [patch 
> attributed|https://github.com/apache/hive/commit/c21a526b0a752df2a51d20a2729cc8493c228799]
>  to HIVE-1364 which mentions the _"max length on Oracle 9i/10g/11g"_ as the 
> reason. However, nowadays the limit can be increased because:
> * Oracle DB's {{varchar2}} supports 32767 bytes now, by setting the 
> configuration parameter {{MAX_STRING_SIZE}} to {{EXTENDED}}. 
> ([source|http://docs.oracle.com/database/121/SQLRF/sql_elements001.htm#SQLRF55623])
> * Postgres supports a max of 1GB for {{character}} datatype. 
> ([source|http://www.postgresql.org/docs/8.3/static/datatype-character.html])
> * MySQL can support upto 65535 bytes for the entire row. So long as the 
> {{PARAM_KEY}} value + {{PARAM_VALUE}} is less than 65535, we should be good. 
> ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * SQL Server's {{varchar}} max length is 8000 and can go beyond using 
> "varchar(max)" with the same limitation as MySQL being 65535 bytes for the 
> entire row. ([source|http://dev.mysql.com/doc/refman/5.0/en/char.html])
> * Derby's {{varchar}} can be upto 32672 bytes. 
> ([source|https://db.apache.org/derby/docs/10.7/ref/rrefsqlj41207.html])
> h2. Proposal
> Can these columns not use CLOB-like types as for example as used by 
> {{TBLS.VIEW_EXPANDED_TEXT}}? It would seem that suitable type equivalents 
> exist for all targeted database platforms:
> * MySQL: {{mediumtext}}
> * Postgres: {{text}}
> * Oracle: {{CLOB}}
> * Derby: {{LONG VARCHAR}}
> I'd suggest that the candidates for type change are:
> * {{COLUMNS_V2.TYPE_NAME}}
> * {{TABLE_PARAMS.PARAM_VALUE}}
> * {{SERDE_PARAMS.PARAM_VALUE}}
> * {{SD_PARAMS.PARAM_VALUE}}
> After updating the maximum length the metastore database needs to be 
> configured and restarted with the new settings. Altering {{MAX_STRING_SIZE}} 
> will update database objects and possibly invalidate them, as follows:
> * Tables with virtual columns will be updated with new data type metadata for 
> virtual columns of {{VARCHAR2(4000)}}, 4000-byte {{NVARCHAR2}}, or 
> {{RAW(2000)}} type.
> * Functional indexes will become unusable if a change to their associated 
> virtual columns causes the index key to exceed index key length limits. 
> Attempts to rebuild such indexes will fail with {{ORA-01450: maximum key 
> length exceeded}}.
> * Views will be invalidated if they contain {{VARCHAR2(4000)}}, 4000-byte 
> {{NVARCHAR2}}, or {{RAW(2000)}} typed expression columns.
> * Materialized views will be updated with new metadata {{VARCHAR2(4000)}}, 
> 4000-byte {{NVARCHAR2}}, and {{RAW(2000)}} typed expression columns
> * So the limitation could be raised to 32672 bytes, with the caveat that 
> MySQL and SQL Server limit the row length to 65535 bytes, so that should also 
> be validated to provide consistency.
> Finally, will this limitation persist in the work resulting from HIVE-9452?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15979) Support character_length and octet_length

[Teddy Choi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Teddy Choi updated HIVE-15979:
--
Status: Patch Available  (was: Open)

> Support character_length and octet_length
> -
>
> Key: HIVE-15979
> URL: https://issues.apache.org/jira/browse/HIVE-15979
> Project: Hive
>  Issue Type: Sub-task
>  Components: SQL
>Reporter: Carter Shanklin
>Assignee: Teddy Choi
> Attachments: HIVE-15979.1.patch
>
>
> SQL defines standard ways to get number of characters and octets. SQL 
> reference: section 6.28. Example:
> vagrant=# select character_length('欲速则不达');
>  character_length
> --
> 5
> (1 row)
> vagrant=# select octet_length('欲速则不达');
>  octet_length
> --
>15
> (1 row)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15708) Upgrade calcite version to 1.12

[Remus Rusanu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884729#comment-15884729
 ] 

Remus Rusanu commented on HIVE-15708:
-

Looks like a bunch of the golden file changes are incorrect because of 
CALCITE-1659 truncation of sub-second fractional part in TIMESTAMP. Will need 
to revert them and do a new run after I get a new Calcite 1.12-SNAPSHOT drop 
with the issue fixed.

> Upgrade calcite version to 1.12
> ---
>
> Key: HIVE-15708
> URL: https://issues.apache.org/jira/browse/HIVE-15708
> Project: Hive
>  Issue Type: Task
>  Components: CBO, Logical Optimizer
>Affects Versions: 2.2.0
>Reporter: Ashutosh Chauhan
>Assignee: Remus Rusanu
> Attachments: HIVE-15708.01.patch, HIVE-15708.02.patch, 
> HIVE-15708.03.patch, HIVE-15708.04.patch, HIVE-15708.05.patch, 
> HIVE-15708.06.patch, HIVE-15708.07.patch, HIVE-15708.08.patch, 
> HIVE-15708.09.patch, HIVE-15708.10.patch, HIVE-15708.11.patch, 
> HIVE-15708.12.patch
>
>
> Currently we are on 1.10 Need to upgrade calcite version to 1.11



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-15979) Support character_length and octet_length

[Teddy Choi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Teddy Choi reassigned HIVE-15979:
-

Assignee: Teddy Choi

> Support character_length and octet_length
> -
>
> Key: HIVE-15979
> URL: https://issues.apache.org/jira/browse/HIVE-15979
> Project: Hive
>  Issue Type: Sub-task
>  Components: SQL
>Reporter: Carter Shanklin
>Assignee: Teddy Choi
>
> SQL defines standard ways to get number of characters and octets. SQL 
> reference: section 6.28. Example:
> vagrant=# select character_length('欲速则不达');
>  character_length
> --
> 5
> (1 row)
> vagrant=# select octet_length('欲速则不达');
>  octet_length
> --
>15
> (1 row)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15975) Support the MOD function

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884708#comment-15884708
 ] 

Hive QA commented on HIVE-15975:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12854761/HIVE-15975.4.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3794/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3794/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3794/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Tests exited with: ExecutionException: java.util.concurrent.ExecutionException: 
java.io.IOException: Could not create 
/data/hiveptest/logs/PreCommit-HIVE-Build-3794/succeeded/204_TestAcidOnTez
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12854761 - PreCommit-HIVE-Build

> Support the MOD function
> 
>
> Key: HIVE-15975
> URL: https://issues.apache.org/jira/browse/HIVE-15975
> Project: Hive
>  Issue Type: Sub-task
>  Components: SQL
>Reporter: Carter Shanklin
>Assignee: Teddy Choi
> Attachments: HIVE-15975.1.patch, HIVE-15975.2.patch, 
> HIVE-15975.3.patch, HIVE-15975.4.patch
>
>
> SQL defines the mod expression as a function allowing 2 numeric value 
> expressions. Hive allows the infix notation using %. It would be good for 
> Hive to support the standard approach as well. SQL standard reference T441



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15668) change REPL DUMP syntax to use "LIMIT" instead of "BATCH" keyword

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884694#comment-15884694
 ] 

Lefty Leverenz commented on HIVE-15668:
---

Doc note:  This needs to be documented in the wiki.

* [HiveReplicationv2Development -- REPL DUMP | 
https://cwiki.apache.org/confluence/display/Hive/HiveReplicationv2Development#HiveReplicationv2Development-REPLDUMP]

Added a TODOC2.2 label.

> change REPL DUMP syntax to use "LIMIT" instead of "BATCH" keyword
> -
>
> Key: HIVE-15668
> URL: https://issues.apache.org/jira/browse/HIVE-15668
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
>  Labels: TODOC2.2
> Fix For: 2.2.0
>
> Attachments: HIVE-15668.2.patch, HIVE-15668.patch
>
>
> Currently, REPL DUMP syntax goes:
> {noformat}
> REPL DUMP [[.]] [FROM  [BATCH ]]
> {noformat}
> The BATCH directive says that when doing an event dump, to not dump out more 
> than _batchSize_ number of events. However, there is a clearer keyword for 
> the same effect, and that is LIMIT. Thus, rephrasing the syntax as follows 
> makes it clearer:
> {noformat}
> REPL DUMP [[.]] [FROM  [LIMIT ]]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (HIVE-15975) Support the MOD function

[Teddy Choi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884688#comment-15884688
 ] 

Teddy Choi edited comment on HIVE-15975 at 2/26/17 9:58 AM:


- Fixed udf_length.q error
- Applied synonyms changes in udf_modulo.q.out
- Merged udf_mod.q into udf_modulo.q


was (Author: teddy.choi):
- Fixed udf_length.q error
- Applied synonyms changes in udf_modulo.q.out.
- Merged udf_mod.q into udf_modulo.q

> Support the MOD function
> 
>
> Key: HIVE-15975
> URL: https://issues.apache.org/jira/browse/HIVE-15975
> Project: Hive
>  Issue Type: Sub-task
>  Components: SQL
>Reporter: Carter Shanklin
>Assignee: Teddy Choi
> Attachments: HIVE-15975.1.patch, HIVE-15975.2.patch, 
> HIVE-15975.3.patch, HIVE-15975.4.patch
>
>
> SQL defines the mod expression as a function allowing 2 numeric value 
> expressions. Hive allows the infix notation using %. It would be good for 
> Hive to support the standard approach as well. SQL standard reference T441



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15975) Support the MOD function

[Teddy Choi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Teddy Choi updated HIVE-15975:
--
Attachment: HIVE-15975.4.patch

- Fixed udf_length.q error
- Applied synonyms changes in udf_modulo.q.out.

> Support the MOD function
> 
>
> Key: HIVE-15975
> URL: https://issues.apache.org/jira/browse/HIVE-15975
> Project: Hive
>  Issue Type: Sub-task
>  Components: SQL
>Reporter: Carter Shanklin
>Assignee: Teddy Choi
> Attachments: HIVE-15975.1.patch, HIVE-15975.2.patch, 
> HIVE-15975.3.patch, HIVE-15975.4.patch
>
>
> SQL defines the mod expression as a function allowing 2 numeric value 
> expressions. Hive allows the infix notation using %. It would be good for 
> Hive to support the standard approach as well. SQL standard reference T441



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (HIVE-15975) Support the MOD function

[Teddy Choi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884688#comment-15884688
 ] 

Teddy Choi edited comment on HIVE-15975 at 2/26/17 9:58 AM:


- Fixed udf_length.q error
- Applied synonyms changes in udf_modulo.q.out.
- Merged udf_mod.q into udf_modulo.q


was (Author: teddy.choi):
- Fixed udf_length.q error
- Applied synonyms changes in udf_modulo.q.out.

> Support the MOD function
> 
>
> Key: HIVE-15975
> URL: https://issues.apache.org/jira/browse/HIVE-15975
> Project: Hive
>  Issue Type: Sub-task
>  Components: SQL
>Reporter: Carter Shanklin
>Assignee: Teddy Choi
> Attachments: HIVE-15975.1.patch, HIVE-15975.2.patch, 
> HIVE-15975.3.patch, HIVE-15975.4.patch
>
>
> SQL defines the mod expression as a function allowing 2 numeric value 
> expressions. Hive allows the infix notation using %. It would be good for 
> Hive to support the standard approach as well. SQL standard reference T441



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15975) Support the MOD function

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884680#comment-15884680
 ] 

Hive QA commented on HIVE-15975:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12854697/HIVE-15975.3.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10267 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[show_functions] 
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_length] (batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_modulo] (batchId=68)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=140)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=223)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=211)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3793/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3793/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3793/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12854697 - PreCommit-HIVE-Build

> Support the MOD function
> 
>
> Key: HIVE-15975
> URL: https://issues.apache.org/jira/browse/HIVE-15975
> Project: Hive
>  Issue Type: Sub-task
>  Components: SQL
>Reporter: Carter Shanklin
>Assignee: Teddy Choi
> Attachments: HIVE-15975.1.patch, HIVE-15975.2.patch, 
> HIVE-15975.3.patch
>
>
> SQL defines the mod expression as a function allowing 2 numeric value 
> expressions. Hive allows the infix notation using %. It would be good for 
> Hive to support the standard approach as well. SQL standard reference T441



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15975) Support the MOD function

[Teddy Choi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Teddy Choi updated HIVE-15975:
--
Attachment: HIVE-15975.3.patch

Renamed mod.q to udf_mod.q

> Support the MOD function
> 
>
> Key: HIVE-15975
> URL: https://issues.apache.org/jira/browse/HIVE-15975
> Project: Hive
>  Issue Type: Sub-task
>  Components: SQL
>Reporter: Carter Shanklin
>Assignee: Teddy Choi
> Attachments: HIVE-15975.1.patch, HIVE-15975.2.patch, 
> HIVE-15975.3.patch
>
>
> SQL defines the mod expression as a function allowing 2 numeric value 
> expressions. Hive allows the infix notation using %. It would be good for 
> Hive to support the standard approach as well. SQL standard reference T441



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)