[jira] [Created] (HIVE-11978) LLAP: NPE in Expr toString

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-11978:
---

 Summary: LLAP: NPE in Expr toString
 Key: HIVE-11978
 URL: https://issues.apache.org/jira/browse/HIVE-11978
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Derby version used by Hive

[Richard Hillegas]

Thanks! Some responses inline...

"kulkarni.swar...@gmail.com"  wrote on
09/28/2015 10:08:08 AM:

> From: "kulkarni.swar...@gmail.com" 
> To: "dev@hive.apache.org" 
> Date: 09/28/2015 10:08 AM
> Subject: Re: Derby version used by Hive
>
> Richard,
>
> A quick eye-balling of the code doesn't show anything that could
> potentially be a blocker for this upgrade. Also +1 on staying on the
latest
> and greatest. Please feel free to open up a JIRA and submit the patch.

Great! I'll try my hand at this after Derby 10.12.1.1 is published to the
maven repositories next week.

>
> Also just out of curiosity, what are you really using a derby backed
store
> for?

Right now just for testing. Its standards compliance makes it a good
starting point for implementing a portable SQL layer.

Thanks,
-Rick

>
> On Mon, Sep 28, 2015 at 11:02 AM, Richard Hillegas 
> wrote:
>
> >
> >
> > I haven't received a response to the following message, which I posted
last
> > week. Maybe my message rambled too much. Here is an attempt to pose my
> > question more succinctly:
> >
> > Q: Does anyone know of any reason why we can't upgrade Hive's Derby
version
> > to 10.12.1.1, the new version being vetted by the Derby community right
> > now?
> >
> > Thanks,
> > -Rick
> >
> > > I am following the Hive build instructions here:
> > >
> >
> > https://cwiki.apache.org/confluence/display/Hive/
> GettingStarted#GettingStarted-InstallationandConfiguration
> > > .
> > >
> > > I noticed that Hive development seems to be using an old version of
> > Derby:
> > > 10.10.2.0. Is there some defect in the most recent Derby version
> > > (10.11.1.1) which prevents Hive from upgrading to 10.11.1.1? The only
> > > Hive-tagged Derby bug which I can find is
> > > https://issues.apache.org/jira/browse/DERBY-6358. That issue doesn't
> > seem
> > > to be version-specific and it mentions a resolved Hive issue:
> > > https://issues.apache.org/jira/browse/HIVE-8739.
> > >
> > > Staying with 10.10.2.0 makes sense if you need to run on some ancient
> > JVMs:
> > > Java SE 5 or Java ME CDC/Foundation Profile 1.1. Hadoop, however,
> > requires
> > > at least Java 6 according to
> > > https://wiki.apache.org/hadoop/HadoopJavaVersions.
> > >
> > > Note that the Derby community expects to release version 10.12.1.1
soon:
> > > https://wiki.apache.org/db-derby/DerbyTenTwelveOneRelease. This might
be
> > a
> > > good opportunity for Hive to upgrade to a more capable version of
Derby.
> > >
> > > I mention this because the Derby version used by Hive ends up on the
> > > classpath used by downstream projects (like Spark). That makes it
awkward
> > > for downstream projects to use more current Derby versions. Do you
know
> > of
> > > any reason that downstream projects shouldn't override the Derby
version
> > > currently preferred by Hive?
> > >
> > > Thanks,
> > > -Rick
> >
>
>
>
> --
> Swarnim

Re: Derby version used by Hive

[kulkarni.swar...@gmail.com]

Richard,

A quick eye-balling of the code doesn't show anything that could
potentially be a blocker for this upgrade. Also +1 on staying on the latest
and greatest. Please feel free to open up a JIRA and submit the patch.

Also just out of curiosity, what are you really using a derby backed store
for?

On Mon, Sep 28, 2015 at 11:02 AM, Richard Hillegas 
wrote:

>
>
> I haven't received a response to the following message, which I posted last
> week. Maybe my message rambled too much. Here is an attempt to pose my
> question more succinctly:
>
> Q: Does anyone know of any reason why we can't upgrade Hive's Derby version
> to 10.12.1.1, the new version being vetted by the Derby community right
> now?
>
> Thanks,
> -Rick
>
> > I am following the Hive build instructions here:
> >
>
> https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-InstallationandConfiguration
> > .
> >
> > I noticed that Hive development seems to be using an old version of
> Derby:
> > 10.10.2.0. Is there some defect in the most recent Derby version
> > (10.11.1.1) which prevents Hive from upgrading to 10.11.1.1? The only
> > Hive-tagged Derby bug which I can find is
> > https://issues.apache.org/jira/browse/DERBY-6358. That issue doesn't
> seem
> > to be version-specific and it mentions a resolved Hive issue:
> > https://issues.apache.org/jira/browse/HIVE-8739.
> >
> > Staying with 10.10.2.0 makes sense if you need to run on some ancient
> JVMs:
> > Java SE 5 or Java ME CDC/Foundation Profile 1.1. Hadoop, however,
> requires
> > at least Java 6 according to
> > https://wiki.apache.org/hadoop/HadoopJavaVersions.
> >
> > Note that the Derby community expects to release version 10.12.1.1 soon:
> > https://wiki.apache.org/db-derby/DerbyTenTwelveOneRelease. This might be
> a
> > good opportunity for Hive to upgrade to a more capable version of Derby.
> >
> > I mention this because the Derby version used by Hive ends up on the
> > classpath used by downstream projects (like Spark). That makes it awkward
> > for downstream projects to use more current Derby versions. Do you know
> of
> > any reason that downstream projects shouldn't override the Derby version
> > currently preferred by Hive?
> >
> > Thanks,
> > -Rick
>



-- 
Swarnim

[jira] [Created] (HIVE-11975) mssql scripts contains invalid 'GO' statement

[Huan Huang (JIRA)]

Huan Huang created HIVE-11975:
-

 Summary: mssql scripts contains invalid 'GO' statement
 Key: HIVE-11975
 URL: https://issues.apache.org/jira/browse/HIVE-11975
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 0.12.0, 0.13.0, 0.14.0, 1.2.0, 1.1.0
Reporter: Huan Huang
Assignee: Huan Huang


upgrade scripts dont execute as a result



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11976) Extend CBO rules to being able to apply rules only once on a given operator

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-11976:
--

 Summary: Extend CBO rules to being able to apply rules only once 
on a given operator
 Key: HIVE-11976
 URL: https://issues.apache.org/jira/browse/HIVE-11976
 Project: Hive
  Issue Type: New Feature
  Components: CBO
Affects Versions: 2.0.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


Create a way to bail out quickly from HepPlanner if the rule has been already 
applied on a certain operator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11980) Follow up on HIVE-11696, exception is thrown from CTAS from the table with table-level serde is Parquet while partition-level serde is JSON

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-11980:
---

 Summary: Follow up on HIVE-11696, exception is thrown from CTAS 
from the table with table-level serde is Parquet while partition-level serde is 
JSON
 Key: HIVE-11980
 URL: https://issues.apache.org/jira/browse/HIVE-11980
 Project: Hive
  Issue Type: Bug
  Components: Serializers/Deserializers
Affects Versions: 2.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


Apparently, getStructFieldsDataAsList() also needs to handle the case of List 
in addition to ArrayWritable similar to getStructFieldData.

{noformat}
Caused by: java.lang.UnsupportedOperationException: Cannot inspect 
java.util.ArrayList
at 
org.apache.hadoop.hive.ql.io.parquet.serde.ArrayWritableObjectInspector.getStructFieldsDataAsList(ArrayWritableObjectInspector.java:172)
at 
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:354)
at 
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:257)
at 
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.doSerialize(LazySimpleSerDe.java:241)
at 
org.apache.hadoop.hive.serde2.AbstractEncodingAwareSerDe.serialize(AbstractEncodingAwareSerDe.java:55)
at 
org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:720)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:813)
at 
org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:813)
at 
org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
at 
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:162)
at 
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:508)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11977) Hive should handle an external avro table with zero length files present

[Aaron Dossett (JIRA)]

Aaron Dossett created HIVE-11977:


 Summary: Hive should handle an external avro table with zero 
length files present
 Key: HIVE-11977
 URL: https://issues.apache.org/jira/browse/HIVE-11977
 Project: Hive
  Issue Type: Bug
Reporter: Aaron Dossett


If a zero length file is in the top level directory housing an external avro 
table,  all hive queries on the table fail.

This issue is that org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader 
creates a new org.apache.avro.file.DataFileReader and DataFileReader throws an 
exception when trying to read an empty file (because the empty file lacks the 
magic number marking it as avro).  

AvroGenericRecordReader should detect an empty file and then behave reasonably.

Caused by: java.io.IOException: Not a data file.
at org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:102)
at org.apache.avro.file.DataFileReader.(DataFileReader.java:97)
at 
org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader.(AvroGenericRecordReader.java:81)
at 
org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat.getRecordReader(AvroContainerInputFormat.java:51)
at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:246)
... 25 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11983) Hive streaming API's uses incorrect logic to assign buckets to incoming records

[Roshan Naik (JIRA)]

Roshan Naik created HIVE-11983:
--

 Summary: Hive streaming API's uses incorrect logic to assign 
buckets to incoming records
 Key: HIVE-11983
 URL: https://issues.apache.org/jira/browse/HIVE-11983
 Project: Hive
  Issue Type: Bug
  Components: HCatalog
Affects Versions: 1.2.1
Reporter: Roshan Naik
Assignee: Roshan Naik


The Streaming API tries to distribute records evenly into buckets. 
All records in every Transaction that is part of TransactionBatch goes to the 
same bucket and a new bucket number is chose for each TransactionBatch.

Fix: API needs to hash each record to determine which bucket it belongs to. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11982) Some test case for union all with recent changes

[Yongzhi Chen (JIRA)]

Yongzhi Chen created HIVE-11982:
---

 Summary: Some test case for union all with recent changes
 Key: HIVE-11982
 URL: https://issues.apache.org/jira/browse/HIVE-11982
 Project: Hive
  Issue Type: Bug
Reporter: Yongzhi Chen
Assignee: Yongzhi Chen


The tests throw java.lang.IndexOutOfBoundsException again. 
It was supposed to be fixed by HIVE-11271



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11974) Table name created using long name does not display full name in show tables command

[Chetan Bhat (JIRA)]

Chetan Bhat created HIVE-11974:
--

 Summary: Table name created using long name does not display full 
name in show tables command
 Key: HIVE-11974
 URL: https://issues.apache.org/jira/browse/HIVE-11974
 Project: Hive
  Issue Type: Bug
  Components: Beeline
Affects Versions: 0.13.0
 Environment: SUSE 11
Reporter: Chetan Bhat
Priority: Minor


User creates a Hive table with long name having 180 characters.
The table is created.
User executes the show tables command.

Actual Output
The table name does not display the complete name with 180 characters in 
Beeline show tables output.

Expected Output :
The table name should display the complete table name with 180 characters in 
Beeline show tables output.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

re: Derby version used by Hive

[Richard Hillegas]

I haven't received a response to the following message, which I posted last
week. Maybe my message rambled too much. Here is an attempt to pose my
question more succinctly:

Q: Does anyone know of any reason why we can't upgrade Hive's Derby version
to 10.12.1.1, the new version being vetted by the Derby community right
now?

Thanks,
-Rick

> I am following the Hive build instructions here:
>
https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-InstallationandConfiguration
> .
>
> I noticed that Hive development seems to be using an old version of
Derby:
> 10.10.2.0. Is there some defect in the most recent Derby version
> (10.11.1.1) which prevents Hive from upgrading to 10.11.1.1? The only
> Hive-tagged Derby bug which I can find is
> https://issues.apache.org/jira/browse/DERBY-6358. That issue doesn't seem
> to be version-specific and it mentions a resolved Hive issue:
> https://issues.apache.org/jira/browse/HIVE-8739.
>
> Staying with 10.10.2.0 makes sense if you need to run on some ancient
JVMs:
> Java SE 5 or Java ME CDC/Foundation Profile 1.1. Hadoop, however,
requires
> at least Java 6 according to
> https://wiki.apache.org/hadoop/HadoopJavaVersions.
>
> Note that the Derby community expects to release version 10.12.1.1 soon:
> https://wiki.apache.org/db-derby/DerbyTenTwelveOneRelease. This might be
a
> good opportunity for Hive to upgrade to a more capable version of Derby.
>
> I mention this because the Derby version used by Hive ends up on the
> classpath used by downstream projects (like Spark). That makes it awkward
> for downstream projects to use more current Derby versions. Do you know
of
> any reason that downstream projects shouldn't override the Derby version
> currently preferred by Hive?
>
> Thanks,
> -Rick

[jira] [Created] (HIVE-11973) IN operator fails when the column type is DATE

[sanjiv singh (JIRA)]

sanjiv singh created HIVE-11973:
---

 Summary: IN operator fails when the column type is DATE 
 Key: HIVE-11973
 URL: https://issues.apache.org/jira/browse/HIVE-11973
 Project: Hive
  Issue Type: Bug
  Components: Parser, Query Processor
Affects Versions: 1.0.0
Reporter: sanjiv singh


Test DLL :

CREATE TABLE `date_dim`(
  `d_date_sk` int, 
  `d_date_id` string, 
  `d_date` date, 
  `d_current_week` string, 
  `d_current_month` string, 
  `d_current_quarter` string, 
  `d_current_year` string) ;

Hive query :

SELECT *  
FROM   date_dim 
WHERE d_date  IN ('2000-03-22','2001-03-22')  ;

In 1.0.0 ,  the above query fails with:

FAILED: SemanticException [Error 10014]: Line 1:180 Wrong arguments 
''2001-03-22'': The arguments for IN should be the same type! Types are: {date 
IN (string, string)}

I saw below bug  which is related to IN operator for partition column. 
 
HIVE-8099 : IN operator for partition column fails when the partition column 
type is DATE.

Raising bug for the same as same error but for un-partitioned column.






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 38211: HIVE-11684

[Jesús Camacho Rodríguez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/
---

(Updated Sept. 28, 2015, 11:40 a.m.)


Review request for hive and John Pullokkaran.


Bugs: HIVE-11684
https://issues.apache.org/jira/browse/HIVE-11684


Repository: hive-git


Description
---

Implement limit pushdown through outer join in CBO


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
dffdb5c14114d9c2d1e4c44a36f6b60966dcfc86 
  itests/src/test/resources/testconfiguration.properties 
4f7b25f2563683933b9d677626f0447705927966 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
9c731b8c5e58570570812b525b8bf0edb0f4c0a9 
  ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION 
  ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION 
  ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out 
PRE-CREATION 

Diff: https://reviews.apache.org/r/38211/diff/


Testing
---

Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only 
Tez)


Thanks,

Jesús Camacho Rodríguez

Re: Review Request 38211: HIVE-11684

[Jesús Camacho Rodríguez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/
---

(Updated Sept. 28, 2015, 9:27 a.m.)


Review request for hive and John Pullokkaran.


Bugs: HIVE-11684
https://issues.apache.org/jira/browse/HIVE-11684


Repository: hive-git


Description
---

Implement limit pushdown through outer join in CBO


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
dffdb5c14114d9c2d1e4c44a36f6b60966dcfc86 
  itests/src/test/resources/testconfiguration.properties 
4f7b25f2563683933b9d677626f0447705927966 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
9c731b8c5e58570570812b525b8bf0edb0f4c0a9 
  ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION 
  ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION 
  ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out 
PRE-CREATION 

Diff: https://reviews.apache.org/r/38211/diff/


Testing
---

Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only 
Tez)


Thanks,

Jesús Camacho Rodríguez

Re: Review Request 38211: HIVE-11684

[Jesús Camacho Rodríguez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/
---

(Updated Sept. 28, 2015, 9:04 a.m.)


Review request for hive and John Pullokkaran.


Bugs: HIVE-11684
https://issues.apache.org/jira/browse/HIVE-11684


Repository: hive-git


Description
---

Implement limit pushdown through outer join in CBO


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
dffdb5c14114d9c2d1e4c44a36f6b60966dcfc86 
  itests/src/test/resources/testconfiguration.properties 
4f7b25f2563683933b9d677626f0447705927966 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
9c731b8c5e58570570812b525b8bf0edb0f4c0a9 
  ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION 
  ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION 
  ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out 
PRE-CREATION 

Diff: https://reviews.apache.org/r/38211/diff/


Testing
---

Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only 
Tez)


Thanks,

Jesús Camacho Rodríguez

Build failed in Jenkins: HIVE-TRUNK-JAVA8 #117

[hiveqa]

See 


Changes:

[Xuefu Zhang] HIVE-10624: Update the initial script to make beeline bucked cli 
as default and allow user choose old hive cli by env (Ferdinand via Xuefu)

[Xuefu Zhang] HIVE-10623: Implement hive cli options using beeline 
functionality (Ferdinand via Xuefu)

[Ferdinand Xu] HIVE-10822: CLI start script throwing error message on console 
(Chinna Rao Lalam, reviewed by Ferdinand Xu)

[Ferdinand Xu] HIVE-10821 Beeline-CLI: Implement CLI source command using 
Beeline functionality(Ferdinand Xu, Reviewed By Zhang Xuefu and Chinna Rao 
Lalam)

[Ferdinand Xu] HIVE-10904 Use beeline-log4j.properties for migrated CLI 
[beeline-cli Branch] (Chinna Rao Lalam, reviewed by Ferdinand Xu)

[Ferdinand Xu] HIVE-10905 Quit fails ending with ';' [beeline-cli 
Branch](Chinna Rao Lalam, reviewed by Ferdinand Xu)

[Ferdinand Xu] HIVE-6791: Support variable substition for Beeline shell command 
(Ferdinand Xu, reviewed by Xuefu Zhang)

[Ferdinand Xu] HIVE-11203: Beeline force option doesn't force execution when 
errors occurred in a script.(Ferdinand, reviewed by Xuefu Zhang)

[Ferdinand Xu] HIVE-11191: Beeline-cli: support hive.cli.errors.ignore in new 
CLI(Ferdinand Xu, reviewed by Xuefu Zhang)

[Ferdinand Xu] HIVE-11236 BeeLine-Cli: use the same output format as old CLI in 
the new CLI(Ferdinand Xu, reviewed by Xuefu Zhang)

[Ferdinand Xu] HIVE-11226 BeeLine-Cli: support hive.cli.prompt in new CLI 
(Ferdinand Xu, reviewed by Xuefu Zhang)

[Ferdinand Xu] HIVE-11280: Support executing script file from hdfs in new CLI 
[Beeline-CLI branch](Ferdinand Xu, reviewed by Xuefu Zhang)

[Ferdinand Xu] HIVE-11352: Avoid the double connections with 'e' 
option[beeline-cli branch](Ferdinand Xu via Xuefu Zhang)

[Ferdinand Xu] HIVE-11336: Support initial file option for new CLI [beeline-cli 
branch](Ferdinand Xu, reviewed by Xuefu Zhang)

[Ferdinand Xu] HIVE-11579: Invoke the set command will close standard error 
output[beeline-cli](Ferdinand Xu, reviewed by Xuefu Zhang)

[Ferdinand Xu] HIVE-11624:Beeline-cli: support hive.cli.print.header in new 
CLI[beeline-cli branch](Ke Jia, reviewed by Ferdinand Xu)

[Ferdinand Xu] HIVE-11637: Support hive.cli.print.current.db in new 
CLI[beeline-cli branch](Ferdinand Xu, reviewed by Dong Chen)

[Ferdinand Xu] HIVE-11640: Shell command doesn't work for new CLI[Beeline-cli 
branch](Ferdinand Xu, reviewed by Xuefu Zhang)

[Ferdinand Xu] HIVE-11746: Connect command should not to be allowed from 
user[beeline-cli branch] (Ferdinand Xu, reviewed by Xuefu Zhang)

[Ferdinand Xu] HIVE-11770: Use the static variable from beeline instead of 
untils from JDBC(Ferdinand Xu, reviewed by Xuefu Zhang)

[Ferdinand Xu] HIVE-11796: CLI option is not updated when executing the initial 
files[beeline-cli](Ferdinand Xu, reviewed by Xuefu Zhang)

[Ferdinand Xu] HIVE-11779 Beeline-cli: Format the output of describe pretty 
table command in new CLI[beeline-cli branch](Ke Jia, reviewed by Ferdinand Xu)

[Ferdinand Xu] HIVE-10824: Need to update start script changes in .cmd 
files(Ferdinand Xu, reviewed by Xuefu Zhang)

[Ferdinand Xu] HIVE-11717： nohup mode is not support for new hive cli（Ferdinand 
Xu, reviewed by Xuefu Zhang）

[Ferdinand Xu] HIVE-11944: Address the review items on HIVE-11778(Ferdinand Xu, 
reviewed by Xuefu Zhang)

[Ferdinand Xu] HIVE-11943: Set old CLI as the default Client when using hive 
script(Ferdinand Xu, reviewed by Xuefu Zhang)

[Ashutosh Chauhan] HIVE-11791 : Add unit test for HIVE-10122 (Illya Yalovyy via 
Ashutosh Chauhan, Gopal V)

[Jimmy Xiang] HIVE-11939: TxnDbUtil should turn off jdbc auto commit (Jimmy, 
reviewed by Alan)

[Xuefu Zhang] HIVE-11827: STORED AS AVRO fails SELECT COUNT(*) when empty 
(Yongzhi via Xuefu)

[thejas] HIVE-11613 : schematool should return non zero exit status for info 
command, if state is inconsistent (Thejas Nair, reviewed by Prasad Mujumdar)

[daijy] HIVE-11950: WebHCat status file doesn't show UTF8 character (Daniel Dai 
reviewed by Thejas Nair)

[Sergio Pena] HIVE-10048: JDBC - Support SSL encryption regardless of 
Authentication mechanism (Mubashir Kazia, reviewed by Sergio Pena)

--
[...truncated 326 lines...]
A
ptest2/src/main/java/org/apache/hive/ptest/execution/context/ServiceNotAvailableException.java
Aptest2/src/main/java/org/apache/hive/ptest/execution/conf
A
ptest2/src/main/java/org/apache/hive/ptest/execution/conf/ExecutionContextConfiguration.java
Aptest2/src/main/java/org/apache/hive/ptest/execution/conf/TestBatch.java
A
ptest2/src/main/java/org/apache/hive/ptest/execution/conf/UnitTestBatch.java
A
ptest2/src/main/java/org/apache/hive/ptest/execution/conf/QFileTestBatch.java
Aptest2/src/main/java/org/apache/hive/ptest/execution/conf/Context.java
A
ptest2/src/main/java/org/apache/hive/ptest/execution/conf/TestConfiguration.java
A

Splitting tests in PTest2

[Prasanth Jayachandran]

Hello everyone

Regarding HIVE-11642, the tests for MiniLlapCliDriver are not split across 
multiple nodes. I can see from the logs that MiniTezCliDriver are broken down 
into many tests and ran on different machines. 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
 


How can we do the same for MiniLlapCliDriver tests? Do we need to do something 
in PTest2 to enable splitting of tests?
MiniLlapCliDriver runs the same tests are MiniTez but in llap mode.

Thanks
Prasanth

[jira] [Created] (HIVE-11984) Add HS2 open operation metrics

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-11984:
--

 Summary: Add HS2 open operation metrics
 Key: HIVE-11984
 URL: https://issues.apache.org/jira/browse/HIVE-11984
 Project: Hive
  Issue Type: Improvement
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor


Some metrics for open operations should be helpful to track operations not 
closed/cancelled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Splitting tests in PTest2

[Szehon Ho]

Yes thanks for looking into it.  Can you edit testconfiguration.properties
to add a section for LlapCliDriver tests, see HIVE-7254 for details.  That
way the framework can identify which tests it needs to run and then split
it into parallel batches across test nodes.  Otherwise it will just run all
of the LlapCliDriver tests using -Dtest=TestMiniLlapCliDriver in one huge
batch.

Once you finish, let me know and I'll tweak the test properties to set a
batch size for that one.

Thanks
Szehon

On Mon, Sep 28, 2015 at 4:23 PM, Prasanth Jayachandran  wrote:

> Hello everyone
>
> Regarding HIVE-11642, the tests for MiniLlapCliDriver are not split across
> multiple nodes. I can see from the logs that MiniTezCliDriver are broken
> down into many tests and ran on different machines.
> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
> <
> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
> >
>
> How can we do the same for MiniLlapCliDriver tests? Do we need to do
> something in PTest2 to enable splitting of tests?
> MiniLlapCliDriver runs the same tests are MiniTez but in llap mode.
>
> Thanks
> Prasanth

Re: Splitting tests in PTest2

[Prasanth Jayachandran]

Thanks Szehon! I will look at it and update you once done.

> On Sep 28, 2015, at 6:37 PM, Szehon Ho  wrote:
> 
> Yes thanks for looking into it.  Can you edit testconfiguration.properties
> to add a section for LlapCliDriver tests, see HIVE-7254 for details.  That
> way the framework can identify which tests it needs to run and then split
> it into parallel batches across test nodes.  Otherwise it will just run all
> of the LlapCliDriver tests using -Dtest=TestMiniLlapCliDriver in one huge
> batch.
> 
> Once you finish, let me know and I'll tweak the test properties to set a
> batch size for that one.
> 
> Thanks
> Szehon
> 
> On Mon, Sep 28, 2015 at 4:23 PM, Prasanth Jayachandran > wrote:
> 
>> Hello everyone
>> 
>> Regarding HIVE-11642, the tests for MiniLlapCliDriver are not split across
>> multiple nodes. I can see from the logs that MiniTezCliDriver are broken
>> down into many tests and ran on different machines.
>> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
>> <
>> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
>>> 
>> 
>> How can we do the same for MiniLlapCliDriver tests? Do we need to do
>> something in PTest2 to enable splitting of tests?
>> MiniLlapCliDriver runs the same tests are MiniTez but in llap mode.
>> 
>> Thanks
>> Prasanth

[jira] [Created] (HIVE-11979) LLAP: merge master into branch

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-11979:
---

 Summary: LLAP: merge master into branch
 Key: HIVE-11979
 URL: https://issues.apache.org/jira/browse/HIVE-11979
 Project: Hive
  Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
 Fix For: llap


Again because of hbase-metastore changes




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11981) ORC Schema Evolution Issues (Vectorized, ACID, and Non-Vectorized)

[Matt McCline (JIRA)]

Matt McCline created HIVE-11981:
---

 Summary: ORC Schema Evolution Issues (Vectorized, ACID, and 
Non-Vectorized)
 Key: HIVE-11981
 URL: https://issues.apache.org/jira/browse/HIVE-11981
 Project: Hive
  Issue Type: Bug
  Components: Hive, Transactions
Reporter: Matt McCline
Assignee: Matt McCline
Priority: Critical


High priority issues with schema evolution for the ORC file format.

Schema evolution here is limited to adding new columns and a few cases of 
column type-widening (e.g. int to bigint).

Renaming columns, deleting column, moving columns and other schema evolution 
were not pursued due to lack of importance and lack of time.  Also, it appears 
a much more sophisticated metadata would be needed to support them.

The biggest issues for users have been adding new columns for ACID table 
(HIVE-11421 Support Schema evolution for ACID tables) and vectorization 
(HIVE-10598 Vectorization borks when column is added to table).




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11987) CompactionTxnHandler.createValidCompactTxnList() can use much less memory

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-11987:
-

 Summary: CompactionTxnHandler.createValidCompactTxnList() can use 
much less memory
 Key: HIVE-11987
 URL: https://issues.apache.org/jira/browse/HIVE-11987
 Project: Hive
  Issue Type: Bug
  Components: Metastore, Transactions
Affects Versions: 1.1.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman
Priority: Minor


This method only needs HWM and list of txn IDs in 'o' state.

It's currently implemented to get the list from TxnHandler.getOpenTxnsInfo(),
which returns (txn id, state, host, user) for each txn and includes Aborted 
txns.

This can easily be 120 bytes or more per txn overhead (over 1 Java long) which 
not an issue in general but when the system is misconfigured, the number of 
opened/aborted txns can get into the millions.  This creates unnecessary memory 
pressure on metastore.

Should consider fixing this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 38838: HIVE-11913 : Verify existence of tests for new changes in HiveQA

[Szehon Ho]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38838/
---

Review request for hive, Brock Noland and Sergio Pena.


Bugs: HIVE-11913
https://issues.apache.org/jira/browse/HIVE-11913


Repository: hive-git


Description
---

Make HiveQA give -1 if there are no tests added or modified (Test.*.java or 
.*.q) in the git patch.  Basic idea is to add a new test phase "TestCheckPhase" 
to read the git patch and parse for newly added tests, and channel this 
information to the JIRA reporting service.

There are some changes as I also did auto-cleanup the JIRAService class to meet 
the coding guidelines.


Diffs
-

  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/JIRAService.java 
c7be572 
  testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java 
7217ef9 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/TestCheckPhase.java
 PRE-CREATION 
  
testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestTestCheckPhase.java
 PRE-CREATION 
  testutils/ptest2/src/test/resources/HIVE-10761.6.patch PRE-CREATION 
  testutils/ptest2/src/test/resources/HIVE-11271.4.patch PRE-CREATION 
  testutils/ptest2/src/test/resources/HIVE-9377.1.patch PRE-CREATION 
  testutils/ptest2/src/test/resources/remove-test.patch PRE-CREATION 

Diff: https://reviews.apache.org/r/38838/diff/


Testing
---

Added unit tests.  Will need to see in production.


Thanks,

Szehon Ho

Re: Splitting tests in PTest2

[Prasanth Jayachandran]

@Szehon.. MiniLlapCliDriver  uses the same set of files from minitez. Can that 
be reused as such without duplicating?


> On Sep 28, 2015, at 6:49 PM, Prasanth Jayachandran 
>  wrote:
> 
> Thanks Szehon! I will look at it and update you once done.
> 
>> On Sep 28, 2015, at 6:37 PM, Szehon Ho  wrote:
>> 
>> Yes thanks for looking into it.  Can you edit testconfiguration.properties
>> to add a section for LlapCliDriver tests, see HIVE-7254 for details.  That
>> way the framework can identify which tests it needs to run and then split
>> it into parallel batches across test nodes.  Otherwise it will just run all
>> of the LlapCliDriver tests using -Dtest=TestMiniLlapCliDriver in one huge
>> batch.
>> 
>> Once you finish, let me know and I'll tweak the test properties to set a
>> batch size for that one.
>> 
>> Thanks
>> Szehon
>> 
>> On Mon, Sep 28, 2015 at 4:23 PM, Prasanth Jayachandran >> wrote:
>> 
>>> Hello everyone
>>> 
>>> Regarding HIVE-11642, the tests for MiniLlapCliDriver are not split across
>>> multiple nodes. I can see from the logs that MiniTezCliDriver are broken
>>> down into many tests and ran on different machines.
>>> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
>>> <
>>> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
 
>>> 
>>> How can we do the same for MiniLlapCliDriver tests? Do we need to do
>>> something in PTest2 to enable splitting of tests?
>>> MiniLlapCliDriver runs the same tests are MiniTez but in llap mode.
>>> 
>>> Thanks
>>> Prasanth
> 

Re: Splitting tests in PTest2

[Szehon Ho]

OK it may be possible, I will take a look!

Thanks
Szehon

On Mon, Sep 28, 2015 at 4:56 PM, Prasanth Jayachandran <
pjayachand...@hortonworks.com> wrote:

> @Szehon.. MiniLlapCliDriver  uses the same set of files from minitez. Can
> that be reused as such without duplicating?
>
>
> > On Sep 28, 2015, at 6:49 PM, Prasanth Jayachandran <
> pjayachand...@hortonworks.com> wrote:
> >
> > Thanks Szehon! I will look at it and update you once done.
> >
> >> On Sep 28, 2015, at 6:37 PM, Szehon Ho  wrote:
> >>
> >> Yes thanks for looking into it.  Can you edit
> testconfiguration.properties
> >> to add a section for LlapCliDriver tests, see HIVE-7254 for details.
> That
> >> way the framework can identify which tests it needs to run and then
> split
> >> it into parallel batches across test nodes.  Otherwise it will just run
> all
> >> of the LlapCliDriver tests using -Dtest=TestMiniLlapCliDriver in one
> huge
> >> batch.
> >>
> >> Once you finish, let me know and I'll tweak the test properties to set a
> >> batch size for that one.
> >>
> >> Thanks
> >> Szehon
> >>
> >> On Mon, Sep 28, 2015 at 4:23 PM, Prasanth Jayachandran <
> prasan...@apache.org
> >>> wrote:
> >>
> >>> Hello everyone
> >>>
> >>> Regarding HIVE-11642, the tests for MiniLlapCliDriver are not split
> across
> >>> multiple nodes. I can see from the logs that MiniTezCliDriver are
> broken
> >>> down into many tests and ran on different machines.
> >>>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
> >>> <
> >>>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
> 
> >>>
> >>> How can we do the same for MiniLlapCliDriver tests? Do we need to do
> >>> something in PTest2 to enable splitting of tests?
> >>> MiniLlapCliDriver runs the same tests are MiniTez but in llap mode.
> >>>
> >>> Thanks
> >>> Prasanth
> >
>
>

Avro column type in Hive

[Sergey Shelukhin]

Hi. 
I noticed that when I create an Avro table using a very large schema file,
mysql metastore silently truncates the TYPE_NAME in COLUMNS_V2 table to
the size of varchar (4000); however, when I do describe on the table, it
still displays the whole type name (around 100Kb long) that I presume it
gets from deserializer.
Is the value in TYPE_NAME used for anything for Avro tables?

[jira] [Created] (HIVE-11985) handle long typenames from Avro schema in metastore

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-11985:
---

 Summary: handle long typenames from Avro schema in metastore
 Key: HIVE-11985
 URL: https://issues.apache.org/jira/browse/HIVE-11985
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Splitting tests in PTest2

[Szehon Ho]

OK so there is a bit of a chicken-egg problem, I cannot change it until
11642 is in, otherwise it will break the other builds.

Once its in I can update the properties file to reference the new
LlapCliDriver.  Maybe you can check it in and I will try enabling it
tonight.

On Mon, Sep 28, 2015 at 5:05 PM, Szehon Ho  wrote:

> OK it may be possible, I will take a look!
>
> Thanks
> Szehon
>
> On Mon, Sep 28, 2015 at 4:56 PM, Prasanth Jayachandran <
> pjayachand...@hortonworks.com> wrote:
>
>> @Szehon.. MiniLlapCliDriver  uses the same set of files from minitez. Can
>> that be reused as such without duplicating?
>>
>>
>> > On Sep 28, 2015, at 6:49 PM, Prasanth Jayachandran <
>> pjayachand...@hortonworks.com> wrote:
>> >
>> > Thanks Szehon! I will look at it and update you once done.
>> >
>> >> On Sep 28, 2015, at 6:37 PM, Szehon Ho  wrote:
>> >>
>> >> Yes thanks for looking into it.  Can you edit
>> testconfiguration.properties
>> >> to add a section for LlapCliDriver tests, see HIVE-7254 for details.
>> That
>> >> way the framework can identify which tests it needs to run and then
>> split
>> >> it into parallel batches across test nodes.  Otherwise it will just
>> run all
>> >> of the LlapCliDriver tests using -Dtest=TestMiniLlapCliDriver in one
>> huge
>> >> batch.
>> >>
>> >> Once you finish, let me know and I'll tweak the test properties to set
>> a
>> >> batch size for that one.
>> >>
>> >> Thanks
>> >> Szehon
>> >>
>> >> On Mon, Sep 28, 2015 at 4:23 PM, Prasanth Jayachandran <
>> prasan...@apache.org
>> >>> wrote:
>> >>
>> >>> Hello everyone
>> >>>
>> >>> Regarding HIVE-11642, the tests for MiniLlapCliDriver are not split
>> across
>> >>> multiple nodes. I can see from the logs that MiniTezCliDriver are
>> broken
>> >>> down into many tests and ran on different machines.
>> >>>
>> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
>> >>> <
>> >>>
>> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
>> 
>> >>>
>> >>> How can we do the same for MiniLlapCliDriver tests? Do we need to do
>> >>> something in PTest2 to enable splitting of tests?
>> >>> MiniLlapCliDriver runs the same tests are MiniTez but in llap mode.
>> >>>
>> >>> Thanks
>> >>> Prasanth
>> >
>>
>>
>

Re: Avro column type in Hive

[Chaoyu Tang]

Yes, when you described the avro table, what you get back was actually from
your avro schema instead of database table. The avro table is NOT
considered as a metastore backed SerDe. But that it has its columns
populated to DB (e.g. HIVE-6308
) is mainly for column
statistics purpose, which obviously is not applicable to your case which
has a type name > 100kb.

Chaoyu

On Mon, Sep 28, 2015 at 8:12 PM, Sergey Shelukhin 
wrote:

> Hi.
> I noticed that when I create an Avro table using a very large schema file,
> mysql metastore silently truncates the TYPE_NAME in COLUMNS_V2 table to
> the size of varchar (4000); however, when I do describe on the table, it
> still displays the whole type name (around 100Kb long) that I presume it
> gets from deserializer.
> Is the value in TYPE_NAME used for anything for Avro tables?
>
>

Re: Splitting tests in PTest2

[Prasanth Jayachandran]

Szehon,

I committed a different patch now to llap branch, HIVE-11923. This will let 
llap and tez test to run much faster hopefully before the 2h timeout. Once all 
tests passes, committed then we can split up the tests to run in parallel. Even 
after HIVE-11923 if the patch didn’t complete before timeout then I will commit 
the patch and provide you an update to split up the tests.

Thanks
Prasanth

> On Sep 28, 2015, at 7:15 PM, Szehon Ho  wrote:
> 
> OK so there is a bit of a chicken-egg problem, I cannot change it until
> 11642 is in, otherwise it will break the other builds.
> 
> Once its in I can update the properties file to reference the new
> LlapCliDriver.  Maybe you can check it in and I will try enabling it
> tonight.
> 
> On Mon, Sep 28, 2015 at 5:05 PM, Szehon Ho  wrote:
> 
>> OK it may be possible, I will take a look!
>> 
>> Thanks
>> Szehon
>> 
>> On Mon, Sep 28, 2015 at 4:56 PM, Prasanth Jayachandran <
>> pjayachand...@hortonworks.com> wrote:
>> 
>>> @Szehon.. MiniLlapCliDriver  uses the same set of files from minitez. Can
>>> that be reused as such without duplicating?
>>> 
>>> 
 On Sep 28, 2015, at 6:49 PM, Prasanth Jayachandran <
>>> pjayachand...@hortonworks.com> wrote:
 
 Thanks Szehon! I will look at it and update you once done.
 
> On Sep 28, 2015, at 6:37 PM, Szehon Ho  wrote:
> 
> Yes thanks for looking into it.  Can you edit
>>> testconfiguration.properties
> to add a section for LlapCliDriver tests, see HIVE-7254 for details.
>>> That
> way the framework can identify which tests it needs to run and then
>>> split
> it into parallel batches across test nodes.  Otherwise it will just
>>> run all
> of the LlapCliDriver tests using -Dtest=TestMiniLlapCliDriver in one
>>> huge
> batch.
> 
> Once you finish, let me know and I'll tweak the test properties to set
>>> a
> batch size for that one.
> 
> Thanks
> Szehon
> 
> On Mon, Sep 28, 2015 at 4:23 PM, Prasanth Jayachandran <
>>> prasan...@apache.org
>> wrote:
> 
>> Hello everyone
>> 
>> Regarding HIVE-11642, the tests for MiniLlapCliDriver are not split
>>> across
>> multiple nodes. I can see from the logs that MiniTezCliDriver are
>>> broken
>> down into many tests and ran on different machines.
>> 
>>> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
>> <
>> 
>>> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
>>> 
>> 
>> How can we do the same for MiniLlapCliDriver tests? Do we need to do
>> something in PTest2 to enable splitting of tests?
>> MiniLlapCliDriver runs the same tests are MiniTez but in llap mode.
>> 
>> Thanks
>> Prasanth
 
>>> 
>>> 
>> 

[jira] [Created] (HIVE-11986) describe extended command is missing tableType, while describe formatted outputs it

[Aviv (JIRA)]

Aviv created HIVE-11986:
---

 Summary: describe extended command is missing tableType, while 
describe formatted outputs it
 Key: HIVE-11986
 URL: https://issues.apache.org/jira/browse/HIVE-11986
 Project: Hive
  Issue Type: Bug
  Components: Hive, HiveServer2, Thrift API
Affects Versions: 0.13.0
Reporter: Aviv
Priority: Minor


The output of this command "describe extended" is missing the tableType: 
property for some tables. it is being truncated after "serialization.format=   
, field.delim="

The output for: describe extended hivesampletable is:
"Detailed Table Information Table(tableName:hivesampletable, 
dbName:default, owner:hdpinternaluser, createTime:1440859101, lastAccessTime:0, 
retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:clientid, type:string, 
comment:null), FieldSchema(name:querytime, type:string, comment:null), 
FieldSchema(name:market, type:string, comment:null), 
FieldSchema(name:deviceplatform, type:string, comment:null), 
FieldSchema(name:devicemake, type:string, comment:null), 
FieldSchema(name:devicemodel, type:string, comment:null), 
FieldSchema(name:state, type:string, comment:null), FieldSchema(name:country, 
type:string, comment:null), FieldSchema(name:querydwelltime, type:double, 
comment:null), FieldSchema(name:sessionid, type:bigint, comment:null), 
FieldSchema(name:sessionpagevieworder, type:bigint, comment:null)], 
location:wasb://eddhivet...@eddhivetest.blob.core.windows.net/hive/warehouse/hivesampletable,
 inputFormat:org.apache.hadoop.mapred.TextInputFormat, 
outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, 
compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, 
serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, 
parameters:{serialization.format= , field.delim="

while the output for describe formatted hivesampletable is:
"# Detailed Table Information
Database:   default  
Owner:  hdpinternaluser  
CreateTime: Sat Aug 29 14:38:21 GMT 2015 
LastAccessTime: UNKNOWN  
Protect Mode:   None 
Retention:  0
Location:   
wasb://eddhivet...@eddhivetest.blob.core.windows.net/hive/warehouse/hivesampletable
  
Table Type: MANAGED_TABLE
Table Parameters:
COLUMN_STATS_ACCURATE   true
numFiles1   
numRows 59793   
rawDataSize 4895922 
totalSize   5015508 
transient_lastDdlTime   1441320153  
 
# Storage Information
SerDe Library:  org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe  
 
InputFormat:org.apache.hadoop.mapred.TextInputFormat 
OutputFormat:   
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat   
Compressed: No   
Num Buckets:-1   
Bucket Columns: []   
Sort Columns:   []   
Storage Desc Params: 
field.delim \t  
serialization.format\t  "


I am using Azure HDInsight , which uses hive-0.13.0.2.1.15.1-0001



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Avro column type in Hive

[kulkarni.swar...@gmail.com]

Sergey,

Is your table a partitioned or a non-partitioned one? I have usually seen
this problem manifest itself for partitioned tables and that is mostly
where the pruning bites. So if you now try to add a partition to this
table, you might see an exception like:

java.sql.BatchUpdateException: Data truncation: Data too long for column
'TYPE_NAME' at row 1)

The "TYPE_NAME" is not actually a definition of the Avro schema.  Instead,
it is a definition of the type structure in Hive terms.  I assume it is
used for things such as validating the query before it is executed, etc.

On Mon, Sep 28, 2015 at 7:38 PM, Chaoyu Tang  wrote:

> Yes, when you described the avro table, what you get back was actually from
> your avro schema instead of database table. The avro table is NOT
> considered as a metastore backed SerDe. But that it has its columns
> populated to DB (e.g. HIVE-6308
> ) is mainly for column
> statistics purpose, which obviously is not applicable to your case which
> has a type name > 100kb.
>
> Chaoyu
>
> On Mon, Sep 28, 2015 at 8:12 PM, Sergey Shelukhin 
> wrote:
>
> > Hi.
> > I noticed that when I create an Avro table using a very large schema
> file,
> > mysql metastore silently truncates the TYPE_NAME in COLUMNS_V2 table to
> > the size of varchar (4000); however, when I do describe on the table, it
> > still displays the whole type name (around 100Kb long) that I presume it
> > gets from deserializer.
> > Is the value in TYPE_NAME used for anything for Avro tables?
> >
> >
>



-- 
Swarnim