[jira] [Created] (HIVE-13286) Query ID is being reused across queries

2016-03-14 Thread Vikram Dixit K (JIRA) 
Vikram Dixit K created HIVE-13286:
-

 Summary: Query ID is being reused across queries
 Key: HIVE-13286
 URL: https://issues.apache.org/jira/browse/HIVE-13286
 Project: Hive
  Issue Type: Bug
  Components: Parser
Affects Versions: 2.0.0
Reporter: Vikram Dixit K
Assignee: Pengcheng Xiong
Priority: Critical


[~aihuaxu] I see this commit made via HIVE-11488. I see that query id is being 
reused across queries. This defeats the purpose of a query id. I am not sure 
what the purpose of the change in that jira is but it breaks the assumption 
about a query id being unique for each query. Please take a look into this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13285) Orc concatenation may drop old files from moving to final path

2016-03-14 Thread Prasanth Jayachandran (JIRA) 
Prasanth Jayachandran created HIVE-13285:


 Summary: Orc concatenation may drop old files from moving to final 
path
 Key: HIVE-13285
 URL: https://issues.apache.org/jira/browse/HIVE-13285
 Project: Hive
  Issue Type: Bug
  Components: ORC
Affects Versions: 2.0.0, 1.1.0, 1.2.0, 1.0.0, 0.14.0, 1.3.0, 2.1.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran
Priority: Critical


ORC concatenation uses combine hive input format for merging files. Under 
specific case where all files within a combine split are incompatible for merge 
(old files without stripe statistics) then these files are added to 
incompatible file set. But this file set is not processed as closeOp() will not 
be called (no output file writer will exist which will skip super.closeOp()). 
As a result, these incompatible files are not moved to final path.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13284) Make ORC Reader resilient to 0 length files

2016-03-14 Thread Prasanth Jayachandran (JIRA) 
Prasanth Jayachandran created HIVE-13284:


 Summary: Make ORC Reader resilient to 0 length files
 Key: HIVE-13284
 URL: https://issues.apache.org/jira/browse/HIVE-13284
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.1.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


HIVE-13040 creates 0 length ORC files. Reading such files will throw following 
exception. ORC is resilient to corrupt footers but not 0 length files.

{code}
Processing data file file:/app/warehouse/concat_incompat/00_0 [length: 0]
Exception in thread "main" java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkIndex(Buffer.java:540)
at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
at 
org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:510)
at 
org.apache.hadoop.hive.ql.io.orc.ReaderImpl.(ReaderImpl.java:361)
at 
org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:83)
at 
org.apache.hadoop.hive.ql.io.orc.FileDump.getReader(FileDump.java:239)
at 
org.apache.hadoop.hive.ql.io.orc.FileDump.printMetaDataImpl(FileDump.java:312)
at 
org.apache.hadoop.hive.ql.io.orc.FileDump.printMetaData(FileDump.java:291)
at org.apache.hadoop.hive.ql.io.orc.FileDump.main(FileDump.java:138)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 44756: Support masking and filtering of rows/columns

2016-03-14 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44756/
---

(Updated March 14, 2016, 10:50 p.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

HIVE-13125


Diffs (updated)
-

  
itests/util/src/main/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/SQLStdHiveAuthorizationValidatorForTest.java
 fd39c67 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 2dcb6d6 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TableMask.java PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveAuthorizationValidator.java
 59aabe4 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveAuthorizer.java
 c93e334 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveAuthorizerImpl.java
 00fa8cf 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveV1Authorizer.java
 8a03989 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/DummyHiveAuthorizationValidator.java
 26e3a2c 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/SQLStdHiveAuthorizationValidator.java
 9f586be 
  ql/src/test/queries/clientpositive/masking_1.q PRE-CREATION 
  ql/src/test/queries/clientpositive/masking_2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/masking_3.q PRE-CREATION 
  ql/src/test/queries/clientpositive/masking_4.q PRE-CREATION 
  ql/src/test/queries/clientpositive/masking_disablecbo_1.q PRE-CREATION 
  ql/src/test/queries/clientpositive/masking_disablecbo_2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/masking_disablecbo_3.q PRE-CREATION 
  ql/src/test/queries/clientpositive/masking_disablecbo_4.q PRE-CREATION 
  ql/src/test/results/clientpositive/masking_1.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/masking_2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/masking_3.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/masking_4.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/masking_disablecbo_1.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/masking_disablecbo_2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/masking_disablecbo_3.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/masking_disablecbo_4.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/44756/diff/


Testing
---


Thanks,

pengcheng xiong

[jira] [Created] (HIVE-13283) LLAP: make sure IO elevator is enabled by default in the daemons

2016-03-14 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-13283:
---

 Summary: LLAP: make sure IO elevator is enabled by default in the 
daemons
 Key: HIVE-13283
 URL: https://issues.apache.org/jira/browse/HIVE-13283
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13282) GroupBy and select operator encounter ArrayIndexOutOfBoundsException

2016-03-14 Thread Vikram Dixit K (JIRA) 
Vikram Dixit K created HIVE-13282:
-

 Summary: GroupBy and select operator encounter 
ArrayIndexOutOfBoundsException
 Key: HIVE-13282
 URL: https://issues.apache.org/jira/browse/HIVE-13282
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 2.0.0, 1.2.1, 2.1.0
Reporter: Vikram Dixit K
Assignee: Vikram Dixit K


The group by and select operators run into the ArrayIndexOutOfBoundsException 
when they incorrectly initialize themselves with tag 0 but the incoming tag id 
is different.

{code}
select count(*) from
(select rt1.id from
(select t1.key as id, t1.value as od from tab t1 group by key, value) rt1) vt1
join
(select rt2.id from
(select t2.key as id, t2.value as od from tab_part t2 group by key, value) rt2) 
vt2
where vt1.id=vt2.id;
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13281) Update some default configs for LLAP

2016-03-14 Thread Siddharth Seth (JIRA) 
Siddharth Seth created HIVE-13281:
-

 Summary: Update some default configs for LLAP
 Key: HIVE-13281
 URL: https://issues.apache.org/jira/browse/HIVE-13281
 Project: Hive
  Issue Type: Task
Reporter: Siddharth Seth


Disable uber mode.
Enable llap.io by default



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13280) Error when more than 1 mapper for HBase storage handler

2016-03-14 Thread Damien Carol (JIRA) 
Damien Carol created HIVE-13280:
---

 Summary: Error when more than 1 mapper for HBase storage handler
 Key: HIVE-13280
 URL: https://issues.apache.org/jira/browse/HIVE-13280
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Damien Carol






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13279) SHOW TABLE EXTENDED doesn't show the correct lastUpdateTime of partition's file system

2016-03-14 Thread Aleksey Vovchenko (JIRA) 
Aleksey Vovchenko created HIVE-13279:


 Summary: SHOW TABLE EXTENDED doesn't show the correct 
lastUpdateTime of partition's file system
 Key: HIVE-13279
 URL: https://issues.apache.org/jira/browse/HIVE-13279
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Aleksey Vovchenko
Assignee: Aleksey Vovchenko


h2. STEP 1. Create test Tables

Execute in command line:

{noformat} 
nano test.data
{noformat} 

Add to file:

{noformat}
1,aa
2,aa
3,ff
4,sad
5,adsf
6,adsf
7,affss
{noformat}

{noformat}
hadoop fs -put test.data /
{noformat} 

{noformat}
hive> create table test (x int, y string, z string) ROW FORMAT DELIMITED FIELDS 
TERMINATED BY ',';
hive> create table ptest(x int, y string) partitioned by(z string); 
hive> LOAD DATA  INPATH '/test.data' OVERWRITE INTO TABLE test;
hive> insert overwrite table ptest partition(z=65) select * from test;
hive> insert overwrite table ptest partition(z=67) select * from test;
{noformat}

h2. STEP 2. Compare lastUpdateTime

Execute in Hive shell:

{noformat}
hive> SHOW TABLE EXTENDED FROM default LIKE 'ptest' PARTITION(z='65');
hive> SHOW TABLE EXTENDED FROM default LIKE 'ptest' PARTITION(z='67');
{noformat}

lastUpdateTime should be different.

h2. STEP 3. Put data into hdfs and compare lastUpdateTime

Execute in command line:

{noformat}
hadoop fs -put test.data /user/hive/warehouse/ptest
{noformat}

Execute in Hive shell:

{noformat}
hive> SHOW TABLE EXTENDED FROM default LIKE 'ptest' PARTITION(z='65');
hive> SHOW TABLE EXTENDED FROM default LIKE 'ptest' PARTITION(z='67');
{noformat}

lastUpdateTime should be different but they are same.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 44783: HIVE-13183: More logs in operation logs

2016-03-14 Thread Amareshwari Sriramadasu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44783/#review123391
---




ql/src/java/org/apache/hadoop/hive/ql/exec/TaskRunner.java (line 106)


Can you avoid printStackTrace() and change it log ?



ql/src/java/org/apache/hadoop/hive/ql/exec/TaskRunner.java (line 108)


Can we directly log instead of printStackTrace?


- Amareshwari Sriramadasu


On March 14, 2016, 9:23 a.m., Rajat Khandelwal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44783/
> ---
> 
> (Updated March 14, 2016, 9:23 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-13183
> https://issues.apache.org/jira/browse/HIVE-13183
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hive/common/util/StreamPrinter.java 
> 72638fdb966a63c64e44f2d952ecc49b4b0b5121 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TaskRunner.java 
> f6fd08104f6d5a8c93bca1f5d77468096e823144 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 
> f5500a4c4ece56fcd756c9c2159169228e542a6e 
>   ql/src/java/org/apache/hadoop/hive/ql/session/OperationLog.java 
> 2ecdde9799683cd104323af170d8905fb3005b9e 
> 
> Diff: https://reviews.apache.org/r/44783/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Rajat Khandelwal
> 
>

Re: Review Request 44783: HIVE-13183: More logs in operation logs

2016-03-14 Thread Rajat Khandelwal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44783/
---

(Updated March 14, 2016, 2:53 p.m.)


Review request for hive.


Bugs: HIVE-13183
https://issues.apache.org/jira/browse/HIVE-13183


Repository: hive-git


Description
---


Diffs (updated)
-

  common/src/java/org/apache/hive/common/util/StreamPrinter.java 
72638fdb966a63c64e44f2d952ecc49b4b0b5121 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TaskRunner.java 
f6fd08104f6d5a8c93bca1f5d77468096e823144 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 
f5500a4c4ece56fcd756c9c2159169228e542a6e 
  ql/src/java/org/apache/hadoop/hive/ql/session/OperationLog.java 
2ecdde9799683cd104323af170d8905fb3005b9e 

Diff: https://reviews.apache.org/r/44783/diff/


Testing
---


Thanks,

Rajat Khandelwal

Review Request 44783: HIVE-13183: More logs in operation logs

2016-03-14 Thread Rajat Khandelwal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44783/
---

Review request for hive.


Bugs: HIVE-13183
https://issues.apache.org/jira/browse/HIVE-13183


Repository: hive-git


Description
---


Diffs
-

  common/src/java/org/apache/hive/common/util/StreamPrinter.java 
72638fdb966a63c64e44f2d952ecc49b4b0b5121 
  
itests/hive-unit/src/test/java/org/apache/hive/service/cli/TestEmbeddedThriftBinaryCLIService.java
 de66d9efb1cace9d32174e3020920d5e4002dc85 
  
itests/hive-unit/src/test/java/org/apache/hive/service/cli/session/TestQueryDisplay.java
 9765b9dbe1ae097c108437c85b11b7bcee8f49da 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
b50c5a2c4e396975f3323b16426445f869ff7b89 
  ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java 
c87c825f95d2ff49bc62f892eb1cea5abf3c65c6 
  ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java 
4933b34710afebde923b548572ffd7f98b7eb728 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 
5253521557d09f12163624606373d64cde005d70 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 
e199e5ee65404d2e9a38d20fa9d2ff72754962e9 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TaskRunner.java 
f6fd08104f6d5a8c93bca1f5d77468096e823144 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 
ce020a52721b439c4a9a81ee75e9829f2da00221 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java 
5bc3d9e732330af7890d7379334ed42be116ccbe 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 
f5500a4c4ece56fcd756c9c2159169228e542a6e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
dd59bf275eecea178b8154962a79a385b02fda40 
  ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java 
45cd533270a550ae07510fe3974ddceaed30af51 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveTableScan.java
 e9e9d0b314e9f9f2ac8e27689f48ed1077afa36c 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
 d39744b95cc51fa3042d019fb3bd05136d5dedc3 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
607eb4fd1b69390010914af236e92a34428e2088 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
c36aa9de2549bceeb6fcf048fe2b56fe74fbe51b 
  ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java 
91352b22d0663b928e6f4f655914bdcf44cfa65c 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
633c212bc32bff68a63a6e7552cfde741c5553e5 
  ql/src/java/org/apache/hadoop/hive/ql/session/OperationLog.java 
2ecdde9799683cd104323af170d8905fb3005b9e 
  ql/src/test/queries/clientnegative/authorization_view_5.q PRE-CREATION 
  ql/src/test/queries/clientnegative/authorization_view_6.q PRE-CREATION 
  ql/src/test/queries/clientnegative/authorization_view_7.q PRE-CREATION 
  ql/src/test/queries/clientnegative/authorization_view_disable_cbo_5.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/authorization_view_disable_cbo_6.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/authorization_view_disable_cbo_7.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/authorization_view_2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/authorization_view_3.q PRE-CREATION 
  ql/src/test/queries/clientpositive/authorization_view_4.q PRE-CREATION 
  ql/src/test/queries/clientpositive/authorization_view_disable_cbo_2.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/authorization_view_disable_cbo_3.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/authorization_view_disable_cbo_4.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/vector_udf1.q PRE-CREATION 
  ql/src/test/results/clientnegative/authorization_view_5.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/authorization_view_6.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/authorization_view_7.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/authorization_view_disable_cbo_5.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/authorization_view_disable_cbo_6.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/authorization_view_disable_cbo_7.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/authorization_view_2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/authorization_view_3.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/authorization_view_4.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/authorization_view_disable_cbo_2.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/authorization_view_disable_cbo_3.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/authorization_view_disable_cbo_4.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/subquery_views.q.out 
6ab2ad0e670bd03cb5a61e3f16b2780845faa991 
  ql/src/test/results/clientpositive/tez/explainuser_1.q.out 
a3ff85ca967e18529d8addfff2375bb797bc44fa

[jira] [Created] (HIVE-13278) Many redundant 'File not found' messages appeared in container log during query execution with Hive on Spark

2016-03-14 Thread Xin Hao (JIRA) 
Xin Hao created HIVE-13278:
--

 Summary: Many redundant 'File not found' messages appeared in 
container log during query execution with Hive on Spark
 Key: HIVE-13278
 URL: https://issues.apache.org/jira/browse/HIVE-13278
 Project: Hive
  Issue Type: Bug
 Environment: Hive on Spark engine
Found based on :
Apache Hive 2.0.0
Apache Spark 1.6.0
Reporter: Xin Hao
Priority: Minor


Many redundant 'File not found' messages appeared in container log during query 
execution with Hive on Spark

Error message example:
16/03/14 01:45:06 INFO exec.Utilities: File not found: File does not exist: 
/tmp/hive/hadoop/2d378538-f5d3-493c-9276-c62dd6634fb4/hive_2016-03-14_01-44-16_835_623058724409492515-6/-mr-10010/0a6d0cae-1eb3-448c-883b-590b3b198a73/reduce.xml
at 
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at 
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1932)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1853)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1825)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:565)
at 
org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13277) Exception "Unable to create serializer 'org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer' " occurred during query execution on spark engine when vect

2016-03-14 Thread Xin Hao (JIRA) 
Xin Hao created HIVE-13277:
--

 Summary: Exception "Unable to create serializer 
'org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer' " 
occurred during query execution on spark engine when vectorized execution is 
switched on
 Key: HIVE-13277
 URL: https://issues.apache.org/jira/browse/HIVE-13277
 Project: Hive
  Issue Type: Bug
 Environment: Hive on Spark engine
Hive Version: Apache Hive 2.0.0
Spark Version: Apache Spark 1.6.0
Reporter: Xin Hao


Found when executing TPCx-BB query2 for Hive on Spark engine, and switch on :
Found during TPCx-BB query2 execution on spark engine when vectorized execution 
is switched on:
(1) set hive.vectorized.execution.enabled=true; 
(2) set hive.vectorized.execution.reduce.enabled=true; (default value for 
Apache Hive 2.0.0)
It's OK for spark engine when hive.vectorized.execution.enabled is switched off:
(1) set hive.vectorized.execution.enabled=false;
(2) set hive.vectorized.execution.reduce.enabled=true;

For MR engine, the query could pass and no exception occurred when vectorized 
execution is either switched on or switched off.

Detail Error Message is below:
2016-03-14T10:09:33,692 INFO  [stderr-redir-1]: client.SparkClientImpl 
(SparkClientImpl.java:run(593)) - 16/03/14 10:09:33 INFO 
spark.MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 154 bytes
2016-03-14T10:09:33,818 INFO  [stderr-redir-1]: client.SparkClientImpl 
(SparkClientImpl.java:run(593)) - 16/03/14 10:09:33 WARN 
scheduler.TaskSetManager: Lost task 0.0 in stage 4.0 (TID 25, bhx3): 
java.lang.RuntimeException: Failed to load plan: 
hdfs://bhx3:8020/tmp/hive/root/40b90ebd-32d4-47bc-a5ab-12ff1c05d0d2/hive_2016-03-14_10-08-56_307_7692316402338632647-1/-mr-10002/ab0c0021-0c1a-496e-9703-87d5879353c8/reduce.xml:
 org.apache.hive.com.esotericsoftware.kryo.KryoException: 
java.lang.IllegalArgumentException: Unable to create serializer 
"org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for 
class: org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator
2016-03-14T10:09:33,818 INFO  [stderr-redir-1]: client.SparkClientImpl 
(SparkClientImpl.java:run(593)) - Serialization trace:
2016-03-14T10:09:33,818 INFO  [stderr-redir-1]: client.SparkClientImpl 
(SparkClientImpl.java:run(593)) - childOperators 
(org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator)
2016-03-14T10:09:33,818 INFO  [stderr-redir-1]: client.SparkClientImpl 
(SparkClientImpl.java:run(593)) - childOperators 
(org.apache.hadoop.hive.ql.exec.vector.VectorLimitOperator)
2016-03-14T10:09:33,818 INFO  [stderr-redir-1]: client.SparkClientImpl 
(SparkClientImpl.java:run(593)) - childOperators 
(org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator)
2016-03-14T10:09:33,818 INFO  [stderr-redir-1]: client.SparkClientImpl 
(SparkClientImpl.java:run(593)) - reducer 
(org.apache.hadoop.hive.ql.plan.ReduceWork)
2016-03-14T10:09:33,818 INFO  [stderr-redir-1]: client.SparkClientImpl 
(SparkClientImpl.java:run(593)) -at 
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:451)
2016-03-14T10:09:33,818 INFO  [stderr-redir-1]: client.SparkClientImpl 
(SparkClientImpl.java:run(593)) -at 
org.apache.hadoop.hive.ql.exec.Utilities.getReduceWork(Utilities.java:306)
2016-03-14T10:09:33,819 INFO  [stderr-redir-1]: client.SparkClientImpl 
(SparkClientImpl.java:run(593)) -at 
org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.init(SparkReduceRecordHandler.java:117)
2016-03-14T10:09:33,819 INFO  [stderr-redir-1]: client.SparkClientImpl 
(SparkClientImpl.java:run(593)) -at 
org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunction.call(HiveReduceFunction.java:46)
2016-03-14T10:09:33,819 INFO  [stderr-redir-1]: client.SparkClientImpl 
(SparkClientImpl.java:run(593)) -at 
org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunction.call(HiveReduceFunction.java:28)
2016-03-14T10:09:33,819 INFO  [stderr-redir-1]: client.SparkClientImpl 
(SparkClientImpl.java:run(593)) -at 
org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:192)
2016-03-14T10:09:33,819 INFO  [stderr-redir-1]: client.SparkClientImpl 
(SparkClientImpl.java:run(593)) -at 
org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:192)
2016-03-14T10:09:33,819 INFO  [stderr-redir-1]: client.SparkClientImpl 
(SparkClientImpl.java:run(593)) -at 
org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
2016-03-14T10:09:33,819 INFO  [stderr-redir-1]: client.SparkClientImpl 
(SparkClientImpl.java:run(593)) -at 
org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
2016-03-14T10:09:33,819 INFO  [stderr-redir-1]: client.SparkClientImpl 
(SparkClientImpl.java:run(593)) -at