[jira] [Commented] (HIVE-12303) HCatRecordSerDe throw a IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-12303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15033341#comment-15033341
]
Xiaowei Wang commented on HIVE-12303:
-
[~sushanth]
The table uses a cutom serde . There is a column "applogs",a list of
struct in the table .Usually ,"applogs" will get the right struct elements ,but
sometime there is some dirty data ,it cannot get the right elements . A liltte
dirty data will not cause the fail of the job .
> HCatRecordSerDe throw a IndexOutOfBoundsException
>
>
> Key: HIVE-12303
> URL: https://issues.apache.org/jira/browse/HIVE-12303
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.14.0, 1.2.1
>Reporter: Xiaowei Wang
>Assignee: Xiaowei Wang
> Fix For: 1.2.1
>
> Attachments: HIVE-12303.0.patch
>
>
> When access hive table using hcatlog in Pig,sometime it throws a exception !
> Exception
> {noformat}
> 2015-10-30 06:44:35,219 WARN [Thread-4] org.apache.hadoop.mapred.YarnChild:
> Exception running child :
> org.apache.pig.backend.executionengine.ExecException: ERROR 6018: Error
> converting read value to tuple
> at
> org.apache.hive.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:76)
> at org.apache.hive.hcatalog.pig.HCatLoader.getNext(HCatLoader.java:59)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:204)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
> at
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.lang.IndexOutOfBoundsException: Index: 24, Size: 24
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hive.hcatalog.data.HCatRecordSerDe.serializeStruct(HCatRecordSerDe.java:175)
> at
> org.apache.hive.hcatalog.data.HCatRecordSerDe.serializeList(HCatRecordSerDe.java:244)
> at
> org.apache.hive.hcatalog.data.HCatRecordSerDe.serializeField(HCatRecordSerDe.java:196)
> at
> org.apache.hive.hcatalog.data.LazyHCatRecord.get(LazyHCatRecord.java:53)
> at
> org.apache.hive.hcatalog.data.LazyHCatRecord.get(LazyHCatRecord.java:97)
> at
> org.apache.hive.hcatalog.mapreduce.HCatRecordReader.nextKeyValue(HCatRecordReader.java:204)
> at
> org.apache.hive.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:63)
> ... 13 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12303) HCatRecordSerDe throw a IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-12303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15019245#comment-15019245
]
Xiaowei Wang commented on HIVE-12303:
-
Could you give me some advice ? Thanks
> HCatRecordSerDe throw a IndexOutOfBoundsException
>
>
> Key: HIVE-12303
> URL: https://issues.apache.org/jira/browse/HIVE-12303
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.14.0, 1.2.1
>Reporter: Xiaowei Wang
>Assignee: Sushanth Sowmyan
> Fix For: 1.2.1
>
> Attachments: HIVE-12303.0.patch
>
>
> When access hive table using hcatlog in Pig,sometime it throws a exception !
> Exception
> {noformat}
> 2015-10-30 06:44:35,219 WARN [Thread-4] org.apache.hadoop.mapred.YarnChild:
> Exception running child :
> org.apache.pig.backend.executionengine.ExecException: ERROR 6018: Error
> converting read value to tuple
> at
> org.apache.hive.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:76)
> at org.apache.hive.hcatalog.pig.HCatLoader.getNext(HCatLoader.java:59)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:204)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
> at
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.lang.IndexOutOfBoundsException: Index: 24, Size: 24
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hive.hcatalog.data.HCatRecordSerDe.serializeStruct(HCatRecordSerDe.java:175)
> at
> org.apache.hive.hcatalog.data.HCatRecordSerDe.serializeList(HCatRecordSerDe.java:244)
> at
> org.apache.hive.hcatalog.data.HCatRecordSerDe.serializeField(HCatRecordSerDe.java:196)
> at
> org.apache.hive.hcatalog.data.LazyHCatRecord.get(LazyHCatRecord.java:53)
> at
> org.apache.hive.hcatalog.data.LazyHCatRecord.get(LazyHCatRecord.java:97)
> at
> org.apache.hive.hcatalog.mapreduce.HCatRecordReader.nextKeyValue(HCatRecordReader.java:204)
> at
> org.apache.hive.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:63)
> ... 13 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12303) HCatRecordSerDe throw a IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-12303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15019244#comment-15019244
]
Xiaowei Wang commented on HIVE-12303:
-
Could you give me some advice ? Thanks
> HCatRecordSerDe throw a IndexOutOfBoundsException
>
>
> Key: HIVE-12303
> URL: https://issues.apache.org/jira/browse/HIVE-12303
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.14.0, 1.2.1
>Reporter: Xiaowei Wang
>Assignee: Sushanth Sowmyan
> Fix For: 1.2.1
>
> Attachments: HIVE-12303.0.patch
>
>
> When access hive table using hcatlog in Pig,sometime it throws a exception !
> Exception
> {noformat}
> 2015-10-30 06:44:35,219 WARN [Thread-4] org.apache.hadoop.mapred.YarnChild:
> Exception running child :
> org.apache.pig.backend.executionengine.ExecException: ERROR 6018: Error
> converting read value to tuple
> at
> org.apache.hive.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:76)
> at org.apache.hive.hcatalog.pig.HCatLoader.getNext(HCatLoader.java:59)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:204)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
> at
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.lang.IndexOutOfBoundsException: Index: 24, Size: 24
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hive.hcatalog.data.HCatRecordSerDe.serializeStruct(HCatRecordSerDe.java:175)
> at
> org.apache.hive.hcatalog.data.HCatRecordSerDe.serializeList(HCatRecordSerDe.java:244)
> at
> org.apache.hive.hcatalog.data.HCatRecordSerDe.serializeField(HCatRecordSerDe.java:196)
> at
> org.apache.hive.hcatalog.data.LazyHCatRecord.get(LazyHCatRecord.java:53)
> at
> org.apache.hive.hcatalog.data.LazyHCatRecord.get(LazyHCatRecord.java:97)
> at
> org.apache.hive.hcatalog.mapreduce.HCatRecordReader.nextKeyValue(HCatRecordReader.java:204)
> at
> org.apache.hive.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:63)
> ... 13 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12303) HCatRecordSerDe throw a IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-12303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15019246#comment-15019246
]
Xiaowei Wang commented on HIVE-12303:
-
[~sushanth]
> HCatRecordSerDe throw a IndexOutOfBoundsException
>
>
> Key: HIVE-12303
> URL: https://issues.apache.org/jira/browse/HIVE-12303
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.14.0, 1.2.1
>Reporter: Xiaowei Wang
>Assignee: Sushanth Sowmyan
> Fix For: 1.2.1
>
> Attachments: HIVE-12303.0.patch
>
>
> When access hive table using hcatlog in Pig,sometime it throws a exception !
> Exception
> {noformat}
> 2015-10-30 06:44:35,219 WARN [Thread-4] org.apache.hadoop.mapred.YarnChild:
> Exception running child :
> org.apache.pig.backend.executionengine.ExecException: ERROR 6018: Error
> converting read value to tuple
> at
> org.apache.hive.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:76)
> at org.apache.hive.hcatalog.pig.HCatLoader.getNext(HCatLoader.java:59)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:204)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
> at
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.lang.IndexOutOfBoundsException: Index: 24, Size: 24
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hive.hcatalog.data.HCatRecordSerDe.serializeStruct(HCatRecordSerDe.java:175)
> at
> org.apache.hive.hcatalog.data.HCatRecordSerDe.serializeList(HCatRecordSerDe.java:244)
> at
> org.apache.hive.hcatalog.data.HCatRecordSerDe.serializeField(HCatRecordSerDe.java:196)
> at
> org.apache.hive.hcatalog.data.LazyHCatRecord.get(LazyHCatRecord.java:53)
> at
> org.apache.hive.hcatalog.data.LazyHCatRecord.get(LazyHCatRecord.java:97)
> at
> org.apache.hive.hcatalog.mapreduce.HCatRecordReader.nextKeyValue(HCatRecordReader.java:204)
> at
> org.apache.hive.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:63)
> ... 13 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12303) HCatRecordSerDe throw a IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-12303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14985077#comment-14985077
]
Xiaowei Wang commented on HIVE-12303:
-
The schema is
{noformat}
# col_name data_type comment
ip string from deserializer
manualtime string from deserializer
timezonestring from deserializer
pbparamsmap from deserializer
pageurl string from deserializer
useragent string from deserializer
yyidstring from deserializer
suv string from deserializer
linestring from deserializer
applogs
array>
from deserializer
# Partition Information
# col_name data_type comment
logdate string
# Detailed Table Information
Database: default
Owner: hive
CreateTime: Fri Nov 08 11:38:00 CST 2013
LastAccessTime: UNKNOWN
Protect Mode: None
Retention: 0
Location:
viewfs://nsX/user/hive/warehouse/default.db/web/uigs/web_uigs_wapsearch
Table Type: EXTERNAL_TABLE
Table Parameters:
EXTERNALTRUE
last_modified_byslave
last_modified_time 1414463853
transient_lastDdlTime 1414463853
# Storage Information
SerDe Library: com.custom.datacat.hive.DataCatSerde
InputFormat:com.custom.datadir.plugin.SymlinkLzoTextInputFormat
OutputFormat:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
Compressed: No
Num Buckets:-1
Bucket Columns: []
Sort Columns: []
Storage Desc Params:
datacat.fieldInspector
applogs:com.custom.datacat.hive.DataCatListObjectInspector:\t:com.custom.datacat.hive.DataCatMapObjectInspector
datacat.lineInspector
com.custom.datacat.wapapp.WapAppSearchInspector:
serialization.format1
{noformat}
> HCatRecordSerDe throw a IndexOutOfBoundsException
>
>
> Key: HIVE-12303
> URL: https://issues.apache.org/jira/browse/HIVE-12303
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.14.0, 1.2.1
>Reporter: Xiaowei Wang
>Assignee: Sushanth Sowmyan
> Fix For: 1.2.1
>
> Attachments: HIVE-12303.0.patch
>
>
> When access hive table using hcatlog in Pig,sometime it throws a exception !
> Exception
> {noformat}
> 2015-10-30 06:44:35,219 WARN [Thread-4] org.apache.hadoop.mapred.YarnChild:
> Exception running child :
> org.apache.pig.backend.executionengine.ExecException: ERROR 6018: Error
> converting read value to tuple
> at
> org.apache.hive.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:76)
> at org.apache.hive.hcatalog.pig.HCatLoader.getNext(HCatLoader.java:59)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:204)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
> at
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
>
[jira] [Commented] (HIVE-12303) HCatRecordSerDe throw a IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-12303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14982425#comment-14982425
]
Hive QA commented on HIVE-12303:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12769722/HIVE-12303.0.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 9734 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.hwi.TestHWISessionManager.testHiveDriver
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5861/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5861/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5861/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12769722 - PreCommit-HIVE-TRUNK-Build
> HCatRecordSerDe throw a IndexOutOfBoundsException
>
>
> Key: HIVE-12303
> URL: https://issues.apache.org/jira/browse/HIVE-12303
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.14.0, 1.2.1
>Reporter: Xiaowei Wang
>Assignee: Sushanth Sowmyan
> Fix For: 1.2.1
>
> Attachments: HIVE-12303.0.patch
>
>
> When access hive table using hcatlog in Pig,sometime it throws a exception !
> Exception
> {noformat}
> 2015-10-30 06:44:35,219 WARN [Thread-4] org.apache.hadoop.mapred.YarnChild:
> Exception running child :
> org.apache.pig.backend.executionengine.ExecException: ERROR 6018: Error
> converting read value to tuple
> at
> org.apache.hive.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:76)
> at org.apache.hive.hcatalog.pig.HCatLoader.getNext(HCatLoader.java:59)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:204)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
> at
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.lang.IndexOutOfBoundsException: Index: 24, Size: 24
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hive.hcatalog.data.HCatRecordSerDe.serializeStruct(HCatRecordSerDe.java:175)
> at
> org.apache.hive.hcatalog.data.HCatRecordSerDe.serializeList(HCatRecordSerDe.java:244)
> at
> org.apache.hive.hcatalog.data.HCatRecordSerDe.serializeField(HCatRecordSerDe.java:196)
> at
> org.apache.hive.hcatalog.data.LazyHCatRecord.get(LazyHCatRecord.java:53)
> at
> org.apache.hive.hcatalog.data.LazyHCatRecord.get(LazyHCatRecord.java:97)
> at
> org.apache.hive.hcatalog.mapreduce.HCatRecordReader.nextKeyValue(HCatRecordReader.java:204)
> at
> org.apache.hive.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:63)
> ... 13 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12303) HCatRecordSerDe throw a IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-12303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14982458#comment-14982458
]
Xiaowei Wang commented on HIVE-12303:
-
Is this a bug ?
[~ashutoshc] [~xuefuz] [~gopalv]
> HCatRecordSerDe throw a IndexOutOfBoundsException
>
>
> Key: HIVE-12303
> URL: https://issues.apache.org/jira/browse/HIVE-12303
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.14.0, 1.2.1
>Reporter: Xiaowei Wang
>Assignee: Sushanth Sowmyan
> Fix For: 1.2.1
>
> Attachments: HIVE-12303.0.patch
>
>
> When access hive table using hcatlog in Pig,sometime it throws a exception !
> Exception
> {noformat}
> 2015-10-30 06:44:35,219 WARN [Thread-4] org.apache.hadoop.mapred.YarnChild:
> Exception running child :
> org.apache.pig.backend.executionengine.ExecException: ERROR 6018: Error
> converting read value to tuple
> at
> org.apache.hive.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:76)
> at org.apache.hive.hcatalog.pig.HCatLoader.getNext(HCatLoader.java:59)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:204)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
> at
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.lang.IndexOutOfBoundsException: Index: 24, Size: 24
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hive.hcatalog.data.HCatRecordSerDe.serializeStruct(HCatRecordSerDe.java:175)
> at
> org.apache.hive.hcatalog.data.HCatRecordSerDe.serializeList(HCatRecordSerDe.java:244)
> at
> org.apache.hive.hcatalog.data.HCatRecordSerDe.serializeField(HCatRecordSerDe.java:196)
> at
> org.apache.hive.hcatalog.data.LazyHCatRecord.get(LazyHCatRecord.java:53)
> at
> org.apache.hive.hcatalog.data.LazyHCatRecord.get(LazyHCatRecord.java:97)
> at
> org.apache.hive.hcatalog.mapreduce.HCatRecordReader.nextKeyValue(HCatRecordReader.java:204)
> at
> org.apache.hive.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:63)
> ... 13 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12303) HCatRecordSerDe throw a IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-12303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14982835#comment-14982835
]
Sushanth Sowmyan commented on HIVE-12303:
-
Can you provide the following info:
a) The schema of the table in hive, and what type of storage is used.
b) The schema of partitions (if any) in hive.
c) What pig says the schema is, when you do a describe of that table.
Basically, HCatRecordSerDe uses the size it's told. If it's told that there are
> 24 columns or elements in a struct, it'll try to access them, and in this
case, fail since only 24 items were present in the raw data beneath. So I would
compare the data with the schema first in this case.
> HCatRecordSerDe throw a IndexOutOfBoundsException
>
>
> Key: HIVE-12303
> URL: https://issues.apache.org/jira/browse/HIVE-12303
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.14.0, 1.2.1
>Reporter: Xiaowei Wang
>Assignee: Sushanth Sowmyan
> Fix For: 1.2.1
>
> Attachments: HIVE-12303.0.patch
>
>
> When access hive table using hcatlog in Pig,sometime it throws a exception !
> Exception
> {noformat}
> 2015-10-30 06:44:35,219 WARN [Thread-4] org.apache.hadoop.mapred.YarnChild:
> Exception running child :
> org.apache.pig.backend.executionengine.ExecException: ERROR 6018: Error
> converting read value to tuple
> at
> org.apache.hive.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:76)
> at org.apache.hive.hcatalog.pig.HCatLoader.getNext(HCatLoader.java:59)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:204)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
> at
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.lang.IndexOutOfBoundsException: Index: 24, Size: 24
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hive.hcatalog.data.HCatRecordSerDe.serializeStruct(HCatRecordSerDe.java:175)
> at
> org.apache.hive.hcatalog.data.HCatRecordSerDe.serializeList(HCatRecordSerDe.java:244)
> at
> org.apache.hive.hcatalog.data.HCatRecordSerDe.serializeField(HCatRecordSerDe.java:196)
> at
> org.apache.hive.hcatalog.data.LazyHCatRecord.get(LazyHCatRecord.java:53)
> at
> org.apache.hive.hcatalog.data.LazyHCatRecord.get(LazyHCatRecord.java:97)
> at
> org.apache.hive.hcatalog.mapreduce.HCatRecordReader.nextKeyValue(HCatRecordReader.java:204)
> at
> org.apache.hive.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:63)
> ... 13 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
