[jira] [Commented] (HIVE-2540) LATERAL VIEW with EXPLODE produces ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HIVE-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976820#comment-13976820
]
Lars Francke commented on HIVE-2540:
We're still seeing this issue with Hive 0.12 (slightly patched as it's the one
from CDH5):
{quote}
2014-04-22 16:18:50,304 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row
{product_id:140719,active:null,avg_product_price:null,base_cats:[[1247,818,821,902,905],[1247,818,821,902,906],[1247,818,821,902,909]]}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:175)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row
{product_id:140719,active:null,avg_product_price:null,base_cats:[[1247,818,821,902,905],[1247,818,821,902,906],[1247,818,821,902,909]]}
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:529)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:157)
... 8 more
Caused by: java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)
at java.util.ArrayList$Itr.next(ArrayList.java:831)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDTFExplode.process(GenericUDTFExplode.java:92)
at
org.apache.hadoop.hive.ql.exec.UDTFOperator.processOp(UDTFOperator.java:113)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:844)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:844)
at
org.apache.hadoop.hive.ql.exec.LateralViewForwardOperator.processOp(LateralViewForwardOperator.java:37)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:844)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:844)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:519)
... 9 more
{quote}
I haven't had time to check whether this patch is actually included but I'd
expect it to be. Shall I open a new issue?
LATERAL VIEW with EXPLODE produces ConcurrentModificationException
--
Key: HIVE-2540
URL: https://issues.apache.org/jira/browse/HIVE-2540
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.7.1, 0.9.0
Reporter: David Phillips
Assignee: Navis
Fix For: 0.10.0
Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-2540.D2805.1.patch,
ASF.LICENSE.NOT.GRANTED--HIVE-2540.D2805.2.patch,
ASF.LICENSE.NOT.GRANTED--HIVE-2540.D2805.3.patch
The following produces {{ConcurrentModificationException}} on the {{for}}
loop inside EXPLODE:
{code}
create table foo as select array(1, 2) a from src limit 1;
select a, x.b from foo lateral view explode(a) x as b;
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-2540) LATERAL VIEW with EXPLODE produces ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HIVE-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13547856#comment-13547856
]
Hudson commented on HIVE-2540:
--
Integrated in Hive-trunk-hadoop2 #54 (See
[https://builds.apache.org/job/Hive-trunk-hadoop2/54/])
HIVE-2540 LATERAL VIEW with EXPLODE produces ConcurrentModificationException
(Navis via namit) (Revision 1343036)
Result = ABORTED
namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1343036
Files :
* /hive/trunk/ql/src/test/queries/clientpositive/udf_explode.q
* /hive/trunk/ql/src/test/queries/clientpositive/udtf_explode.q
* /hive/trunk/ql/src/test/results/clientpositive/udf_explode.q.out
* /hive/trunk/ql/src/test/results/clientpositive/udtf_explode.q.out
* /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyArray.java
* /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyMap.java
LATERAL VIEW with EXPLODE produces ConcurrentModificationException
--
Key: HIVE-2540
URL: https://issues.apache.org/jira/browse/HIVE-2540
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.7.1, 0.9.0
Reporter: David Phillips
Assignee: Navis
Fix For: 0.10.0
Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-2540.D2805.1.patch,
ASF.LICENSE.NOT.GRANTED--HIVE-2540.D2805.2.patch,
ASF.LICENSE.NOT.GRANTED--HIVE-2540.D2805.3.patch
The following produces {{ConcurrentModificationException}} on the {{for}}
loop inside EXPLODE:
{code}
create table foo as select array(1, 2) a from src limit 1;
select a, x.b from foo lateral view explode(a) x as b;
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2540) LATERAL VIEW with EXPLODE produces ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HIVE-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478509#comment-13478509
]
Navis commented on HIVE-2540:
-
It's not on hive-0.9.x. Would be included hive-0.10.0 maybe?
LATERAL VIEW with EXPLODE produces ConcurrentModificationException
--
Key: HIVE-2540
URL: https://issues.apache.org/jira/browse/HIVE-2540
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.7.1, 0.9.0
Reporter: David Phillips
Assignee: Navis
Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-2540.D2805.1.patch,
ASF.LICENSE.NOT.GRANTED--HIVE-2540.D2805.2.patch,
ASF.LICENSE.NOT.GRANTED--HIVE-2540.D2805.3.patch
The following produces {{ConcurrentModificationException}} on the {{for}}
loop inside EXPLODE:
{code}
create table foo as select array(1, 2) a from src limit 1;
select a, x.b from foo lateral view explode(a) x as b;
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2540) LATERAL VIEW with EXPLODE produces ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HIVE-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13284255#comment-13284255
]
Hudson commented on HIVE-2540:
--
Integrated in Hive-trunk-h0.21 #1453 (See
[https://builds.apache.org/job/Hive-trunk-h0.21/1453/])
HIVE-2540 LATERAL VIEW with EXPLODE produces ConcurrentModificationException
(Navis via namit) (Revision 1343036)
Result = SUCCESS
namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1343036
Files :
* /hive/trunk/ql/src/test/queries/clientpositive/udf_explode.q
* /hive/trunk/ql/src/test/queries/clientpositive/udtf_explode.q
* /hive/trunk/ql/src/test/results/clientpositive/udf_explode.q.out
* /hive/trunk/ql/src/test/results/clientpositive/udtf_explode.q.out
* /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyArray.java
* /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyMap.java
LATERAL VIEW with EXPLODE produces ConcurrentModificationException
--
Key: HIVE-2540
URL: https://issues.apache.org/jira/browse/HIVE-2540
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.7.1, 0.9.0
Reporter: David Phillips
Assignee: Navis
Attachments: HIVE-2540.D2805.1.patch, HIVE-2540.D2805.2.patch,
HIVE-2540.D2805.3.patch
The following produces {{ConcurrentModificationException}} on the {{for}}
loop inside EXPLODE:
{code}
create table foo as select array(1, 2) a from src limit 1;
select a, x.b from foo lateral view explode(a) x as b;
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2540) LATERAL VIEW with EXPLODE produces ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HIVE-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13284106#comment-13284106
]
Namit Jain commented on HIVE-2540:
--
I will try again.
LATERAL VIEW with EXPLODE produces ConcurrentModificationException
--
Key: HIVE-2540
URL: https://issues.apache.org/jira/browse/HIVE-2540
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.7.1, 0.9.0
Reporter: David Phillips
Assignee: Navis
Attachments: HIVE-2540.D2805.1.patch, HIVE-2540.D2805.2.patch,
HIVE-2540.D2805.3.patch
The following produces {{ConcurrentModificationException}} on the {{for}}
loop inside EXPLODE:
{code}
create table foo as select array(1, 2) a from src limit 1;
select a, x.b from foo lateral view explode(a) x as b;
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2540) LATERAL VIEW with EXPLODE produces ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HIVE-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281423#comment-13281423
]
Namit Jain commented on HIVE-2540:
--
The test that you added is failing.
LATERAL VIEW with EXPLODE produces ConcurrentModificationException
--
Key: HIVE-2540
URL: https://issues.apache.org/jira/browse/HIVE-2540
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.7.1, 0.9.0
Reporter: David Phillips
Assignee: Navis
Attachments: HIVE-2540.D2805.1.patch, HIVE-2540.D2805.2.patch,
HIVE-2540.D2805.3.patch
The following produces {{ConcurrentModificationException}} on the {{for}}
loop inside EXPLODE:
{code}
create table foo as select array(1, 2) a from src limit 1;
select a, x.b from foo lateral view explode(a) x as b;
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2540) LATERAL VIEW with EXPLODE produces ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HIVE-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13281543#comment-13281543
]
Navis commented on HIVE-2540:
-
It passes in my environment. I'll test again after rebasing on trunk.
LATERAL VIEW with EXPLODE produces ConcurrentModificationException
--
Key: HIVE-2540
URL: https://issues.apache.org/jira/browse/HIVE-2540
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.7.1, 0.9.0
Reporter: David Phillips
Assignee: Navis
Attachments: HIVE-2540.D2805.1.patch, HIVE-2540.D2805.2.patch,
HIVE-2540.D2805.3.patch
The following produces {{ConcurrentModificationException}} on the {{for}}
loop inside EXPLODE:
{code}
create table foo as select array(1, 2) a from src limit 1;
select a, x.b from foo lateral view explode(a) x as b;
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2540) LATERAL VIEW with EXPLODE produces ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HIVE-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253561#comment-13253561
]
Kamil Mroczek commented on HIVE-2540:
-
I am getting the same error. It seems to work when the array has one element,
but as soon as more than one is added this error appears.
LATERAL VIEW with EXPLODE produces ConcurrentModificationException
--
Key: HIVE-2540
URL: https://issues.apache.org/jira/browse/HIVE-2540
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.7.1
Reporter: David Phillips
The following produces {{ConcurrentModificationException}} on the {{for}}
loop inside EXPLODE:
{code}
create table foo as select array(1, 2) a from src limit 1;
select a, x.b from foo lateral view explode(a) x as b;
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2540) LATERAL VIEW with EXPLODE produces ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HIVE-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13145585#comment-13145585
]
Bejoy KS commented on HIVE-2540:
The exception is happening when I'm trying to use explode map as well
DDL
CREATE EXTERNAL TABLE ver_data
(
key STRING,
id STRING,
date_time STRING,
model STRING,
version STRING,
sid STRING,
message STRING,
tags MAPSTRING,STRING
);
HQL Query
SELECT ver_data.*,t1.* FROM ver_data LATERAL VIEW explode(tags) t1 AS
param_name,param_value;
This Query throws a concurrent modification exception on a LinkedHashMap while
doing the join. If I modify the query and avoids the map column('tags') in the
retrieval/join,then the query executes successfully.
The below modified query executes successfully
SELECT ver_data.key, ver_data.id, ver_data.date_time, ver_data.model,
ver_data.version, ver_data.sid, ver_data.message,
t1.* FROM ver_data LATERAL VIEW explode(tags) t1 AS param_name,param_value;
Stack Trace When using Joins on maps with Lateral View
2011-11-07 07:18:22,679 FATAL ExecMapper:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row {key:,id:10001,...}
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:546)
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.util.ConcurrentModificationException
at
java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:392)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:391)
LATERAL VIEW with EXPLODE produces ConcurrentModificationException
--
Key: HIVE-2540
URL: https://issues.apache.org/jira/browse/HIVE-2540
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.7.1
Reporter: David Phillips
The following produces {{ConcurrentModificationException}} on the {{for}}
loop inside EXPLODE:
{code}
create table foo as select array(1, 2) a from src limit 1;
select a, x.b from foo lateral view explode(a) x as b;
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2540) LATERAL VIEW with EXPLODE produces ConcurrentModificationException
[
https://issues.apache.org/jira/browse/HIVE-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13145602#comment-13145602
]
Bejoy KS commented on HIVE-2540:
The map issue on explode was seen on a latest build from hive trunk.
Maps were supported with LATERAL VIEW explode() with JIRA HIVE-1735 on 0.8
branch.
LATERAL VIEW with EXPLODE produces ConcurrentModificationException
--
Key: HIVE-2540
URL: https://issues.apache.org/jira/browse/HIVE-2540
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.7.1
Reporter: David Phillips
The following produces {{ConcurrentModificationException}} on the {{for}}
loop inside EXPLODE:
{code}
create table foo as select array(1, 2) a from src limit 1;
select a, x.b from foo lateral view explode(a) x as b;
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
