[jira] [Commented] (HIVE-16675) Fix ConcurrentModificationException in SparkClientImpl#startDriver
[ https://issues.apache.org/jira/browse/HIVE-16675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024354#comment-16024354 ] liyunzhang_intel commented on HIVE-16675: - [~Ferd] and [~stakiar]: thanks for review > Fix ConcurrentModificationException in SparkClientImpl#startDriver > -- > > Key: HIVE-16675 > URL: https://issues.apache.org/jira/browse/HIVE-16675 > Project: Hive > Issue Type: Bug >Reporter: liyunzhang_intel >Assignee: liyunzhang_intel > Fix For: 3.0.0 > > Attachments: HIVE-16675.1.patch, HIVE-16675.patch > > > the exception is > {noformat} > 2017-05-16T00:29:37,480 WARN [Driver] client.SparkClientImpl: > Exception while waiting for child process. > 3926 java.util.ConcurrentModificationException > 3927 at > java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) > ~[?:1.8.0_91] > 3928 at > java.util.ArrayList$Itr.next(ArrayList.java:851) ~[?:1.8.0_91] > 3929 at > org.apache.hive.spark.client.SparkClientImpl$3.run(SparkClientImpl.java:495) > [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] > 3930 at java.lang.Thread.run(Thread.java:745) > [?:1.8.0_91] > {noformat} > It seems that {{SparkClientImpl.java#childErrorLog}} is read while it is > written. It is better to change {{SparkClientImpl.java#childErrorLog}} from > ArrayList to CopyOnWriteArrayList to avoid the exception. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16675) Fix ConcurrentModificationException in SparkClientImpl#startDriver
[ https://issues.apache.org/jira/browse/HIVE-16675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022265#comment-16022265 ] Sahil Takiar commented on HIVE-16675: - +1 LGTM > Fix ConcurrentModificationException in SparkClientImpl#startDriver > -- > > Key: HIVE-16675 > URL: https://issues.apache.org/jira/browse/HIVE-16675 > Project: Hive > Issue Type: Bug >Reporter: liyunzhang_intel >Assignee: liyunzhang_intel > Attachments: HIVE-16675.1.patch, HIVE-16675.patch > > > the exception is > {noformat} > 2017-05-16T00:29:37,480 WARN [Driver] client.SparkClientImpl: > Exception while waiting for child process. > 3926 java.util.ConcurrentModificationException > 3927 at > java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) > ~[?:1.8.0_91] > 3928 at > java.util.ArrayList$Itr.next(ArrayList.java:851) ~[?:1.8.0_91] > 3929 at > org.apache.hive.spark.client.SparkClientImpl$3.run(SparkClientImpl.java:495) > [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] > 3930 at java.lang.Thread.run(Thread.java:745) > [?:1.8.0_91] > {noformat} > It seems that {{SparkClientImpl.java#childErrorLog}} is read while it is > written. It is better to change {{SparkClientImpl.java#childErrorLog}} from > ArrayList to CopyOnWriteArrayList to avoid the exception. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16675) Fix ConcurrentModificationException in SparkClientImpl#startDriver
[ https://issues.apache.org/jira/browse/HIVE-16675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16022230#comment-16022230 ] Ferdinand Xu commented on HIVE-16675: - [~stakiar], do you have further comments? > Fix ConcurrentModificationException in SparkClientImpl#startDriver > -- > > Key: HIVE-16675 > URL: https://issues.apache.org/jira/browse/HIVE-16675 > Project: Hive > Issue Type: Bug >Reporter: liyunzhang_intel >Assignee: liyunzhang_intel > Attachments: HIVE-16675.1.patch, HIVE-16675.patch > > > the exception is > {noformat} > 2017-05-16T00:29:37,480 WARN [Driver] client.SparkClientImpl: > Exception while waiting for child process. > 3926 java.util.ConcurrentModificationException > 3927 at > java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) > ~[?:1.8.0_91] > 3928 at > java.util.ArrayList$Itr.next(ArrayList.java:851) ~[?:1.8.0_91] > 3929 at > org.apache.hive.spark.client.SparkClientImpl$3.run(SparkClientImpl.java:495) > [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] > 3930 at java.lang.Thread.run(Thread.java:745) > [?:1.8.0_91] > {noformat} > It seems that {{SparkClientImpl.java#childErrorLog}} is read while it is > written. It is better to change {{SparkClientImpl.java#childErrorLog}} from > ArrayList to CopyOnWriteArrayList to avoid the exception. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16675) Fix ConcurrentModificationException in SparkClientImpl#startDriver
[ https://issues.apache.org/jira/browse/HIVE-16675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1604#comment-1604 ] liyunzhang_intel commented on HIVE-16675: - [~Ferd]: help review HIVE-16675.1.patch. thanks. > Fix ConcurrentModificationException in SparkClientImpl#startDriver > -- > > Key: HIVE-16675 > URL: https://issues.apache.org/jira/browse/HIVE-16675 > Project: Hive > Issue Type: Bug >Reporter: liyunzhang_intel >Assignee: liyunzhang_intel > Attachments: HIVE-16675.1.patch, HIVE-16675.patch > > > the exception is > {noformat} > 2017-05-16T00:29:37,480 WARN [Driver] client.SparkClientImpl: > Exception while waiting for child process. > 3926 java.util.ConcurrentModificationException > 3927 at > java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) > ~[?:1.8.0_91] > 3928 at > java.util.ArrayList$Itr.next(ArrayList.java:851) ~[?:1.8.0_91] > 3929 at > org.apache.hive.spark.client.SparkClientImpl$3.run(SparkClientImpl.java:495) > [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] > 3930 at java.lang.Thread.run(Thread.java:745) > [?:1.8.0_91] > {noformat} > It seems that {{SparkClientImpl.java#childErrorLog}} is read while it is > written. It is better to change {{SparkClientImpl.java#childErrorLog}} from > ArrayList to CopyOnWriteArrayList to avoid the exception. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16675) Fix ConcurrentModificationException in SparkClientImpl#startDriver
[ https://issues.apache.org/jira/browse/HIVE-16675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019267#comment-16019267 ] Hive QA commented on HIVE-16675: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12869204/HIVE-16675.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10742 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite] (batchId=236) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partition_wise_fileformat6] (batchId=7) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr] (batchId=144) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5378/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5378/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5378/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12869204 - PreCommit-HIVE-Build > Fix ConcurrentModificationException in SparkClientImpl#startDriver > -- > > Key: HIVE-16675 > URL: https://issues.apache.org/jira/browse/HIVE-16675 > Project: Hive > Issue Type: Bug >Reporter: liyunzhang_intel >Assignee: liyunzhang_intel > Attachments: HIVE-16675.1.patch, HIVE-16675.patch > > > the exception is > {noformat} > 2017-05-16T00:29:37,480 WARN [Driver] client.SparkClientImpl: > Exception while waiting for child process. > 3926 java.util.ConcurrentModificationException > 3927 at > java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) > ~[?:1.8.0_91] > 3928 at > java.util.ArrayList$Itr.next(ArrayList.java:851) ~[?:1.8.0_91] > 3929 at > org.apache.hive.spark.client.SparkClientImpl$3.run(SparkClientImpl.java:495) > [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] > 3930 at java.lang.Thread.run(Thread.java:745) > [?:1.8.0_91] > {noformat} > It seems that {{SparkClientImpl.java#childErrorLog}} is read while it is > written. It is better to change {{SparkClientImpl.java#childErrorLog}} from > ArrayList to CopyOnWriteArrayList to avoid the exception. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16675) Fix ConcurrentModificationException in SparkClientImpl#startDriver
[ https://issues.apache.org/jira/browse/HIVE-16675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16016407#comment-16016407 ] Sahil Takiar commented on HIVE-16675: - Why use a {{CopyOnWriteArrayList}}? {{childErrorLog}} is going to be mutated by the {{Redirector}} object up to 1000 times ({{MAX_ERR_LOG_LINES_FOR_RPC)}}. {{CopyOnWriteArrayList}} will create a new list for each mutation, so that means 1000 lists will be created, which seems inefficient. Why not use {{Collections. synchronizedList}} or even {{LinkedBlockingQueue}}. > Fix ConcurrentModificationException in SparkClientImpl#startDriver > -- > > Key: HIVE-16675 > URL: https://issues.apache.org/jira/browse/HIVE-16675 > Project: Hive > Issue Type: Bug >Reporter: liyunzhang_intel >Assignee: liyunzhang_intel > Attachments: HIVE-16675.patch > > > the exception is > {noformat} > 2017-05-16T00:29:37,480 WARN [Driver] client.SparkClientImpl: > Exception while waiting for child process. > 3926 java.util.ConcurrentModificationException > 3927 at > java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) > ~[?:1.8.0_91] > 3928 at > java.util.ArrayList$Itr.next(ArrayList.java:851) ~[?:1.8.0_91] > 3929 at > org.apache.hive.spark.client.SparkClientImpl$3.run(SparkClientImpl.java:495) > [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] > 3930 at java.lang.Thread.run(Thread.java:745) > [?:1.8.0_91] > {noformat} > It seems that {{SparkClientImpl.java#childErrorLog}} is read while it is > written. It is better to change {{SparkClientImpl.java#childErrorLog}} from > ArrayList to CopyOnWriteArrayList to avoid the exception. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16675) Fix ConcurrentModificationException in SparkClientImpl#startDriver
[ https://issues.apache.org/jira/browse/HIVE-16675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16015251#comment-16015251 ] Ferdinand Xu commented on HIVE-16675: - LGTM +1 > Fix ConcurrentModificationException in SparkClientImpl#startDriver > -- > > Key: HIVE-16675 > URL: https://issues.apache.org/jira/browse/HIVE-16675 > Project: Hive > Issue Type: Bug >Reporter: liyunzhang_intel >Assignee: liyunzhang_intel > Attachments: HIVE-16675.patch > > > the exception is > {noformat} > 2017-05-16T00:29:37,480 WARN [Driver] client.SparkClientImpl: > Exception while waiting for child process. > 3926 java.util.ConcurrentModificationException > 3927 at > java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) > ~[?:1.8.0_91] > 3928 at > java.util.ArrayList$Itr.next(ArrayList.java:851) ~[?:1.8.0_91] > 3929 at > org.apache.hive.spark.client.SparkClientImpl$3.run(SparkClientImpl.java:495) > [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] > 3930 at java.lang.Thread.run(Thread.java:745) > [?:1.8.0_91] > {noformat} > It seems that {{SparkClientImpl.java#childErrorLog}} is read while it is > written. It is better to change {{SparkClientImpl.java#childErrorLog}} from > ArrayList to CopyOnWriteArrayList to avoid the exception. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16675) Fix ConcurrentModificationException in SparkClientImpl#startDriver
[ https://issues.apache.org/jira/browse/HIVE-16675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16013348#comment-16013348 ] liyunzhang_intel commented on HIVE-16675: - [~Ferd]: please help review. I guess that the failled 3 tests are not related with HIVE-16675.patch > Fix ConcurrentModificationException in SparkClientImpl#startDriver > -- > > Key: HIVE-16675 > URL: https://issues.apache.org/jira/browse/HIVE-16675 > Project: Hive > Issue Type: Bug >Reporter: liyunzhang_intel >Assignee: liyunzhang_intel > Attachments: HIVE-16675.patch > > > the exception is > {noformat} > 2017-05-16T00:29:37,480 WARN [Driver] client.SparkClientImpl: > Exception while waiting for child process. > 3926 java.util.ConcurrentModificationException > 3927 at > java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) > ~[?:1.8.0_91] > 3928 at > java.util.ArrayList$Itr.next(ArrayList.java:851) ~[?:1.8.0_91] > 3929 at > org.apache.hive.spark.client.SparkClientImpl$3.run(SparkClientImpl.java:495) > [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] > 3930 at java.lang.Thread.run(Thread.java:745) > [?:1.8.0_91] > {noformat} > It seems that {{SparkClientImpl.java#childErrorLog}} is read while it is > written. It is better to change {{SparkClientImpl.java#childErrorLog}} from > ArrayList to CopyOnWriteArrayList to avoid the exception. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16675) Fix ConcurrentModificationException in SparkClientImpl#startDriver
[ https://issues.apache.org/jira/browse/HIVE-16675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16012104#comment-16012104 ] Hive QA commented on HIVE-16675: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12868255/HIVE-16675.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10717 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite] (batchId=236) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[table_nonprintable] (batchId=140) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join30] (batchId=149) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5279/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5279/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5279/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12868255 - PreCommit-HIVE-Build > Fix ConcurrentModificationException in SparkClientImpl#startDriver > -- > > Key: HIVE-16675 > URL: https://issues.apache.org/jira/browse/HIVE-16675 > Project: Hive > Issue Type: Bug >Reporter: liyunzhang_intel >Assignee: liyunzhang_intel > Attachments: HIVE-16675.patch > > > the exception is > {noformat} > 2017-05-16T00:29:37,480 WARN [Driver] client.SparkClientImpl: > Exception while waiting for child process. > 3926 java.util.ConcurrentModificationException > 3927 at > java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) > ~[?:1.8.0_91] > 3928 at > java.util.ArrayList$Itr.next(ArrayList.java:851) ~[?:1.8.0_91] > 3929 at > org.apache.hive.spark.client.SparkClientImpl$3.run(SparkClientImpl.java:495) > [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT] > 3930 at java.lang.Thread.run(Thread.java:745) > [?:1.8.0_91] > {noformat} > It seems that {{SparkClientImpl.java#childErrorLog}} is read while it is > written. It is better to change {{SparkClientImpl.java#childErrorLog}} from > ArrayList to CopyOnWriteArrayList to avoid the exception. -- This message was sent by Atlassian JIRA (v6.3.15#6346)