Koji Noguchi created PIG-5415:
---------------------------------
Summary: [spark] TestScriptLanguage conflict between multiple
SparkContext (after spark2.4 upgrade)
Key: PIG-5415
URL: https://issues.apache.org/jira/browse/PIG-5415
Project: Pig
Issue Type: Bug
Components: spark
Reporter: Koji Noguchi
Assignee: Koji Noguchi
{noformat}
2021-10-12 17:54:40,073 [main] ERROR org.apache.pig.scripting.BoundScript - Pig
pipeline failed to complete
java.util.concurrent.ExecutionException:
org.apache.pig.backend.executionengine.ExecException: ERROR 0:
java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.
This stopped SparkContext was created at:
org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.startSparkIfNeeded(SparkLauncher.java:640)
org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.launchPig(SparkLauncher.java:184)
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:290)
org.apache.pig.PigServer.launchPlan(PigServer.java:1479)
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1464)
org.apache.pig.PigServer.execute(PigServer.java:1453)
org.apache.pig.PigServer.executeBatch(PigServer.java:489)
org.apache.pig.PigServer.executeBatch(PigServer.java:472)
org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:172)
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:235)
org.apache.pig.scripting.BoundScript$MyCallable.call(BoundScript.java:347)
org.apache.pig.scripting.BoundScript$MyCallable.call(BoundScript.java:323)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
The currently active SparkContext was created at:
org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.startSparkIfNeeded(SparkLauncher.java:640)
org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.launchPig(SparkLauncher.java:184)
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:290)
org.apache.pig.PigServer.launchPlan(PigServer.java:1479)
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1464)
org.apache.pig.PigServer.execute(PigServer.java:1453)
org.apache.pig.PigServer.executeBatch(PigServer.java:489)
org.apache.pig.PigServer.executeBatch(PigServer.java:472)
org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:172)
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:235)
org.apache.pig.scripting.BoundScript$MyCallable.call(BoundScript.java:347)
org.apache.pig.scripting.BoundScript$MyCallable.call(BoundScript.java:323)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
