[
https://issues.apache.org/jira/browse/FLINK-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Omar Alvarez closed FLINK-3616.
-------------------------------
Resolution: Implemented
> Python API version inconsistencies
> ----------------------------------
>
> Key: FLINK-3616
> URL: https://issues.apache.org/jira/browse/FLINK-3616
> Project: Flink
> Issue Type: Bug
> Components: Python API
> Affects Versions: 0.10.0
> Environment: Rocks 6.1 SP1, CentOS release 6.7
> (2.6.32-573.7.1.el6.x86_64), java/oraclejdk/1.8.0_45, Python 2.7.9
> Reporter: Omar Alvarez
> Labels: Python
>
> I'm trying to run a python script that uses several packages present in my
> python 2.7.9 installation. The script is failing with an import error due to
> discrepacies in the python versions that the flink environment uses. I have
> two python instalations 2.6 and 2.7.
> When I run my Flink python script everything goes fine, it uses python 2.7,
> until the taskmanager starts executing the job. The problem is that inside
> the job the python version used is the 2.6 ignoring the systemwide config
> that points to python 2.7.
> Is there anyway to tell flink where to find the desired python version
> executable?
> This is the error:
> {code:title=PyExample|borderStyle=solid}
> Trajectory loading complete!
> 03/15/2016 21:17:17 Job execution switched to status RUNNING.
> 03/15/2016 21:17:17 DataSource (ValueSource)(1/1) switched to SCHEDULED
> 03/15/2016 21:17:17 DataSource (ValueSource)(1/1) switched to DEPLOYING
> 03/15/2016 21:17:17 DataSource (ValueSource)(1/1) switched to RUNNING
> 03/15/2016 21:17:17 MapPartition (PythonMap)(1/1) switched to SCHEDULED
> 03/15/2016 21:17:17 MapPartition (PythonMap)(1/1) switched to DEPLOYING
> 03/15/2016 21:17:17 DataSource (ValueSource)(1/1) switched to FINISHED
> 03/15/2016 21:17:17 MapPartition (PythonMap)(1/1) switched to RUNNING
> 03/15/2016 21:17:19 MapPartition (PythonMap)(1/1) switched to FAILED
> java.lang.Exception: The user defined 'open()' method caused an exception:
> External process for task MapPartition (PythonMap) terminated prematurely.
> Traceback (most recent call last):
> File
> "/tmp/flink-dist-cache-f108fb65-b75b-4b6a-bffd-4b6c5e3a3229/a2016bdf5ba366bbcbd68a6cc1ac537b/flink/plan.py",
> line 9, in <module>
> import MDAnalysis
> ImportError: No module named MDAnalysis
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:484)
> at
> org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:354)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: External process for task MapPartition
> (PythonMap) terminated prematurely.
> Traceback (most recent call last):
> File
> "/tmp/flink-dist-cache-f108fb65-b75b-4b6a-bffd-4b6c5e3a3229/a2016bdf5ba366bbcbd68a6cc1ac537b/flink/plan.py",
> line 9, in <module>
> import MDAnalysis
> ImportError: No module named MDAnalysis
> at
> org.apache.flink.languagebinding.api.java.python.streaming.PythonStreamer.startPython(PythonStreamer.java:116)
> at
> org.apache.flink.languagebinding.api.java.python.streaming.PythonStreamer.setupProcess(PythonStreamer.java:58)
> at
> org.apache.flink.languagebinding.api.java.common.streaming.Streamer.open(Streamer.java:67)
> at
> org.apache.flink.languagebinding.api.java.python.functions.PythonMapPartition.open(PythonMapPartition.java:47)
> at
> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:480)
> ... 3 more
> 03/15/2016 21:17:19 Job execution switched to status FAILING.
> 03/15/2016 21:17:19 DataSink (TextSink)(1/1) switched to CANCELED
> 03/15/2016 21:17:19 Job execution switched to status FAILED.
> org.apache.flink.client.program.ProgramInvocationException: The program
> execution failed: Job execution failed.
> at org.apache.flink.client.program.Client.runBlocking(Client.java:370)
> at org.apache.flink.client.program.Client.runBlocking(Client.java:348)
> at org.apache.flink.client.program.Client.runBlocking(Client.java:315)
> at
> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:70)
> at
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:804)
> at
> org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.runPlan(PythonPlanBinder.java:117)
> at
> org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.main(PythonPlanBinder.java:85)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:497)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:395)
> at org.apache.flink.client.program.Client.runBlocking(Client.java:252)
> at
> org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:675)
> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:326)
> at
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:977)
> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1027)
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job
> execution failed.
> at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$5.apply$mcV$sp(JobManager.scala:563)
> at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$5.apply(JobManager.scala:509)
> at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$5.apply(JobManager.scala:509)
> at
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
> at
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: java.lang.Exception: The user defined 'open()' method caused an
> exception: External process for task MapPartition (PythonMap) terminated
> prematurely.
> Traceback (most recent call last):
> File
> "/tmp/flink-dist-cache-f108fb65-b75b-4b6a-bffd-4b6c5e3a3229/a2016bdf5ba366bbcbd68a6cc1ac537b/flink/plan.py",
> line 9, in <module>
> import MDAnalysis
> ImportError: No module named MDAnalysis
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:484)
> at
> org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:354)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: External process for task MapPartition
> (PythonMap) terminated prematurely.
> Traceback (most recent call last):
> File
> "/tmp/flink-dist-cache-f108fb65-b75b-4b6a-bffd-4b6c5e3a3229/a2016bdf5ba366bbcbd68a6cc1ac537b/flink/plan.py",
> line 9, in <module>
> import MDAnalysis
> ImportError: No module named MDAnalysis
> at
> org.apache.flink.languagebinding.api.java.python.streaming.PythonStreamer.startPython(PythonStreamer.java:116)
> at
> org.apache.flink.languagebinding.api.java.python.streaming.PythonStreamer.setupProcess(PythonStreamer.java:58)
> at
> org.apache.flink.languagebinding.api.java.common.streaming.Streamer.open(Streamer.java:67)
> at
> org.apache.flink.languagebinding.api.java.python.functions.PythonMapPartition.open(PythonMapPartition.java:47)
> at
> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:480)
> ... 3 more
> The exception above occurred while trying to run your command.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
