Re: Problem running a pipeline on a remote Flink cluster

Mon, 13 May 2019 03:37:50 -0700 

Hi Jorik,
It looks like the version of the Flink Runner and the Flink cluster version do not match. For example, if you use Flink 1.7, make sure to use the beam-runners-flink-1.7 artifact. 

For more information: https://beam.apache.org/documentation/runners/flink/

Thanks,
Max

On 12.05.19 18:48, Jorik van Kemenade wrote:

Hi guys,


I am running into a problem when trying to upload my Beam job to a flink 
cluster.



However, when testing the pipeline using the local cluster there is no 
problem.


So for extra clarification, I have no problem running:


mvn exec:java -D 
exec.mainClass=com.asml.wfa.swinfra.dataflow.DataFlowModeling -D exec.args=



"--runner=FlinkRunner 
--filesToStage=C:\dev\beam-experiments\target\dataflow-bundled-0.1.jar



--inputDirectory=C:\Localdata\LIS\local-lis\input-modeling 
--outputDirectory=C:\Localdata\LIS\local-lis\output


--tempDirectory=C:\Localdata\LIS\local-lis\temp" -P flink-runner

However as soon as I add a flink master I run into problems:


mvn exec:java -D 
exec.mainClass=com.asml.wfa.swinfra.dataflow.DataFlowModeling -D exec.args=



"--runner=FlinkRunner 
--filesToStage=C:\dev\beam-experiments\target\dataflow-bundled-0.1.jar



--inputDirectory=C:\Localdata\LIS\local-lis\input-modeling 
--outputDirectory=C:\Localdata\LIS\local-lis\output



--tempDirectory=C:\Localdata\LIS\local-lis\temp 
--flinkMaster=localhost:8081" -P flink-runner


The error I get is the following:

Pipeline execution failed: Could not retrieve the execution result. (JobID


: 0aa23a9193720ecf0bb7442a1ce8037a): Failed to submit JobGraph. 
[Internal server error., <Exception on server side:


[ERROR] java.lang.StackOverflowError

[ERROR]         at java.io.FileInputStream.read(Unknown Source)


[ERROR]         at 
org.apache.flink.core.fs.local.LocalDataInputStream.read(LocalDataInputStream.java:73)



[ERROR]         at 
java.io.ObjectInputStream$PeekInputStream.read(Unknown Source)



[ERROR]         at 
java.io.ObjectInputStream$PeekInputStream.readFully(Unknown Source)



[ERROR]         at 
java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(Unknown Source)



[ERROR]         at 
java.io.ObjectInputStream$BlockDataInputStream.readUTF(Unknown Source)


[ERROR]         at java.io.ObjectInputStream.readString(Unknown Source)

[ERROR]         at java.io.ObjectInputStream.readObject0(Unknown Source)

[ERROR]         at java.io.ObjectInputStream.readObject(Unknown Source)

[ERROR]         at java.util.HashMap.readObject(Unknown Source)


[ERROR]         at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown 
Source)



[ERROR]         at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)


[ERROR]         at java.lang.reflect.Method.invoke(Unknown Source)


[ERROR]         at java.io.ObjectStreamClass.invokeReadObject(Unknown 
Source)


[ERROR]         at java.io.ObjectInputStream.readSerialData(Unknown Source)


[ERROR]         at java.io.ObjectInputStream.readOrdinaryObject(Unknown 
Source)


[ERROR]         at java.io.ObjectInputStream.readObject0(Unknown Source)


[ERROR]         at java.io.ObjectInputStream.defaultReadFields(Unknown 
Source)


[ERROR]         at java.io.ObjectInputStream.readSerialData(Unknown Source)


[ERROR]         at java.io.ObjectInputStream.readOrdinaryObject(Unknown 
Source)


[ERROR]         at java.io.ObjectInputStream.readObject0(Unknown Source)


[ERROR]         at java.io.ObjectInputStream.defaultReadFields(Unknown 
Source)


[ERROR]         at java.io.ObjectInputStream.readSerialData(Unknown Source)


[ERROR]         at java.io.ObjectInputStream.readOrdinaryObject(Unknown 
Source)


[ERROR]         at java.io.ObjectInputStream.readObject0(Unknown Source)


[ERROR]         at java.io.ObjectInputStream.defaultReadFields(Unknown 
Source)


[ERROR]         at java.io.ObjectInputStream.readSerialData(Unknown Source)


[ERROR]         at java.io.ObjectInputStream.readOrdinaryObject(Unknown 
Source)


[ERROR]         at java.io.ObjectInputStream.readObject0(Unknown Source)


[ERROR]         at java.io.ObjectInputStream.defaultReadFields(Unknown 
Source)


[ERROR]         at java.io.ObjectInputStream.readSerialData(Unknown Source)


[ERROR]         at java.io.ObjectInputStream.readOrdinaryObject(Unknown 
Source)


[ERROR]         at java.io.ObjectInputStream.readObject0(Unknown Source)

[ERROR]         at java.io.ObjectInputStream.readObject(Unknown Source)

[ERROR]         at java.util.ArrayList.readObject(Unknown Source)


[ERROR]         at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown 
Source)


(…)


[ERROR]         at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown 
Source)



[ERROR]         at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)


[ERROR]         at java.lang.reflect.Method.invoke(Unknown Source)


[ERROR]         at java.io.ObjectStreamClass.invokeReadObject(Unknown 
Source)


[ERROR]         at java.io.ObjectInputStream.readSerialData(Unknown Source)


[ERROR]         at java.io.ObjectInputStream.readOrdinaryObject(Unknown 
Source)


[ERROR]         at java.io.ObjectInputStream.readObject0(Unknown Source)


[ERROR]         at java.io.ObjectInputStream.defaultReadFields(Unknown 
Source)


[ERROR]         at java.io.ObjectInputStream.readSerialData(Unknown Source)


[ERROR]         at java.io.ObjectInputStream.readOrdinaryObject(Unknown 
Source)


[ERROR]         at java.io.ObjectInputStream.readObject0(Unknown Source)

I looked into the logs of the JobManager for extra information.


But unfortunately there is not much I can find, the startup of the 
JobManager is normal and the first reported error is the 
StackOverFlowError listed above.



It is possible to run the WordCount examples both with and without the 
  “--flinkMaster” argument.



Has anybody ran into this error earlier, or does anybody have an idea 
where I can continue my investigation?


Thanks for your help.

Regards,

Jorik van Kemenade


-- The information contained in this communication and any attachments 
is confidential and may be privileged, and is for the sole use of the 
intended recipient(s). Any unauthorized review, use, disclosure or 
distribution is prohibited. Unless explicitly stated otherwise in the 
body of this communication or the attachment thereto (if any), the 
information is provided on an AS-IS basis without any express or implied 
warranties or liabilities. To the extent you are relying on this 
information, you are doing so at your own risk. If you are not the 
intended recipient, please notify the sender immediately by replying to 
this message and destroy all copies of this message and any attachments. 
Neither the sender nor the company/group of companies he or she 
represents shall be liable for the proper and complete transmission of 
the information contained in this communication, or for any delay in its 
receipt.






















					Reply via email to