Hi Max, Thanks for the quick reply. I am running on Flink 1.7.2 with Beam 1.12 and using beam-runners-flink-1.7. This should be compatibly as far as I am aware, and when I tested it for the WordCount example it ran without problems. Also, it runs successfully if I run on [local], if I am using incompatible versions this should fail right?
Regards, Jorik -----Original Message----- From: Maximilian Michels <[email protected]> Sent: Monday, May 13, 2019 12:37 PM To: [email protected] Cc: Jorik van Kemenade <[email protected]> Subject: Re: Problem running a pipeline on a remote Flink cluster CAUTION: This message is from an external sender Hi Jorik, It looks like the version of the Flink Runner and the Flink cluster version do not match. For example, if you use Flink 1.7, make sure to use the beam-runners-flink-1.7 artifact. For more information: https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbeam.apache.org%2Fdocumentation%2Frunners%2Fflink%2F&data=01%7C01%7Cjorik.van.kemenade%40asml.com%7Cef4c991af5f04a00d38808d6d78eeaf3%7Caf73baa8f5944eb2a39d93e96cad61fc%7C1&sdata=OtyN7Z3zCrESoY3%2FRYXqAyvR14fn2qWz5LCVkBMEYK4%3D&reserved=0 Thanks, Max On 12.05.19 18:48, Jorik van Kemenade wrote: > Hi guys, > > I am running into a problem when trying to upload my Beam job to a > flink cluster. > > However, when testing the pipeline using the local cluster there is no > problem. > > So for extra clarification, I have no problem running: > > mvn exec:java -D > exec.mainClass=com.asml.wfa.swinfra.dataflow.DataFlowModeling -D > exec.args= > > "--runner=FlinkRunner > --filesToStage=C:\dev\beam-experiments\target\dataflow-bundled-0.1.jar > > --inputDirectory=C:\Localdata\LIS\local-lis\input-modeling > --outputDirectory=C:\Localdata\LIS\local-lis\output > > --tempDirectory=C:\Localdata\LIS\local-lis\temp" -P flink-runner > > However as soon as I add a flink master I run into problems: > > mvn exec:java -D > exec.mainClass=com.asml.wfa.swinfra.dataflow.DataFlowModeling -D > exec.args= > > "--runner=FlinkRunner > --filesToStage=C:\dev\beam-experiments\target\dataflow-bundled-0.1.jar > > --inputDirectory=C:\Localdata\LIS\local-lis\input-modeling > --outputDirectory=C:\Localdata\LIS\local-lis\output > > --tempDirectory=C:\Localdata\LIS\local-lis\temp > --flinkMaster=localhost:8081" -P flink-runner > > The error I get is the following: > > Pipeline execution failed: Could not retrieve the execution result. > (JobID > > : 0aa23a9193720ecf0bb7442a1ce8037a): Failed to submit JobGraph. > [Internal server error., <Exception on server side: > > [ERROR] java.lang.StackOverflowError > > [ERROR] at java.io.FileInputStream.read(Unknown Source) > > [ERROR] at > org.apache.flink.core.fs.local.LocalDataInputStream.read(LocalDataInpu > tStream.java:73) > > [ERROR] at > java.io.ObjectInputStream$PeekInputStream.read(Unknown Source) > > [ERROR] at > java.io.ObjectInputStream$PeekInputStream.readFully(Unknown Source) > > [ERROR] at > java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(Unknown > Source) > > [ERROR] at > java.io.ObjectInputStream$BlockDataInputStream.readUTF(Unknown Source) > > [ERROR] at java.io.ObjectInputStream.readString(Unknown > Source) > > [ERROR] at java.io.ObjectInputStream.readObject0(Unknown > Source) > > [ERROR] at java.io.ObjectInputStream.readObject(Unknown > Source) > > [ERROR] at java.util.HashMap.readObject(Unknown Source) > > [ERROR] at > sun.reflect.GeneratedMethodAccessor18.invoke(Unknown > Source) > > [ERROR] at > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > > [ERROR] at java.lang.reflect.Method.invoke(Unknown Source) > > [ERROR] at java.io.ObjectStreamClass.invokeReadObject(Unknown > Source) > > [ERROR] at java.io.ObjectInputStream.readSerialData(Unknown > Source) > > [ERROR] at > java.io.ObjectInputStream.readOrdinaryObject(Unknown > Source) > > [ERROR] at java.io.ObjectInputStream.readObject0(Unknown > Source) > > [ERROR] at java.io.ObjectInputStream.defaultReadFields(Unknown > Source) > > [ERROR] at java.io.ObjectInputStream.readSerialData(Unknown > Source) > > [ERROR] at > java.io.ObjectInputStream.readOrdinaryObject(Unknown > Source) > > [ERROR] at java.io.ObjectInputStream.readObject0(Unknown > Source) > > [ERROR] at java.io.ObjectInputStream.defaultReadFields(Unknown > Source) > > [ERROR] at java.io.ObjectInputStream.readSerialData(Unknown > Source) > > [ERROR] at > java.io.ObjectInputStream.readOrdinaryObject(Unknown > Source) > > [ERROR] at java.io.ObjectInputStream.readObject0(Unknown > Source) > > [ERROR] at java.io.ObjectInputStream.defaultReadFields(Unknown > Source) > > [ERROR] at java.io.ObjectInputStream.readSerialData(Unknown > Source) > > [ERROR] at > java.io.ObjectInputStream.readOrdinaryObject(Unknown > Source) > > [ERROR] at java.io.ObjectInputStream.readObject0(Unknown > Source) > > [ERROR] at java.io.ObjectInputStream.defaultReadFields(Unknown > Source) > > [ERROR] at java.io.ObjectInputStream.readSerialData(Unknown > Source) > > [ERROR] at > java.io.ObjectInputStream.readOrdinaryObject(Unknown > Source) > > [ERROR] at java.io.ObjectInputStream.readObject0(Unknown > Source) > > [ERROR] at java.io.ObjectInputStream.readObject(Unknown > Source) > > [ERROR] at java.util.ArrayList.readObject(Unknown Source) > > [ERROR] at > sun.reflect.GeneratedMethodAccessor22.invoke(Unknown > Source) > > (...) > > [ERROR] at > sun.reflect.GeneratedMethodAccessor22.invoke(Unknown > Source) > > [ERROR] at > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > > [ERROR] at java.lang.reflect.Method.invoke(Unknown Source) > > [ERROR] at java.io.ObjectStreamClass.invokeReadObject(Unknown > Source) > > [ERROR] at java.io.ObjectInputStream.readSerialData(Unknown > Source) > > [ERROR] at > java.io.ObjectInputStream.readOrdinaryObject(Unknown > Source) > > [ERROR] at java.io.ObjectInputStream.readObject0(Unknown > Source) > > [ERROR] at java.io.ObjectInputStream.defaultReadFields(Unknown > Source) > > [ERROR] at java.io.ObjectInputStream.readSerialData(Unknown > Source) > > [ERROR] at > java.io.ObjectInputStream.readOrdinaryObject(Unknown > Source) > > [ERROR] at java.io.ObjectInputStream.readObject0(Unknown > Source) > > I looked into the logs of the JobManager for extra information. > > But unfortunately there is not much I can find, the startup of the > JobManager is normal and the first reported error is the > StackOverFlowError listed above. > > It is possible to run the WordCount examples both with and without the > "--flinkMaster" argument. > > Has anybody ran into this error earlier, or does anybody have an idea > where I can continue my investigation? > > Thanks for your help. > > Regards, > > Jorik van Kemenade > > -- The information contained in this communication and any attachments > is confidential and may be privileged, and is for the sole use of the > intended recipient(s). Any unauthorized review, use, disclosure or > distribution is prohibited. Unless explicitly stated otherwise in the > body of this communication or the attachment thereto (if any), the > information is provided on an AS-IS basis without any express or > implied warranties or liabilities. To the extent you are relying on > this information, you are doing so at your own risk. If you are not > the intended recipient, please notify the sender immediately by > replying to this message and destroy all copies of this message and any > attachments. > Neither the sender nor the company/group of companies he or she > represents shall be liable for the proper and complete transmission of > the information contained in this communication, or for any delay in > its receipt. -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.
