Ah. So maybe try 2.16.0?
On 2019/09/17 23:47:16, Kyle Weaver <[email protected]> wrote: > --experiments=beam_fn_api doesn't apply here, as this is a Java pipeline > using the non-portable version of the Flink runner. > > Kyle Weaver | Software Engineer | github.com/ibzib | [email protected] > > > On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan <[email protected]> > wrote: > > > Could you try adding "--experiments=beam_fn_api" to the runner options? > > Kyle did mention that it should be fixed in 2.16.0. > > > > On 2019/09/17 16:40:41, Matthew Patterson <[email protected]> > > wrote: > > > Folks, > > > > > > Am sure I am doing it wrong, but have been beating head against wall > > long enough I need to ask for help. > > > > > > My goal is to use the python SDK to run jobs on AWS flink cluster, but > > very little is working, so I am trying to back down to the very simplest > > thing that works, and build back up: not going so well. > > > > > > Now I am running, or trying to, on a minikube link cluster. Now running > > from just java/mvn, so I go through the following: > > > > > > 1. bring-up-cluster (see below) > > > 2. grab-the-code (see below) > > > 3. maven-submit (see below) > > > > > > Eventual failure is > > > > > > “[ERROR] Failed to execute goal > > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project > > word-count-beam: An exception occured while executing the Java class. > > Pipeline execution failed: Could not retrieve the execution result. (JobID: > > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph. [Failed to > > deserialize JobGraph.]” > > > > > > and log is attached (I hope) > > > > > > Other details: > > > > > > on osx > > > > > > > > > > > > ➜ minikube version > > > > > > minikube version: v1.3.1 > > > > > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631 > > > > > > > > > ➜ kubectl version > > > > > > Client Version: version.Info{Major:"1", Minor:"14", > > GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", > > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z", > > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"} > > > > > > Server Version: version.Info{Major:"1", Minor:"15", > > GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", > > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", > > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} > > > > > > > > > ➜ flink --version > > > > > > Version: 1.8.1, Commit ID: 7297bac > > > > > > ➜ python -c"import apache_beam; print(apache_beam.__version__)" > > > 2.15.0 > > > > > > ➜ curl localhost:8081 > > > <!-- > > > ~ Licensed to the Apache Software Foundation (ASF) under one > > > ~ or more contributor license agreements. See the NOTICE file > > > ~ distributed with this work for additional information > > > ~ regarding copyright ownership. The ASF licenses this file > > > ~ to you under the Apache License, Version 2.0 (the > > > ~ "License"); you may not use this file except in compliance > > > ~ with the License. You may obtain a copy of the License at > > > ~ > > > ~ http://www.apache.org/licenses/LICENSE-2.0 > > > ~ > > > ~ Unless required by applicable law or agreed to in writing, software > > > ~ distributed under the License is distributed on an "AS IS" BASIS, > > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or > > implied. > > > ~ See the License for the specific language governing permissions and > > > ~ limitations under the License. > > > --> > > > > > > <!doctype html> > > > <html lang="en"> > > > <head> > > > <meta charset="utf-8"> > > > <meta http-equiv="X-UA-Compatible" content="IE=edge"> > > > <meta name="viewport" content="width=device-width, initial-scale=1"> > > > <title>Apache Flink Web Dashboard</title> > > > <link rel="apple-touch-icon" sizes="180x180" > > href="assets/favicon/apple-touch-icon.png"> > > > <link rel="icon" type="image/png" > > href="assets/favicon/favicon-32x32.png" sizes="32x32"> > > > <link rel="icon" type="image/png" > > href="assets/favicon/favicon-16x16.png" sizes="16x16"> > > > <link rel="manifest" href="assets/favicon/manifest.json"> > > > <link rel="mask-icon" href="assets/favicon/safari-pinned-tab.svg" > > color="#aa1919"> > > > <link rel="shortcut icon" href="assets/favicon/favicon.ico"> > > > <meta name="msapplication-config" > > content="assets/favicon/browserconfig.xml"> > > > <meta name="theme-color" content="#ffffff"> > > > <base href="./"><link rel="stylesheet" > > href="styles.30d0912c1ece284d8d9a.css"></head> > > > <body> > > > <flink-root></flink-root> > > > <script type="text/javascript" > > src="runtime.ebbfed06bb8e06aaa889.js"></script><script > > type="text/javascript" src="es2015-polyfills.5e343224e81eefb7658e.js" > > nomodule></script><script type="text/javascript" > > src="polyfills.b37850e8279bc3caafc9.js"></script><script > > type="text/javascript" src="main.dc6e0997d092df051db0.js"></script></body> > > > </html> > > > > > > Thanks for any ideas, > > > Matt > > > > > > >>> bring-up-cluster > > > > > > # [yamls from here]( > > https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/kubernetes.html > > ) > > > minikube start > > > minikube ssh 'sudo ip link set docker0 promisc on' > > > > > > kubectl create -f flink-configuration-configmap.yaml > > > kubectl create -f jobmanager-service.yaml > > > kubectl create -f jobmanager-deployment.yaml > > > kubectl create -f taskmanager-deployment.yaml > > > kubectl apply -f jobmanager-rest-service.yaml > > > > > > >>> grab-the-code > > > mvn archetype:generate \ > > > -DarchetypeGroupId=org.apache.beam \ > > > -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \ > > > -DarchetypeVersion=2.15.0 \ > > > -DgroupId=org.example \ > > > -DartifactId=word-count-beam \ > > > -Dversion="0.1" \ > > > -Dpackage=org.apache.beam.examples \ > > > -DinteractiveMode=false > > > > > > >>> maven-submit > > > > > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in separate > > shell, as blocking > > > mvn package exec:java -X > > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \ > > > -Dexec.args="--runner=FlinkRunner\ > > > --flinkMaster="localhost:8081"\ > > > --filesToStage=target/word-count-beam-bundled-0.1.jar\ > > > --output=<directory-that-exists>" -Pflink-runner > > > > > > > > > > > >
