Tentative -1.

I tried to validate quickstart on Flink on a Dataproc cluster and ran into
a couple of issues.

Here's the script I followed:
$ curl
https://raw.githubusercontent.com/GoogleCloudPlatform/dataproc-initialization-actions/master/flink/flink.sh
-o flink.sh
$ gsutil cp flink.sh gs://my_bucket/flink.sh
$ gcloud dataproc clusters create kirpichov-flink-230-rc3
--initialization-actions gs://my_bucket/flink.sh
--initialization-action-timeout 5m --scopes=
https://www.googleapis.com/auth/cloud-platform
$ mvn package -Pflink-runner
$ gcloud compute scp --zone "us-central1-a"
target/word-count-beam-bundled-0.1.jar kirpichov-flink-230-rc3-m:~/
$ gcloud compute ssh --zone "us-central1-a" "kirpichov-flink-230-rc3-m"
$ sudo su -
$ cd /home/kirpichov
$ yarn application -list
# (get the application id)
$ HADOOP_CONF_DIR=/etc/hadoop/conf /usr/lib/flink/bin/flink run -m
yarn-cluster -yid (application id) -c org.apache.beam.examples.WordCount
word-count-beam-bundled-0.1.jar --runner=FlinkRunner
"--inputFile=gs://apache-beam-samples/shakespeare/*"
"--output=gs://my_bucket/word-count-output"

I ran into the following issues:
1) java.lang.NoSuchMethodError:
com.google.protobuf.Descriptors$Descriptor.getOneofs()Ljava/util/List;
This is I suppose because Flink includes a version of protobuf that clashes
with ours.
I resolved this by changing the configuration of the shade plugin in the
generated wordcount pom.xml:
               <relocations>
                 <relocation>
                   <pattern>com.google.protobuf</pattern>

 <shadedPattern>org.example.shaded.com.google.protobuf</shadedPattern>
                 </relocation>
               </relocations>

2) After this (and repeating instructions starting with "mvn package"), I
got a different error:
Caused by: com.typesafe.config.ConfigException$UnresolvedSubstitution:
reference.conf: 804: Could not resolve substitution to a value:
${akka.stream.materializer}
(full stack: https://gist.github.com/jkff/9fb07e470f81b558e38b1bccfe2e4a99)

The flink version on the Dataproc cluster is 1.3.1. I'm not sure how to
proceed. Has anyone else been able to validate quickstart on Flink on YARN?

On Wed, Feb 14, 2018 at 12:24 PM Valentyn Tymofieiev <valen...@google.com>
wrote:

> +1, Validated Python - Mobile game walkthrough, Python - Quickstart
> (Streaming Alpha).
>
>
> On Wed, Feb 14, 2018 at 10:42 AM, Alan Myrvold <amyrv...@google.com>
> wrote:
>
>> +1 Validated java quickstarts for direct, dataflow, apex, flink, and
>> spark.
>>
>> On Wed, Feb 14, 2018 at 9:21 AM, Lukasz Cwik <lc...@google.com> wrote:
>>
>>> +1 (binding)
>>> Validated several quickstarts including the regression that I originally
>>> reported with Spark.
>>>
>>> On Wed, Feb 14, 2018 at 5:34 AM, Ismaël Mejía <ieme...@gmail.com> wrote:
>>>
>>>> +1 (binding)
>>>>
>>>> Validated SHAs + tag vs source.zip file.
>>>> Run mvn clean install -Prelease OK
>>>> Validated that the 3 regressions reported for RC1 were fixed.
>>>> Run Nexmark on Direct/Flink runner on local mode, no regressions now.
>>>> Installed python version on virtualenv and run local wordcount with
>>>> success.
>>>> Checked that the hadoop-input-format artifact is in the extended
>>>> staging area.
>>>>
>>>> On Tue, Feb 13, 2018 at 5:41 PM, Jean-Baptiste Onofré <j...@nanthrax.net>
>>>> wrote:
>>>> > +1 (binding)
>>>> >
>>>> > Tested the Spark runner (with wordcount example and beam samples)
>>>> > Tested the performance of the direct runner
>>>> >
>>>> > I just updated the spreadsheet.
>>>> >
>>>> > Regards
>>>> > JB
>>>> >
>>>> > On 02/11/2018 06:33 AM, Jean-Baptiste Onofré wrote:
>>>> >> Hi everyone,
>>>> >>
>>>> >> Please review and vote on the release candidate #3 for the version
>>>> 2.3.0, as
>>>> >> follows:
>>>> >>
>>>> >> [ ] +1, Approve the release
>>>> >> [ ] -1, Do not approve the release (please provide specific comments)
>>>> >>
>>>> >>
>>>> >> The complete staging area is available for your review, which
>>>> includes:
>>>> >> * JIRA release notes [1],
>>>> >> * the official Apache source release to be deployed to
>>>> dist.apache.org [2],
>>>> >> which is signed with the key with fingerprint C8282E76 [3],
>>>> >> * all artifacts to be deployed to the Maven Central Repository [4],
>>>> >> * source code tag "v2.3.0-RC3" [5],
>>>> >> * website pull request listing the release and publishing the API
>>>> reference
>>>> >> manual [6].
>>>> >> * Java artifacts were built with Maven 3.3.9 and Oracle JDK
>>>> 1.8.0_111.
>>>> >> * Python artifacts are deployed along with the source release to the
>>>> >> dist.apache.org [2].
>>>> >>
>>>> >> The vote will be open for at least 72 hours. It is adopted by
>>>> majority approval,
>>>> >> with at least 3 PMC affirmative votes.
>>>> >>
>>>> >> Thanks,
>>>> >> JB
>>>> >>
>>>> >> [1]
>>>> >>
>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341608
>>>> >> [2] https://dist.apache.org/repos/dist/dev/beam/2.3.0/
>>>> >> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>>> >> [4]
>>>> https://repository.apache.org/content/repositories/orgapachebeam-1028/
>>>> >> [5] https://github.com/apache/beam/tree/v2.3.0-RC3
>>>> >> [6] https://github.com/apache/beam-site/pull/381
>>>> >>
>>>> >
>>>> > --
>>>> > Jean-Baptiste Onofré
>>>> > jbono...@apache.org
>>>> > http://blog.nanthrax.net
>>>> > Talend - http://www.talend.com
>>>>
>>>
>>>
>>
>

Reply via email to