Re: Running NexMark Tests

2020-05-19 Thread Sruthi Sree Kumar 
PR for the update: https://github.com/apache/beam/pull/11751

Regards,
Sruthi

On Tue, May 19, 2020 at 3:51 PM Maximilian Michels  wrote:

> Looks like an accidental change to me. Running with either version, 1.9
> or 1.10 works, but this should be changed back to using the latest version.
>
> Do you mind creating a PR?
>
> Thanks,
> Max
>
> On 19.05.20 13:02, Sruthi Sree Kumar wrote:
> > On the documentation, the version of Flink runner is changed to 1.9
> > which was 1.10(latest)
> > before
> https://github.com/apache/beam/commit/1d2700818474c008eaa324ac1b5c49c9d2857298#diff-0e75160f4b09a1a300671557930589d9
> .
> >
> > Is this an accidental change or is there any particular reason for this
> > downgrade of version?
> >
> > Regards,
> > Sruthi
> >
> > On Tue, May 12, 2020 at 7:21 PM Maximilian Michels  > > wrote:
> >
> > A heads-up if anybody else sees this, we have removed the flag:
> > https://jira.apache.org/jira/browse/BEAM-9900
> >
> > Further contributions are very welcome :)
> >
> > -Max
> >
> > On 11.05.20 17:05, Sruthi Sree Kumar wrote:
> > > I have opened a PR with the documentation change.
> > > https://github.com/apache/beam/pull/11662
> > >
> > > Regards,
> > > Sruthi
> > >
> > > On 2020/04/21 20:22:17, Ismaël Mejía  > > wrote:
> > >> You need to instruct the Flink runner to shutdown the the source
> > >> otherwise it will stay waiting.
> > >> You can this by adding the extra
> > >> argument`--shutdownSourcesOnFinalWatermark=true`
> > >> And if that works and you want to open a PR to update our
> > >> documentation that would be greatly appreciated.
> > >>
> > >> Regards,
> > >> Ismaël
> > >>
> > >>
> > >> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
> > >>  > > wrote:
> > >>>
> > >>> Hello,
> > >>>
> > >>> I am trying to run nexmark queries using flink runner streaming.
> > Followed the documentation and used the command
> > >>> ./gradlew :sdks:java:testing:nexmark:run \
> > >>>
> > >>> -Pnexmark.runner=":runners:flink:1.10" \
> > >>> -Pnexmark.args="
> > >>> --runner=FlinkRunner
> > >>> --suite=SMOKE
> > >>> --streamTimeout=60
> > >>> --streaming=true
> > >>> --manageResources=false
> > >>> --monitorJobs=true
> > >>> --flinkMaster=[local]"
> > >>>
> > >>>
> > >>> But after the events are read from the source, there is no
> > further progress and the job is always stuck at 99%. Is there any
> > configuration that I am missing?
> > >>>
> > >>> Regards,
> > >>> Sruthi
> > >>
> >
>

Re: Running NexMark Tests

2020-05-19 Thread Sruthi Sree Kumar 
On the documentation, the version of Flink runner is changed to 1.9 which
was 1.10(latest) before
https://github.com/apache/beam/commit/1d2700818474c008eaa324ac1b5c49c9d2857298#diff-0e75160f4b09a1a300671557930589d9
.

Is this an accidental change or is there any particular reason for this
downgrade of version?

Regards,
Sruthi

On Tue, May 12, 2020 at 7:21 PM Maximilian Michels  wrote:

> A heads-up if anybody else sees this, we have removed the flag:
> https://jira.apache.org/jira/browse/BEAM-9900
>
> Further contributions are very welcome :)
>
> -Max
>
> On 11.05.20 17:05, Sruthi Sree Kumar wrote:
> > I have opened a PR with the documentation change.
> > https://github.com/apache/beam/pull/11662
> >
> > Regards,
> > Sruthi
> >
> > On 2020/04/21 20:22:17, Ismaël Mejía  wrote:
> >> You need to instruct the Flink runner to shutdown the the source
> >> otherwise it will stay waiting.
> >> You can this by adding the extra
> >> argument`--shutdownSourcesOnFinalWatermark=true`
> >> And if that works and you want to open a PR to update our
> >> documentation that would be greatly appreciated.
> >>
> >> Regards,
> >> Ismaël
> >>
> >>
> >> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
> >>  wrote:
> >>>
> >>> Hello,
> >>>
> >>> I am trying to run nexmark queries using flink runner streaming.
> Followed the documentation and used the command
> >>> ./gradlew :sdks:java:testing:nexmark:run \
> >>>
> >>> -Pnexmark.runner=":runners:flink:1.10" \
> >>> -Pnexmark.args="
> >>> --runner=FlinkRunner
> >>> --suite=SMOKE
> >>> --streamTimeout=60
> >>> --streaming=true
> >>> --manageResources=false
> >>> --monitorJobs=true
> >>> --flinkMaster=[local]"
> >>>
> >>>
> >>> But after the events are read from the source, there is no further
> progress and the job is always stuck at 99%. Is there any configuration
> that I am missing?
> >>>
> >>> Regards,
> >>> Sruthi
> >>
>

Re: Running NexMark Tests

2020-05-19 Thread Maximilian Michels 
Looks like an accidental change to me. Running with either version, 1.9
or 1.10 works, but this should be changed back to using the latest version.

Do you mind creating a PR?

Thanks,
Max

On 19.05.20 13:02, Sruthi Sree Kumar wrote:
> On the documentation, the version of Flink runner is changed to 1.9
> which was 1.10(latest)
> before 
> https://github.com/apache/beam/commit/1d2700818474c008eaa324ac1b5c49c9d2857298#diff-0e75160f4b09a1a300671557930589d9.
> 
> Is this an accidental change or is there any particular reason for this
> downgrade of version?
> 
> Regards,
> Sruthi
> 
> On Tue, May 12, 2020 at 7:21 PM Maximilian Michels  > wrote:
> 
> A heads-up if anybody else sees this, we have removed the flag:
> https://jira.apache.org/jira/browse/BEAM-9900
> 
> Further contributions are very welcome :)
> 
> -Max
> 
> On 11.05.20 17:05, Sruthi Sree Kumar wrote:
> > I have opened a PR with the documentation change.
> > https://github.com/apache/beam/pull/11662
> >
> > Regards,
> > Sruthi
> >
> > On 2020/04/21 20:22:17, Ismaël Mejía  > wrote:
> >> You need to instruct the Flink runner to shutdown the the source
> >> otherwise it will stay waiting.
> >> You can this by adding the extra
> >> argument`--shutdownSourcesOnFinalWatermark=true`
> >> And if that works and you want to open a PR to update our
> >> documentation that would be greatly appreciated.
> >>
> >> Regards,
> >> Ismaël
> >>
> >>
> >> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
> >>  > wrote:
> >>>
> >>> Hello,
> >>>
> >>> I am trying to run nexmark queries using flink runner streaming.
> Followed the documentation and used the command
> >>> ./gradlew :sdks:java:testing:nexmark:run \
> >>>
> >>>     -Pnexmark.runner=":runners:flink:1.10" \
> >>>     -Pnexmark.args="
> >>>         --runner=FlinkRunner
> >>>         --suite=SMOKE
> >>>         --streamTimeout=60
> >>>         --streaming=true
> >>>         --manageResources=false
> >>>         --monitorJobs=true
> >>>         --flinkMaster=[local]"
> >>>
> >>>
> >>> But after the events are read from the source, there is no
> further progress and the job is always stuck at 99%. Is there any
> configuration that I am missing?
> >>>
> >>> Regards,
> >>> Sruthi
> >>
>

Re: Running NexMark Tests

2020-05-12 Thread Maximilian Michels 
A heads-up if anybody else sees this, we have removed the flag:
https://jira.apache.org/jira/browse/BEAM-9900

Further contributions are very welcome :)

-Max

On 11.05.20 17:05, Sruthi Sree Kumar wrote:
> I have opened a PR with the documentation change.
> https://github.com/apache/beam/pull/11662
> 
> Regards,
> Sruthi
> 
> On 2020/04/21 20:22:17, Ismaël Mejía  wrote: 
>> You need to instruct the Flink runner to shutdown the the source
>> otherwise it will stay waiting.
>> You can this by adding the extra
>> argument`--shutdownSourcesOnFinalWatermark=true`
>> And if that works and you want to open a PR to update our
>> documentation that would be greatly appreciated.
>>
>> Regards,
>> Ismaël
>>
>>
>> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
>>  wrote:
>>>
>>> Hello,
>>>
>>> I am trying to run nexmark queries using flink runner streaming. Followed 
>>> the documentation and used the command
>>> ./gradlew :sdks:java:testing:nexmark:run \
>>>
>>> -Pnexmark.runner=":runners:flink:1.10" \
>>> -Pnexmark.args="
>>> --runner=FlinkRunner
>>> --suite=SMOKE
>>> --streamTimeout=60
>>> --streaming=true
>>> --manageResources=false
>>> --monitorJobs=true
>>> --flinkMaster=[local]"
>>>
>>>
>>> But after the events are read from the source, there is no further progress 
>>> and the job is always stuck at 99%. Is there any configuration that I am 
>>> missing?
>>>
>>> Regards,
>>> Sruthi
>>

Re: Running NexMark Tests

2020-04-22 Thread Maximilian Michels 
The flag is needed when checkpointing is enabled because Flink is unable
to create a new checkpoint when not all operators are running.

By default, operators shut down when all input has been read. That will
trigger sending out the maximum (final) watermark at the sources. The
flag name is a bit confusing in this regard because shutting down the
sources triggers sending out the watermark, not the other way around.

-Max

On 22.04.20 06:26, Kenneth Knowles wrote:
> We should always want to shut down sources on final watermark. All
> incoming data should be dropped anyhow.
> 
> Kenn
> 
> On Tue, Apr 21, 2020 at 1:34 PM Luke Cwik  > wrote:
> 
> +dev
> 
> When would we not want --shutdownSourcesOnFinalWatermark=true ?
> 
> On Tue, Apr 21, 2020 at 1:22 PM Ismaël Mejía  > wrote:
> 
> You need to instruct the Flink runner to shutdown the the source
> otherwise it will stay waiting.
> You can this by adding the extra
> argument`--shutdownSourcesOnFinalWatermark=true`
> And if that works and you want to open a PR to update our
> documentation that would be greatly appreciated.
> 
> Regards,
> Ismaël
> 
> 
> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
>  > wrote:
> >
> > Hello,
> >
> > I am trying to run nexmark queries using flink runner
> streaming. Followed the documentation and used the command
> > ./gradlew :sdks:java:testing:nexmark:run \
> >
> >     -Pnexmark.runner=":runners:flink:1.10" \
> >     -Pnexmark.args="
> >         --runner=FlinkRunner
> >         --suite=SMOKE
> >         --streamTimeout=60
> >         --streaming=true
> >         --manageResources=false
> >         --monitorJobs=true
> >         --flinkMaster=[local]"
> >
> >
> > But after the events are read from the source, there is no
> further progress and the job is always stuck at 99%. Is there
> any configuration that I am missing?
> >
> > Regards,
> > Sruthi
>

Re: Running NexMark Tests

2020-04-21 Thread Kenneth Knowles 
We should always want to shut down sources on final watermark. All incoming
data should be dropped anyhow.

Kenn

On Tue, Apr 21, 2020 at 1:34 PM Luke Cwik  wrote:

> +dev
>
> When would we not want --shutdownSourcesOnFinalWatermark=true ?
>
> On Tue, Apr 21, 2020 at 1:22 PM Ismaël Mejía  wrote:
>
>> You need to instruct the Flink runner to shutdown the the source
>> otherwise it will stay waiting.
>> You can this by adding the extra
>> argument`--shutdownSourcesOnFinalWatermark=true`
>> And if that works and you want to open a PR to update our
>> documentation that would be greatly appreciated.
>>
>> Regards,
>> Ismaël
>>
>>
>> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
>>  wrote:
>> >
>> > Hello,
>> >
>> > I am trying to run nexmark queries using flink runner streaming.
>> Followed the documentation and used the command
>> > ./gradlew :sdks:java:testing:nexmark:run \
>> >
>> > -Pnexmark.runner=":runners:flink:1.10" \
>> > -Pnexmark.args="
>> > --runner=FlinkRunner
>> > --suite=SMOKE
>> > --streamTimeout=60
>> > --streaming=true
>> > --manageResources=false
>> > --monitorJobs=true
>> > --flinkMaster=[local]"
>> >
>> >
>> > But after the events are read from the source, there is no further
>> progress and the job is always stuck at 99%. Is there any configuration
>> that I am missing?
>> >
>> > Regards,
>> > Sruthi
>>
>

Re: Running NexMark Tests

2020-04-21 Thread Luke Cwik 
+dev

When would we not want --shutdownSourcesOnFinalWatermark=true ?

On Tue, Apr 21, 2020 at 1:22 PM Ismaël Mejía  wrote:

> You need to instruct the Flink runner to shutdown the the source
> otherwise it will stay waiting.
> You can this by adding the extra
> argument`--shutdownSourcesOnFinalWatermark=true`
> And if that works and you want to open a PR to update our
> documentation that would be greatly appreciated.
>
> Regards,
> Ismaël
>
>
> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
>  wrote:
> >
> > Hello,
> >
> > I am trying to run nexmark queries using flink runner streaming.
> Followed the documentation and used the command
> > ./gradlew :sdks:java:testing:nexmark:run \
> >
> > -Pnexmark.runner=":runners:flink:1.10" \
> > -Pnexmark.args="
> > --runner=FlinkRunner
> > --suite=SMOKE
> > --streamTimeout=60
> > --streaming=true
> > --manageResources=false
> > --monitorJobs=true
> > --flinkMaster=[local]"
> >
> >
> > But after the events are read from the source, there is no further
> progress and the job is always stuck at 99%. Is there any configuration
> that I am missing?
> >
> > Regards,
> > Sruthi
>