Does the description in [1] match your case?
[1] https://github.com/apache/beam/issues/31085#issuecomment-2115304242
On 5/19/24 10:07, Yarden BenMoshe wrote:
I am not running my pipeline from command-line, so used instead:
options.setExperiments(Collections.singletonList("use_deprecated_read"));
with ExperimentalOptions added to my options interface, however I dont
think there's any effect to using it. in terms of the watermark, i
received again:
WatermarkHold.addHolds: element hold at 2024-05-19T07:52:59.999Z is on
time for key:aaa-bbb-ccc;
window:[2024-05-19T07:52:00.000Z..2024-05-19T07:53:00.000Z);
inputWatermark:-290308-12-21T19:59:05.225Z;
outputWatermark:-290308-12-21T19:59:05.225Z
בתאריך יום ה׳, 16 במאי 2024 ב-17:06 מאת Jan Lukavský
<[email protected]>:
Does using --experiments=use_deprecated_read have any effect?
On 5/16/24 14:30, Yarden BenMoshe wrote:
Hi Jan, my PipelineOptions is as follows:
options.setStreaming(true);
options.setAttachedMode(false);
options.setRunner(FlinkRunner.class);
I've also tried adding:
options.setAutoWatermarkInterval(100L);
as seen in some github issue, without any success so far.
other than that, i am working with parallelism:3 and number of
task slots: 3
Thanks!
Yarden
בתאריך יום ה׳, 16 במאי 2024 ב-15:05 מאת Jan Lukavský
<[email protected]>:
Hi Yarden,
can you please provide all flink-related PipelineOptions you
use for the
job?
Jan
On 5/16/24 13:44, Yarden BenMoshe wrote:
> Hi all,
> I have a project running with Beam 2.51, using Flink
runner. In one of
> my pipelines i have a FixedWindow and had a problem
upgrading until
> now, with a timers issue now resolved, and hopefully
allowing me to
> upgrade to version 2.56
> However, I encounter another problem now which I believe is
related to
> watermarking(?).
> My pipeline's source is a kafka topic.
> My basic window definition is:
>
> PCollection<KV<String, Iterable<CustomObject>>>
windowCustomObjectInfo
> = customObject.apply("windowCustomObjectInfo",
>
Window.into(FixedWindows.of(Duration.standardSeconds(60)))).apply(GroupByKey.create());
>
> and ever since upgrading to version 2.56 I am not getting
any output
> from that window. when enabling TRACE logs, i have this
message:
>
> 2024-05-12 13:50:55,257 TRACE
org.apache.beam.sdk.util.WindowTracing
> [] - WatermarkHold.addHolds: element hold at
2024-05-12T13:50:59.999Z
> is on time for key:test-12345;
> window:[2024-05-12T13:50:00.000Z..2024-05-12T13:51:00.000Z);
> inputWatermark:-290308-12-21T19:59:05.225Z;
> outputWatermark:-290308-12-21T19:59:05.225Z
>
>
> Any hints on where should I look or maybe how I can adjust
my window
> definition? Are you familiar with any change that might be
the cause
> for my issue?
> Thanks
