Hello,
I have questions on watermark generation and trigger about ReadFromKafka
(Python SDK).
1.It only supports the following simple timestamp policies:
- ProcessingTime
- CreateTime
- LogAppendTime
Is there any way to assign event time from timestamp in a Kafka message?
> The Java SDK
You can check
https://github.com/search?q=repo%3Ajohnjcasey%2Fbeam%20withBadRecordErrorHandler=code.
The test codes show how to use them. More doc will be added later.
On Wed, Mar 27, 2024 at 7:15 PM Ruben Vargas
wrote:
> Hello all
>
> Maybe a silly question. Are there any suggestions for
Hello all
Maybe a silly question. Are there any suggestions for implementing a DLQ
in my beam pipeline?
Currently I'm using this library https://github.com/tosun-si/asgarde which
is not bad, the only issue I found is that sometimes it is hard to use
with GroupIntoBatches or other native
KafkaIO is translated using SDF by default. There were some changes how
watermarks are emitted from Impulse (which is what is the SDF "starting
point", which might affect what tou see). Please try one of the following:
a) use --shutdownSourcesAfterIdleMs (e.g. 5000) which should trigger
we are running with flink runner, i also tested with direct runner. same
results
On Wed, Mar 27, 2024 at 2:51 PM Sigalit Eliazov wrote:
> hi,
> this is the pipeline, very simple one
> the onTimer is not fired.
> We are not using any experimental variables.
>
> public class KafkaBeamPipeline {
>
hi,
this is the pipeline, very simple one
the onTimer is not fired.
We are not using any experimental variables.
public class KafkaBeamPipeline {
static class ProcessMessageFn extends DoFn, String> {
@StateId("count")
private final StateSpec> stateSpec =
StateSpecs.value();
Hello to all users, contributors and Committers!
[ You are receiving this email as a subscriber to one or more ASF project
dev or user
mailing lists and is not being sent to you directly. It is important that
we reach all of our
users and contributors/committers so that they may get a chance
Hi,
what is your runner, is it Flink as well in the issue? What is the
source of your Pipeline? Do you use some additional flags, e.g.
--experiments? Do you see that using classical or portable runner?
Jan
On 3/26/24 19:18, Sigalit Eliazov wrote:
Hi all
We encountered issue with timers
