Yes. But the filter us usually a very light weight operation.
From: Mingliang Liu
Sent: Monday, June 3, 2024 7:16 PM
To: Colletta, Edward
Cc: mejri houssem ; user@flink.apache.org
Subject: Re: Implementing Multiple sink
NOTICE: This email is from an external sender - do not click on links
I usually just reuse the stream, sending it to through different filters and
adding different sinks to the filtered streams.
Something like
streamWithMultipleConditions.filter(FilterForCondition1)
.addSink(SinkforCondtiton1);
:[account],groupByType:byAccount,aggregationKey:'2'}
From: Colletta, Edward
Sent: Tuesday, January 25, 2022 1:29 PM
To: M Singh ; Caizhi Weng ;
User-Flink
Subject: RE: Apache Fink - Adding/Removing KeyedStreams at run time without
stopping the application
You don’t have to add keyBy’s at runtime
, January 25, 2022 1:12 PM
To: Caizhi Weng ; User-Flink ;
Colletta, Edward
Subject: Re: Apache Fink - Adding/Removing KeyedStreams at run time without
stopping the application
NOTICE: This email is from an external sender - do not click on links or
attachments unless you recognize the sender
A general pattern for dynamically adding new aggregations could be something
like this
BroadcastStream broadcastStream =
aggregationInstructions
.broadcast(broadcastStateDescriptor);
DataStream
streamReadyToAggregate = dataToAggregate
If have jar files built using flink version 11.2 in dependencies, and I upgrade
my cluster to 11.6, is it safe to run the existing jars on the upgraded cluster
or should I rebuild all jobs against 11.6?
Thanks,
Eddie Colletta
Thanks, will try that.
From: Chesnay Schepler
Sent: Friday, August 20, 2021 8:06 AM
To: Colletta, Edward ; user@flink.apache.org
Subject: Re: failures during job start
NOTICE: This email is from an external sender - do not click on links or
attachments unless you recognize the sender and know
To: Colletta, Edward ; user@flink.apache.org
Subject: Re: failures during job start
NOTICE: This email is from an external sender - do not click on links or
attachments unless you recognize the sender and know the content is safe.
This exception means that a task was deployed, but the task
Any help with this would be appreciated. Is it possible that this is a
data/application issue or a flink config/resource issue?
Using flink 11.2, java 11, session cluster, 5 nodes 32 cores each node.
I have an issue where starting a job takes a long time, and sometimes fails
with
is I'm not aware of any issue related to the
upgrading of the ZK version that may cause the leadership loss.
Thank you~
Xintong Song
On Sun, Jan 31, 2021 at 4:14 AM Colletta, Edward
mailto:edward.colle...@fmr.com>> wrote:
“but I'm not aware of any similar issue reported since the upgra
Thank you.
-Original Message-
From: Tzu-Li (Gordon) Tai
Sent: Monday, March 15, 2021 3:05 AM
To: user@flink.apache.org
Subject: Re: uniqueness of name when constructing a StateDescriptor
NOTICE: This email is from an external sender - do not click on links or
attachments unless you
The documentation for ValueStateDescriptor documents the name parameter as -
"name - The (unique) name for the state."
What is the scope of the uniqueness? Unique within an RichFunction instance?
Unique withing job? Unique within a session cluster?
I ask because I have several jobs that use a
Using FsStateBackend.
I was under the impression that ValueState.value will serialize an object which
is stored in the local state backend, copy the serialized object and
deserializes it. Likewise update() would do the same steps copying the object
back to local state backend.And as a
You can use a tumbling processing time window with an offset of 13 hours + your
time zone offset.
https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/stream/operators/windows.html#tumbling-windows
“but I'm not aware of any similar issue reported since the upgrading”
For the record, we experienced this same error on Flink 1.11.2 this past week.
From: Xintong Song
Sent: Friday, January 29, 2021 7:34 PM
To: user
Subject: Re: Flink 1.11 job hit error "Job leader lost leadership" or
Using flink 11.2 on java 11, session cluster with 16 jobs running on aws ecs
instances. Cluster has 3 JMs and 3 TMs, separate zookeeper cluster has 3 nodes.
One of our taskmanagers crashed today with what seems to be rooted in a
zookeeper timeout. We are wondering if there is any tuning that
...@aliyun.com>>
Sent: Monday, December 21, 2020 11:28 AM
To: Colletta, Edward mailto:edward.colle...@fmr.com>>;
user@flink.apache.org<mailto:user@flink.apache.org>
Subject: Re: RE: checkpointing seems to be throttled.
This email is from an external source - exercise caution regarding link
Doh! Yeah, we set the state backend in code and I read the flink-conf.yaml
file and use the high-availability storage dir.
From: Yun Gao
Sent: Monday, December 21, 2020 11:28 AM
To: Colletta, Edward ; user@flink.apache.org
Subject: Re: RE: checkpointing seems to be throttled.
This email
Thanks for the quick response.
We are using FsStateBackend, and I did see checkpoint files and directories in
the EFS mounted directory.
We do monitor backpressure through rest api periodically and we do not see any.
From: Yun Gao
Sent: Monday, December 21, 2020 10:40 AM
To: Colletta, Edward
Using session cluster with three taskmanagers, cluster.evenly-spread-out-slots
is set to true. 13 jobs running. Average parallelism of each job is 4.
Flink version 1.11.2, Java 11.
Running on AWS EC2 instances with EFS for high-availability.storageDir.
We are seeing very high checkpoint times
Thanks you for the response. We do see the heap actually shrink after
starting new jobs.
From: Matthias Pohl
Sent: Thursday, November 5, 2020 8:20 AM
To: Colletta, Edward
Cc: user@flink.apache.org
Subject: Re: a couple of memory questions
This email is from an external source - exercise
Using Flink 1.9.2 with FsStateBackend, Session cluster.
1. Does heap state get cleaned up when a job is cancelled?
We have jobs that we run on a daily basis. We start each morning and cancel
each evening. We noticed that the process size does not seem to shrink. We
are looking at the
, I did that just for the test. For my prod code, going forward, I am
following flink’s rules for POJO types, adding static to any inner class, and
checking for any POJO warnings in the logs.
From: Arvid Heise
Sent: Sunday, October 11, 2020 3:46 PM
To: Colletta, Edward
Cc: Dawid Wysakowicz
: Dawid Wysakowicz mailto:dwysakow...@apache.org>>
Sent: Thursday, October 8, 2020 6:26 AM
To: Colletta, Edward mailto:edward.colle...@fmr.com>>;
user@flink.apache.org<mailto:user@flink.apache.org>
Subject: Re: state access causing segmentation fault
Hi,
It should be absolutely fine
Using Flink 1.9.2, Java, FsStateBackend. Running Session cluster on EC2
instances.
I have a KeyedProcessFunction that is causing a segmentation fault, crashing
the flink task manager. The seems to be caused by using 3 State variables in
the operator. The crash happens consistently after
25 matches
Mail list logo
