Hi Ammon,
Unfortunately you're right. I think the Flink Kinesis Consumer specific
configs, e.g. keys in the ConsumerConfigConstants class, were overlooked in
the initial design.
One way to workaround this is to use the `SourceFunctionSpec` [1]. Using
that spec, you can use any Flink
there is a delay we need to know. But if the
> customer confirms in time we want to cleanup to keep the state small.
>
>
>
> I dug a little bit into the code. May I create an issue to discuss my
> ideas?
>
>
>
> Cheers,
>
> Stephan
>
>
>
>
>
> *Von:* Tzu
Hi Jose!
For Scala, we would suggest to wait until StateFun 3.0.0 is released, which
is actually happening very soon (likely within 1-2 weeks) as there is an
ongoing release candidate vote [1].
The reason for this is that version 3.0 adds a remote SDK for Java, which
you should be able to use
Hi Le,
Thanks for reaching out with this question! It's actually a good segue to
allow me to introduce you to StateFun 3.0.0 :)
StateFun 3.0+ comes with a new type system that would eliminate this
hassle. You can take a sneak peek here [1].
This is part 1 of a series of tutorials on fundamentals
Hi CZ,
The issue here is that the Scala DataStream API uses Scala macros to decide
the serializer to be used. Since that recognizes Scala case classes, the
CaseClassSerializer will be used.
However, in the State Processor API, those Scala macros do not come into
play, and therefore it directly
Hi,
I'm pulling in Rui Li (cc'ed) who might be able to help you here as he
actively maintains the hive connectors.
Cheers,
Gordon
On Fri, Apr 2, 2021 at 11:36 AM Yik San Chan
wrote:
> The question is cross-posted in StackOverflow
>
Hi Sonam,
Pulling in Till (cc'ed), I believe he would likely be able to help you here.
Cheers,
Gordon
On Fri, Apr 2, 2021 at 8:18 AM Sonam Mandal wrote:
> Hello,
>
> We are experimenting with task local recovery and I wanted to know whether
> there is a way to validate that some tasks of the
checkpoint status and conservatively rollback to previous
> checkpoint and replay all data
>
> On Thu, Mar 11, 2021 at 7:44 AM Tzu-Li (Gordon) Tai
> wrote:
>
>> Hi Kevin,
>>
>> Perhaps the easiest way to answer your question, is to go through how the
>> exactly
Hi Andrey,
Perhaps the functionality you described is worth adding to the State
Processor API.
Your observation on how the library currently works is correct; basically it
tries to restore the state backends as is.
In you current implementation, do you see it worthwhile to try to add this?
Hi,
Interesting idea! Just some initial thoughts and questions, maybe others can
chime in as well.
In general I think the idea of supporting more high-level protocols on top
of the existing StateFun messaging primitives is good.
For example, what probably could be categorized under this effort
Hi,
The scope is per individual operator, i.e. a single KeyedProcessFunction
instance cannot have multiple registered state with the same name.
Cheers,
Gordon
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Hi,
Could you provide info on the Flink version used?
Cheers,
Gordon
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Hi Dan,
For a deeper dive into state backends and how they manage state, or
performance critical aspects such as state serialization and choosing
appropriate state structures, I highly recommend starting from this webinar
done by my colleague Seth Weismann:
Hi Kevin,
Perhaps the easiest way to answer your question, is to go through how the
exactly-once FlinkKafkaProducer using a 2PC implementation on top of
Flink's checkpointing mechanism.
The phases can be broken down as follows (simplified assuming max 1
concurrent checkpoint and that checkpoint
Hi Timothy,
Starting from StateFun 2.2.x, in the module.yaml file, you can set for each
individual state of a function an "expireMode" field, which values can be
either "after-invoke" or "after-write". For example:
```
- function:
meta:
...
spec:
states:
- name:
Hi Alexey,
Are you using the heap backend? If that's the case, then for whatever state
was registered at the time of a savepoint, Flink will attempt to restore it
to the heap backends.
This essentially means that state "B" will be read as well, that would
explain why Flink is trying to locate
t; Alexey
>
> ------
> *From:* Tzu-Li (Gordon) Tai
> *Sent:* Thursday, March 4, 2021 12:58:01 AM
> *To:* Alexey Trenikhun
> *Cc:* Piotr Nowojski ; Flink User Mail List <
> user@flink.apache.org>
> *Subject:* Re: Job downgrade
>
> Hi Alexey,
>
> Are you using the heap bac
b.com/apache/flink-statefun/tree/master/statefun-sdk-java
> Thanks,
>
> Tim
>
> On Wed, Feb 24, 2021 at 11:49 AM Tzu-Li (Gordon) Tai
> wrote:
>
>> Hi Timothy,
>>
>> Starting from StateFun 2.2.x, in the module.yaml file, you can set for
>> each indivi
The Apache Flink community is very happy to announce the release of Apache
Flink Stateful Functions (StateFun) 3.0.0.
StateFun is a cross-platform stack for building Stateful Serverless
applications, making it radically simpler to develop scalable, consistent,
and elastic distributed
Hi,
Could you re-elaborate what exactly you mean?
If you wish to run a Flink job within the IDE, but also have the web UI
running for it, you can use
`StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(Configuration)`
to create the execution environment.
The default port 8081 will be
Hi,
FWIW, there is this JIRA that is tracking a pubsub / broadcast messaging
primitive in StateFun:
https://issues.apache.org/jira/browse/FLINK-16319
This is probably what you are looking for. And I do agree, in the case that
the control stream (which updates the application logic) is high
Hi,
I'm not aware of any breaking changes in the savepoint formats from 1.9.3 to
1.11.3.
Let's first try to rule out any obvious causes of this:
- Were any data types / classes that were used in state changed across the
restores? Remember that keys types are also written as part of state
Congrats on the release!
And thank you for driving this release, Igal.
Cheers
Gordon
On Tue, Aug 31, 2021, 23:13 Igal Shilman wrote:
> The Apache Flink community is very happy to announce the release of Apache
> Flink Stateful Functions (StateFun) 3.1.0.
>
> StateFun is a cross-platform stack
The Apache Flink community is very happy to announce the release of Apache
Flink Kafka Connectors 3.0.1. This release is compatible with the Apache
Flink 1.17.x and 1.18.x release series.
Apache Flink® is an open-source stream processing framework for
distributed, high-performing,
Hi Xianxun,
You can find the list supported Flink versions for each connector here:
https://flink.apache.org/downloads/#apache-flink-connectors
Specifically for the Kafka connector, we're in the process of releasing a
new version for the connector that works with Flink 1.18.
The release
Hi Lorenzo,
The main failure scenario that recommendation is addressing is when the
Flink job fails right after a checkpoint successfully completes, but before
the KafkaSink subtasks receive from the JM the checkpoint completed RPC
notification to commit the transactions. It is possible that
elp with Statefun releases.
>>>>
>>>> Best regards,
>>>>
>>>> Martijn
>>>>
>>>> On Tue, Jun 20, 2023 at 2:21 PM Galen Warren
>>>> wrote:
>>>>
>>>>> Thanks.
>>>>>
>>>>> Marti
default vs. local mode and what versions of Flink and Statefun
> should be referenced, and then you can build a run the local examples
> without any additional steps. Does that sound like a reasonable approach?
>
>
> On Fri, Aug 18, 2023 at 2:17 PM Tzu-Li (Gordon) Tai
> wrote:
>
>
solve the copying problem by requiring the two projects to be
> siblings in the file system and by pre-copying the local build artifacts
> into a location accessible by the existing Docker contexts. This would
> still leave us with the need to have two PRs and releases instead of one,
> t
Hi Robin,
Apart from what Alexander suggested, I think you could also try the
following first:
Let the job use a "new" Kafka source, which you can achieve by simply
assigning a different operator ID than before. If you previously did not
set an ID, then previously by default it would have been a
Hi Filip,
I think what you are seeing is expected. The State Processor API was
intended to allow access only to commonly used user-facing state
structures, while Stateful Functions uses quite a bit of Flink internal
features, including for its state maintenance.
The list state in question in
Hi Robert,
This is a design choice. Flink's KafkaSource doesn't rely on consumer
groups for assigning partitions / rebalancing / offset tracking. It
manually assigns whatever partitions are in the specified topic across its
consumer instances, and rebalances only when the Flink job / KafkaSink is
Sorry, I meant to say "Hi Ben" :-)
On Thu, Mar 30, 2023 at 9:52 AM Tzu-Li (Gordon) Tai
wrote:
> Hi Robert,
>
> This is a design choice. Flink's KafkaSource doesn't rely on consumer
> groups for assigning partitions / rebalancing / offset tracking. It
> manually as
> Perhaps he could weigh in on whether the combination of automated tests
plus those smoke tests should be sufficient for testing with new Flink
versions
What we usually did at the bare minimum for new StateFun releases was the
following:
1. Build tests (including the smoke tests in the e2e
Hi Anirban,
> But this happened only once and now it is not getting reproduced at all.
This does make it sound a lot like
https://issues.apache.org/jira/browse/FLINK-31632.
> 1. What is the default watermarking strategy used in Flink. Can I quickly
check the default parameters being used by
:flink-connector-kafka:3.0.2-18
> [error] Not found
> [error] Not found
> [error] not found:
>
> /home/swissbib/.ivy2/local/org.apache.flink/flink-connector-kafka/3.0.2-18/ivys/ivy.xml
> [error] not found:
>
> https://repo1.maven.org/maven2/org/apache/flink/flink-conne
new version I can add the dependency
> "org.apache.flink" % "flink-connector-kafka" % "3.0.2-1.18",
>
>
> and compile it without any errors.
>
> Günter
>
>
> On 25.11.23 17:40, Tzu-Li (Gordon) Tai wrote:
> > Hi Günter,
> >
&g
The Apache Flink community is very happy to announce the release of Apache
Flink Kafka Connectors 3.0.2. This release is compatible with the Apache
Flink 1.17.x and 1.18.x release series.
Apache Flink® is an open-source stream processing framework for
distributed, high-performing,
-impl:jar:2.17.1:runtime
[INFO] +- org.apache.logging.log4j:log4j-api:jar:2.17.1:runtime
[INFO] \- org.apache.logging.log4j:log4j-core:jar:2.17.1:runtime
```
On Thu, Nov 23, 2023 at 11:48 AM Tzu-Li (Gordon) Tai
wrote:
> Hi all,
>
> There seems to be an issue with the connector release scr
Hi all,
There seems to be an issue with the connector release scripts used in the
release process that doesn't correctly overwrite the flink.version property
in POMs.
I'll kick off a new release for 3.0.2 shortly to address this. Sorry for
overlooking this during the previous release.
Best,
Hi all,
I've cherry-picked FLINK-30400 onto v3.0 branch of flink-connector-kafka.
Treating this thread as justification to start a vote for 3.0.2 RC #1
immediately so we can get out a new release ASAP. Please see the vote
thread here [1].
@guenterh.lists Would you be able to test this
RC and
Hi,
除了 Apache Flink 官方文件以外 [1]，我個人也建議可以看看 Ververica 這一系列的 Flink training 題材:
https://training.ververica.com/
除此之外，學習過程中有遇到任何問題也歡迎可以直接發信件跟我們詢問。
- Gordon
[1] https://flink.apache.org/
On Thu, Feb 14, 2019 at 11:44 AM shen lei wrote:
> 有木有好的经验或者方法分享一下，感谢。最近学的，感觉还是不系统。
The Apache Flink community released the first bugfix release of the
Stateful Functions (StateFun) 2.2 series, version 2.2.1.
This release fixes a critical bug that causes restoring a Stateful
Functions cluster from snapshots (checkpoints or savepoints) to fail under
certain conditions.
*We
The Apache Flink community released the second bugfix release of the
Stateful Functions (StateFun) 2.2 series, version 2.2.2.
*We strongly recommend all users to upgrade to this version.*
*Please check out the release announcement:*
The Apache Flink community is very happy to announce the release of Apache
Flink Stateful Functions (StateFun) 3.0.0.
StateFun is a cross-platform stack for building Stateful Serverless
applications, making it radically simpler to develop scalable, consistent,
and elastic distributed
Congrats on the release!
And thank you for driving this release, Igal.
Cheers
Gordon
On Tue, Aug 31, 2021, 23:13 Igal Shilman wrote:
> The Apache Flink community is very happy to announce the release of Apache
> Flink Stateful Functions (StateFun) 3.1.0.
>
> StateFun is a cross-platform stack
501 - 546 of 546 matches
Mail list logo
