Help needed for passing in multiple filters in filter.includes parameter for my flink metric reporter in flink-conf.yaml?
Hi, can you please look at my StackOverflow query? https://stackoverflow.com/questions/76949195/how-to-include-multiple-filters-in-filter-includes-parameter-for-my-flink-metric Thanks!
[Discussion] Slack Channel
Hello folks, considering how apache flink gains more and more popularity and seeing how other open-source projects use Slack, I wanted to start this thread to see how we can grow the community. First of all one thing I have noticed, although there are people involved with Flink only lately they start noticing there is actually an open-source Slack channel. Maybe we can help somehow raise awareness around that? Like inviting people to join. The other part is around the channels. Currently, there is only one channel #troubleshooting for people to ask questions. I believe this creates a few limitations, as there are quite a few questions daily, but it's hard to differentiate between topics, and people with context on specific parts can't identify them easily. I'm thinking it would be nice to create more channels like: #flink-cdc #flink-paimon #flink-datastream #pyflink #flink-sql #flink-statebackends #flink-monitoring #flink-k8s #job-board etc. to help people have more visibility on different topics, make it easier to find answers to similar questions, and search for things of interest. This can be a first step towards growing the community. Best, Giannis
Re: Flink 1.17.2 planned?
Hi Christian, Thanks for reaching out. Liked Ron pointed out that the community is focusing on the 1.18 release. If you are facing urgent issues, would you like to volunteer as the release manager of 1.17.2 and drive the release? Theoretically, everyone could be the release manager of a bugs fix release. Best regards, Jing On Tue, Aug 22, 2023 at 3:41 AM liu ron wrote: > Hi, Christian > > We released 1.17.1 [1] in May, and the main focus of the community is > currently on the 1.18 release, so 1.17.2 should be planned for after the > 1.18 release! > > [1] > https://flink.apache.org/2023/05/25/apache-flink-1.17.1-release-announcement/ > > > Best, > Ron > > Christian Lorenz via user 于2023年8月21日周一 17:33写道： > >> Hi team, >> >> >> >> are there any infos about a bugfix release 1.17.2 available? E.g. will >> there be another bugfix release of 1.17 / approximate timing? >> >> We are hit by https://issues.apache.org/jira/browse/FLINK-32296 which >> leads to wrong SQL responses in some circumstances. >> >> >> >> Kind regards, >> >> Christian >> >> This e-mail is from Mapp Digital Group and its international legal >> entities and may contain information that is confidential. >> If you are not the intended recipient, do not read, copy or distribute >> the e-mail or any attachments. Instead, please notify the sender and delete >> the e-mail and any attachments. >> >
Request-Response flow for real-time analytics
Hi,
We are currently evaluating Flink for our analytics engine. We would
appreciate any help with our experiment in using flink for real-time
request-response use-case.
To demonstrate the current use-case: our application produces events of the
following form:
{id, customerId, amount, timestamp}
We calculate some continuous aggregates triggered by each event produced
and use them to decide on the action.
Examples of Aggregates: sum of amount total, amount group by customerId,
amount per day(group-by customer), per month etc.
One approach we considered is to correlate the aggregates with the `Id`, So
for the following input events:
{1, "CUST1", 100, $TS1}
{2, "CUST2", 5, $TS2}
{3, "CUST1", 15, $TS3}
We would generate the following(ignoring timestamp for now) into kafka:
{1, "TOTAL", 100} , {1, "GROUPBYCUSTTOTAL", "CUST1", 100}
{2, "TOTAL", 105} , {2, "GROUPBYCUSTTOTAL", "CUST2", 5}
{3, "TOTAL", 120} , {3, "GROUPBYCUSTTOTAL", "CUST1", 115}
And our application would read from kafka and process them.
So the flow looks like:
Application -- kafka---> flink --> kafka <--- Application
We want to keep our current request - response model i.e. we need all
continuous aggregates out for every ingested event into flink, before we
can further process the said event.
Unfortunately we don't see a way to do this in flink-SQL: As the aggregates
would not have the requestId for us to correlate with e.g. for the
following simple continuous query:
SELECT sum(amount) from EVENTS
We have tried doing this with flink-Datastream API: KeyedProcessFunction
with MapState per window, and collecting in processElement and using Kafka
sink.
A sample code for the windowing would look like the following:
public void processElement(Transaction transaction,
KeyedProcessFunction.Context context,
Collector collector) throws Exception {
()
collector.collect(new Aggregate(transaction.getId(),
context.getCurrentKey(), agg0, evTime));
}
If we were to use FlinkSQL instead, how would we accomplish this
functionality?
If there are any alternative approaches to accomplish this while
maintaining our invariant: every event must produce all aggregates that
consume the corresponding event, we would love to hear from the community.
Regards,
Jiten
--
*The information contained in this transmission (including any attachments)
is confidential and may be privileged. It is intended only for the use of
the individual or entity named above. If you are not the intended
recipient; dissemination, distribution, or copy of this communication is
strictly prohibited. If you have received this communication in error,
please erase all copies of this message and its attachments and notify me
immediately.*
