退订
退订 | | 伟波 | | weib...@126.com | 签名由网易邮箱大师定制
Re: Re: [ANNOUNCE] New PMC member: Xintong Song
Congratulations! Regards, Roman On Fri, Jun 18, 2021 at 4:40 AM Yu Li wrote: > > Congratulations, Xintong! > > Best Regards, > Yu > > > On Thu, 17 Jun 2021 at 15:23, Yuan Mei wrote: > > > Congratulations, Xintong :-) > > > > On Thu, Jun 17, 2021 at 11:57 AM Xingbo Huang wrote: > > > > > Congratulations, Xintong! > > > > > > Best, > > > Xingbo > > > > > > Yun Gao 于2021年6月17日周四 上午10:46写道： > > > > > > > Congratulations, Xintong! > > > > > > > > Best, > > > > Yun > > > > > > > > > > > > -- > > > > Sender:Jingsong Li > > > > Date:2021/06/17 10:41:22 > > > > Recipient:dev > > > > Theme:Re: [ANNOUNCE] New PMC member: Xintong Song > > > > > > > > Congratulations, Xintong! > > > > > > > > Best, > > > > Jingsong > > > > > > > > On Thu, Jun 17, 2021 at 10:26 AM Yun Tang wrote: > > > > > > > > > Congratulations, Xintong! > > > > > > > > > > Best > > > > > Yun Tang > > > > > > > > > > From: Leonard Xu > > > > > Sent: Wednesday, June 16, 2021 21:05 > > > > > To: dev (dev@flink.apache.org) > > > > > Subject: Re: [ANNOUNCE] New PMC member: Xintong Song > > > > > > > > > > > > > > > Congratulations, Xintong! > > > > > > > > > > > > > > > Best, > > > > > Leonard > > > > > > 在 2021年6月16日，20:07，Till Rohrmann 写道： > > > > > > > > > > > > Congratulations, Xintong! > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > On Wed, Jun 16, 2021 at 1:47 PM JING ZHANG > > > > wrote: > > > > > > > > > > > >> Congratulations, Xintong! > > > > > >> > > > > > >> > > > > > >> Jiayi Liao 于2021年6月16日周三 下午7:30写道： > > > > > >> > > > > > > > > > > < > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/> > > > > > Congratulations Xintong! > > > > > > > > > > On Wed, Jun 16, 2021 at 7:24 PM Nicholas Jiang < > > > programg...@163.com > > > > > > > > > > wrote: > > > > > > > > > > > Congratulations, Xintong! > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Sent from: > > > > > > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ > > > > > > > > > > > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > > > > -- > > > > Best, Jingsong Lee > > > > > > > > > > > > >
Re: [DISCUSS] Releasing Flink 1.11.4, 1.12.5, 1.13.2
Hi, Thanks for bringing this up. I think before releasing 1.12.x/1.13.x/1.14.x, it would be good to decide what to do with FLINK-23011 [1] and if there is a relatively easy fix, I would wait for it before releasing. Best, Piotrek [1] with https://issues.apache.org/jira/browse/FLINK-23011 pt., 18 cze 2021 o 16:35 Konstantin Knauf napisał(a): > Hi Dawid, > > Thank you for starting the discussion. I'd like to add > https://issues.apache.org/jira/browse/FLINK-23025 to the list for Flink > 1.13.2. > > Cheers, > > Konstantin > > On Fri, Jun 18, 2021 at 3:26 PM Dawid Wysakowicz > wrote: > > > Hi devs, > > > > Quite recently we pushed, in our opinion, quite an important fix[1] for > > unaligned checkpoints which disables UC for broadcast partitioning. > > Without the fix there might be some broadcast state corruption. > > Therefore we think it would be beneficial to release it soonish. What do > > you think? Do you have other issues in mind you'd like to have included > > in these versions. > > > > Would someone be willing to volunteer to help with the releases as a > > release manager? I guess there is a couple of spots to fill in here ;) > > > > Best, > > > > Dawid > > > > [1] https://issues.apache.org/jira/browse/FLINK-22815 > > > > > > > > -- > > Konstantin Knauf > > https://twitter.com/snntrable > > https://github.com/knaufk >
[jira] [Created] (FLINK-23041) Change local alignment timeout back to the global time out
Anton Kalashnikov created FLINK-23041: - Summary: Change local alignment timeout back to the global time out Key: FLINK-23041 URL: https://issues.apache.org/jira/browse/FLINK-23041 Project: Flink Issue Type: Bug Reporter: Anton Kalashnikov Assignee: Anton Kalashnikov Local alignment timeouts are very confusing and especially without timeout on the outputs, they can significantly delay timeouting to UC. Problematic case is when all CBs are received with long delay because of the back pressure, but they arrive at the same time. Alignment time can be low (milliseconds), while start delay is ~1 minute. In that case checkpoint doesn't timeout to UC and is passing the responsibility to timeout down the stream. So it is not so transparant for the user why and when AC switches to UC. As mentioned before, the start delay is not correlated with the alignment timeout because it doesn't take into account time in output buffer. the alignment time is not fully correlated with the alignment timeout because the alignment time doesn't take into account the barrier announcement. Based on this, there is the proposal to change the semantic of alignmentTimeout configuration to such meaning: *The time between the starting of checkpoint(on the checkpont coordinator) and the time when the checkpoint barrier will be received by task.* By this definition, we will have kind of global timeout which says that if the AC isn't finished for alignmentTimeout time it will be switched to UC. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] Releasing Flink 1.11.4, 1.12.5, 1.13.2
Hi Dawid, Thank you for starting the discussion. I'd like to add https://issues.apache.org/jira/browse/FLINK-23025 to the list for Flink 1.13.2. Cheers, Konstantin On Fri, Jun 18, 2021 at 3:26 PM Dawid Wysakowicz wrote: > Hi devs, > > Quite recently we pushed, in our opinion, quite an important fix[1] for > unaligned checkpoints which disables UC for broadcast partitioning. > Without the fix there might be some broadcast state corruption. > Therefore we think it would be beneficial to release it soonish. What do > you think? Do you have other issues in mind you'd like to have included > in these versions. > > Would someone be willing to volunteer to help with the releases as a > release manager? I guess there is a couple of spots to fill in here ;) > > Best, > > Dawid > > [1] https://issues.apache.org/jira/browse/FLINK-22815 > > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
[jira] [Created] (FLINK-23040) Consider ConfigOption fallback keys in FactoryUtil
Timo Walther created FLINK-23040: Summary: Consider ConfigOption fallback keys in FactoryUtil Key: FLINK-23040 URL: https://issues.apache.org/jira/browse/FLINK-23040 Project: Flink Issue Type: Sub-task Components: Table SQL / API Reporter: Timo Walther Assignee: Timo Walther We are currently not taking fallback keys into consideration when performing validation in FactoryUtil. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[DISCUSS] Releasing Flink 1.11.4, 1.12.5, 1.13.2
Hi devs, Quite recently we pushed, in our opinion, quite an important fix[1] for unaligned checkpoints which disables UC for broadcast partitioning. Without the fix there might be some broadcast state corruption. Therefore we think it would be beneficial to release it soonish. What do you think? Do you have other issues in mind you'd like to have included in these versions. Would someone be willing to volunteer to help with the releases as a release manager? I guess there is a couple of spots to fill in here ;) Best, Dawid [1] https://issues.apache.org/jira/browse/FLINK-22815 OpenPGP_signature Description: OpenPGP digital signature
[jira] [Created] (FLINK-23039) Support pluggable transports for HTTP endpoints
Igal Shilman created FLINK-23039:
Summary: Support pluggable transports for HTTP endpoints
Key: FLINK-23039
URL: https://issues.apache.org/jira/browse/FLINK-23039
Project: Flink
Issue Type: Improvement
Components: Stateful Functions
Reporter: Igal Shilman
Fix For: statefun-3.1.0
We've recently learned about a use case that requires using a custom client
that dispatches the HTTP requests (due to some internal reasons).
This can be a useful addition to further customizing the exact client code,
that suites the user's need. (for example adding company specific tracing
information)
This is technically feasible as well, as all it takes is:
1) provide an implementation of this
[interface|https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/reqreply/RequestReplyClient.java]
2) extend the ability to configure it from information present at the
module.yaml.
The current proposal is to add an optional "transport" section to endpoint
definition:
{code:java}
- endpoint:
meta:
kind: http
spec:
functions: com.foo.bar/*
transport:
provider_class: com.foo.bar.ClientProvider
some:
internal:
property: 123
urlPathTemplate: http://bar.foo.com:8080/functions/{function.name}
maxNumBatchRequests: 1 {code}
If the transport is not present we assume that the StateFun's pre-bundled
transport is present.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-23038) Broken link in Documentation Style Guide
Daisy Tsang created FLINK-23038: --- Summary: Broken link in Documentation Style Guide Key: FLINK-23038 URL: https://issues.apache.org/jira/browse/FLINK-23038 Project: Flink Issue Type: Bug Reporter: Daisy Tsang The link to the Flink Glossary is broken here: https://flink.apache.org/contributing/docs-style.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-23037) Document limitations of UC
Dawid Wysakowicz created FLINK-23037: Summary: Document limitations of UC Key: FLINK-23037 URL: https://issues.apache.org/jira/browse/FLINK-23037 Project: Flink Issue Type: Improvement Components: Documentation, Runtime / Checkpointing Reporter: Dawid Wysakowicz Assignee: Dawid Wysakowicz Fix For: 1.14.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-23036) org.apache.flink.util.FlinkRuntimeException: Error while adding data to RocksDB
guxiang created FLINK-23036:
---
Summary: org.apache.flink.util.FlinkRuntimeException: Error while
adding data to RocksDB
Key: FLINK-23036
URL: https://issues.apache.org/jira/browse/FLINK-23036
Project: Flink
Issue Type: Bug
Components: API / Core
Affects Versions: 1.13.1, 1.12.2, 1.9.4
Reporter: guxiang
I had a similar problem to this one
{quote}[http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Checkpoint-failed-because-of-TimeWindow-cannot-be-cast-to-VoidNamespace-td36310.html]
https://issues.apache.org/jira/browse/FLINK-18464
{quote}
I tested three versions of Flink, and all of them experienced this problem
1.9.0 / 1.12.2 / 1.13.1
This is because I defined a state （VoidNamespace） on a trigger and I used the
state in TimeWindow （WindowNamespace）.
Runs locally and does not use backendState.
When you run locally and do not use backendstate, the program runs normally.
However, when backendState is used, an error will be reported
{code:java}
//代码占位符
org.apache.flink.util.FlinkRuntimeException: Error while adding data to RocksDB
at
org.apache.flink.contrib.streaming.state.RocksDBValueState.update(RocksDBValueState.java:109)
at
com.sankuai.grocery.crm.data.mallorg.trigger.MessageProcessOnTimeStateTrigger.onEventTime(MessageProcessOnTimeStateTrigger.java:116)
at
com.sankuai.grocery.crm.data.mallorg.trigger.MessageProcessOnTimeStateTrigger.onEventTime(MessageProcessOnTimeStateTrigger.java:23)
at
org.apache.flink.streaming.runtime.operators.windowing.WindowOperator$Context.onEventTime(WindowOperator.java:944)
at
org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.onEventTime(WindowOperator.java:481)
at
org.apache.flink.streaming.api.operators.InternalTimerServiceImpl.advanceWatermark(InternalTimerServiceImpl.java:302)
at
org.apache.flink.streaming.api.operators.InternalTimeServiceManagerImpl.advanceWatermark(InternalTimeServiceManagerImpl.java:194)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.processWatermark(AbstractStreamOperator.java:626)
at
org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitWatermark(OneInputStreamTask.java:197)
at
org.apache.flink.streaming.runtime.streamstatus.StatusWatermarkValve.findAndOutputNewMinWatermarkAcrossAlignedChannels(StatusWatermarkValve.java:196)
at
org.apache.flink.streaming.runtime.streamstatus.StatusWatermarkValve.inputWatermark(StatusWatermarkValve.java:105)
at
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:206)
at
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:174)
at
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:396)
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:191)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:617)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:581)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:755)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:570)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassCastException:
org.apache.flink.streaming.api.windowing.windows.TimeWindow cannot be cast to
org.apache.flink.runtime.state.VoidNamespace
at
org.apache.flink.runtime.state.VoidNamespaceSerializer.serialize(VoidNamespaceSerializer.java:30)
at
org.apache.flink.contrib.streaming.state.RocksDBKeySerializationUtils.writeNameSpace(RocksDBKeySerializationUtils.java:78)
at
org.apache.flink.contrib.streaming.state.RocksDBSerializedCompositeKeyBuilder.serializeNamespace(RocksDBSerializedCompositeKeyBuilder.java:175)
at
org.apache.flink.contrib.streaming.state.RocksDBSerializedCompositeKeyBuilder.buildCompositeKeyNamespace(RocksDBSerializedCompositeKeyBuilder.java:112)
at
org.apache.flink.contrib.streaming.state.AbstractRocksDBState.serializeCurrentKeyWithGroupAndNamespace(AbstractRocksDBState.java:163)
at
org.apache.flink.contrib.streaming.state.RocksDBValueState.update(RocksDBValueState.java:106)
... 20 more
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Change in accumutors semantics with jobClient
Hi all, I did a fix some time ago regarding accumulators: the/JobClient.getAccumulators()/ was infinitely blocking in local environment for a streaming job (1). The change (2) consisted of giving the current accumulators value for the running job. And when fixing this in the PR, it appeared that I had to change the accumulators semantics with /JobClient/ and I just realized that I forgot to bring this back to the ML: Previously /JobClient/ assumed that getAccumulator() was called on a bounded pipeline and that the user wanted to acquire the *final accumulator values* after the job is finished. But now it returns the *current value of accumulators* immediately to be compatible with unbounded pipelines. If it is run on a bounded pipeline, then to get the final accumulator values after the job is finished, one needs to call /getJobExecutionResult().thenApply(JobExecutionResult::getAllAccumulatorResults)/ (1): https://issues.apache.org/jira/browse/FLINK-18685 (2): https://github.com/apache/flink/pull/14558# Cheers, Etienne
[jira] [Created] (FLINK-23035) Add explicit method to StateChangelogWriter to write metadata
Roman Khachatryan created FLINK-23035: - Summary: Add explicit method to StateChangelogWriter to write metadata Key: FLINK-23035 URL: https://issues.apache.org/jira/browse/FLINK-23035 Project: Flink Issue Type: Sub-task Components: Runtime / State Backends Reporter: Roman Khachatryan Fix For: 1.14.0 Currently, metadata is written to the state changelog using the same StateChangelogWriter.append() method as data. However, it doesn't belong to a specific group, and should be read first on recovery. Because of that, -1 is used. An explicit append() without keygroup would be less fragile (probably still using -1 under the hood). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-23034) NPE in JobDetailsDeserializer during the reading old version of ExecutionState
Anton Kalashnikov created FLINK-23034:
-
Summary: NPE in JobDetailsDeserializer during the reading old
version of ExecutionState
Key: FLINK-23034
URL: https://issues.apache.org/jira/browse/FLINK-23034
Project: Flink
Issue Type: Bug
Reporter: Anton Kalashnikov
Assignee: Anton Kalashnikov
There is no compatibility for ExecutionState:
{noformat}
java.lang.NullPointerExceptionjava.lang.NullPointerException at
org.apache.flink.runtime.messages.webmonitor.JobDetails$JobDetailsDeserializer.deserialize(JobDetails.java:308)
at
org.apache.flink.runtime.messages.webmonitor.JobDetails$JobDetailsDeserializer.deserialize(JobDetails.java:278)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4593)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3479)
at
org.apache.flink.runtime.messages.webmonitor.JobDetailsTest.testJobDetailsCompatibleUnmarshalling(JobDetailsTest.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) at
org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at
org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at
org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at
org.junit.runners.ParentRunner.run(ParentRunner.java:413) at
org.junit.runner.JUnitCore.run(JUnitCore.java:137) at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
at
com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
at
com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:221)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-23033) Support object array in Python DataStream API
Dian Fu created FLINK-23033: --- Summary: Support object array in Python DataStream API Key: FLINK-23033 URL: https://issues.apache.org/jira/browse/FLINK-23033 Project: Flink Issue Type: Improvement Components: API / Python Reporter: Dian Fu Fix For: 1.14.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] FLIP-129 (Update): Registering sources/sinks on Table API without SQL
Thanks everyone for the feedback! Since there haven't been any questions/discussions, I'll be moving this to a vote thread next week. Regards Ingo On Tue, Jun 15, 2021 at 5:36 PM JING ZHANG wrote: > Thanks Ingo for starting this discussion. > > > > Big +1 for this feature. Looking forward to the feature. > > > Best regards, > JING ZHANG > > Jark Wu 于2021年6月15日周二 下午3:48写道： > > > Thanks Ingo for picking up this FLIP. > > > > FLIP-129 is an important piece to have a complete Table SQL story, > > and users have been waiting for a long time. Let's finish it in this > > release! > > Your proposed changes look good to me. > > > > I also cc'd people who voted in previous FLIP-129. > > > > Best, > > Jark > > > > On Thu, 10 Jun 2021 at 19:46, Ingo Bürk wrote: > > > > > Hello everyone, > > > > > > we would like to pick up work on FLIP-129 which aims to improve the > Table > > > API by supporting the creation of sources / sinks without having to go > > > through SQL/DDL. This FLIP was approved a while ago, and some things > have > > > changed since then. We'd like to propose a few changes, see [1], before > > > starting work on it. > > > Our proposal is mainly motivated by reducing the scope in some parts to > > > improve maintainability and relying more on ConfigOptions being the > > single > > > source of truth. We also want to expose this functionality for > > > non-temporary tables. > > > > > > We'd like to open this for discussion to collect any feedback. Once the > > > discussion has stabilized I'll update the FLIP itself and start a new > > vote. > > > > > > [1] > > > > > > > > > https://docs.google.com/document/d/1tpirvF0u723QF005UrgdbvF-Tp0Jbg_qhlbda4mk7Ck/edit?usp=sharing > > > > > > > > > Regards > > > Ingo > > > > > >
Re: [DISCUSS] FLIP-171: Async Sink
Hi Danny, to add I'd propose to use the flink-connector-base package which has the rough equivalent on source-side SourceReaderBase [1]. Since it's such a handy base implementation, I'd like to see it directly in the main flink repository. For the actual connectors, I'm currently working on a proposal for a common connector repository under Flink umbrella. [1] https://github.com/AHeise/flink/blob/master/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/SourceReaderBase.java#L58-L58 On Wed, Jun 16, 2021 at 6:06 PM Hausmann, Steffen wrote: > Hi Danny, > > Right now, I'd expect the core of the Async Sink (without third party > dependencies) to live in its own submodule. For instance > `flink-connector-async` as part of `flink-connectors`. > > I'm currently planning to implement three different sinks to verify that > the design of the sink if flexible enough to support different services: > Amazon Kinesis Data Streams, Amazon Kinesis Data Firehose, and Amazon > DynamoDB. But I'm not sure where to actually put them. To keep is simple, > I'd start with a module that contains all AWS specific connectors. However, > it has the obvious disadvantage that if someone wants to use a single sink, > they would need to pull in all dependencies for all supported services that > are included in this module (mainly the AWS SDK for these services). But I > don't know how much of a problem that's going to be in practice. If the > respective jar grows too big because all the included dependencies, that's > certainly not going to work. But for now I'd just give it a try and then > start a discussion once I have more data to share. > > What's more interesting is whether that module should be part of the Flink > code base or live somewhere else. I'd be great to get some feedback from > the community on this. > > Regarding the Kinesis Data Streams sink, I fully agree that it would be > nice to remove the dependency to the KPL. So it seems to be desirable to > keep the existing and the new FLIP-171 based implementation in separate > modules. Otherwise people would be forced to pull in the KPL dependencies, > even if they are only using the new implementation. In addition, the new > implementation will not support the exact same functionality as the > existing one: the KPL implements a very optimized form of aggregation on a > shard level [1] by maintaining a mapping of shards and their respective key > spaces. The new implementation can in principle support aggregation as > well, but only on a partition key level, which may lead to less efficient > aggregation and higher latencies. > > Cheers, Steffen > > [1] > https://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-concepts.html#kinesis-kpl-concepts-aggretation > > > > On 15.06.21, 19:52, "Cranmer, Danny" > wrote: > > CAUTION: This email originated from outside of the organization. Do > not click links or open attachments unless you can confirm the sender and > know the content is safe. > > > > Hey Steffen, > > I have a few questions regarding the FLIP: > 1. Where do you expect the core code to live, would it be in an > existing module (say flink-clients) or would you introduce a new module? > 2. Which destination implementations do you intend to ship with this > FLIP? I see an example with Kinesis but you also list a bunch of other > candidates. > 3. For the Kinesis implementation, would you add the Sink to the > existing flink-connector-kinesis repo, or create a new module? Reason I ask > is that the existing Kinesis Sink depends on KPL and has a heavy transitive > dependency chain, removing this would substantially reduce application size > and clean the dependency chain > > Thanks, > > On 10/06/2021, 09:09, "Hausmann, Steffen" > wrote: > > CAUTION: This email originated from outside of the organization. > Do not click links or open attachments unless you can confirm the sender > and know the content is safe. > > > > Hey Piotrek, > > Thanks for your comments on the FLIP. I'll address your second > question first, as I think it's more central to this FLIP. Just looking at > the AWS ecosystem, there are several sinks with overlapping functionality. > I've chosen AWS sinks here because I'm most familiar with those, but a > similar argument applies more generically for destination that support > async ingest. > > There is, for instance, a sink for Amazon Kinesis Data Streams > that is part of Apache Flink [1], a sink for Amazon Kinesis Data Firehose > [2], a sink for Amazon DynamoDB [3], and a sink for Amazon Timestream [4]. > All these sinks have implemented their own mechanisms for batching, > persisting, and retrying events. And I'm not sure if all of them properly > participate in checkpointing. [3] even seems to closely mirror [1] as it > contains references to the Kinesis Producer Library, which is unrelated to > Amazon DynamoDB. > > These sinks
[jira] [Created] (FLINK-23032) Refactor HiveSource to make it usable in data stream job
Rui Li created FLINK-23032: -- Summary: Refactor HiveSource to make it usable in data stream job Key: FLINK-23032 URL: https://issues.apache.org/jira/browse/FLINK-23032 Project: Flink Issue Type: Improvement Components: Connectors / Hive Reporter: Rui Li Assignee: Rui Li Fix For: 1.14.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-23031) Support to emit window result with periodic or non_periodic
Aitozi created FLINK-23031: -- Summary: Support to emit window result with periodic or non_periodic Key: FLINK-23031 URL: https://issues.apache.org/jira/browse/FLINK-23031 Project: Flink Issue Type: Improvement Components: Table SQL / Planner Reporter: Aitozi -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-23030) PartitionRequestClientFactory#createPartitionRequestClient should throw when network failure
Jin Xing created FLINK-23030: Summary: PartitionRequestClientFactory#createPartitionRequestClient should throw when network failure Key: FLINK-23030 URL: https://issues.apache.org/jira/browse/FLINK-23030 Project: Flink Issue Type: Bug Components: Runtime / Network Reporter: Jin Xing In current _PartitionRequestClientFactory#createPartitionRequestClient_, _ChannelFuture#await()_ is invoked, thus to build a connection to remote synchronously. But with the doc of _io.netty.util.concurrent.Future_ [1] and its implementation _io.netty.channel.DefaultChannelPromise_ [2], _ChannelFuture#await()_ never throws when completed with failure. I guess what Flink needs is _ChannelFuture#sync()._ [1][https://netty.io/4.1/api/io/netty/util/concurrent/class-use/Future.html] [2] https://github.com/netty/netty/blob/4.1/transport/src/main/java/io/netty/channel/DefaultChannelPromise.java -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-23029) Extend PluginLoader to allow closing of backing ClassLoader
Chesnay Schepler created FLINK-23029: Summary: Extend PluginLoader to allow closing of backing ClassLoader Key: FLINK-23029 URL: https://issues.apache.org/jira/browse/FLINK-23029 Project: Flink Issue Type: Improvement Components: Runtime / Coordination Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.14.0 The PluginLoader is currently never closing the backing classloader, which means that plugins remain loaded until the application shuts down. This for example makes it impossible to use the plugins in the mini cluster, because they would be loaded multiple times. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-23028) Improve documentation for pages of SQL.
Roc Marshal created FLINK-23028: --- Summary: Improve documentation for pages of SQL. Key: FLINK-23028 URL: https://issues.apache.org/jira/browse/FLINK-23028 Project: Flink Issue Type: Improvement Components: Documentation Reporter: Roc Marshal Wrong style in [https://ci.apache.org/projects/flink/flink-docs-master/zh/docs/dev/table/sql/create/#create-function] section. Wrong style in [https://ci.apache.org/projects/flink/flink-docs-master/zh/docs/dev/table/sql/drop/#drop-function] section. Wrong style in [https://ci.apache.org/projects/flink/flink-docs-master/zh/docs/dev/table/sql/alter/#alter-function] section. Add the description about drop catalog in the page of [https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/sql/drop/] . 'catloag' of [https://ci.apache.org/projects/flink/flink-docs-master/zh/docs/dev/table/sql/use/#use-catloag] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-23027) Move chill dependency to flink-scala
Chesnay Schepler created FLINK-23027: Summary: Move chill dependency to flink-scala Key: FLINK-23027 URL: https://issues.apache.org/jira/browse/FLINK-23027 Project: Flink Issue Type: Sub-task Components: API / Type Serialization System Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.14.0 flink-runtime contains a few serializer classes that rely on scala. We could move these to flink-scala, which already depends on scala anyway, and is also bundled in flink-dist. -- This message was sent by Atlassian Jira (v8.3.4#803005)
