[DISCUSS]Rethink the rescale operation, can we do it async

Currently, the rescale operation is to stop the whole job and restart it with different parrellism. But the rescale operation cost a lot and took lots of time to recover if the state size is quite big. And a long-time rescale might cause other problems like latency increase and back pressure. For

Re: Kinesis consumer e2e test

Hi Thomas, I think Stefan Richter is also working on the Kinesis end-to-end test, and seems to be planning to implement it against a real Kinesis service instead of Kinesalite. Perhaps efforts should be synced here. Cheers, Gordon On Thu, Nov 8, 2018 at 1:38 PM Thomas Weise wrote: > Hi, > >

Kinesis consumer e2e test

Hi, I'm planning to add an end-to-end test for the Kinesis consumer. We have done something similar at Lyft, using Kinesalite, which can be run as Docker container. I see that some tests already make use of Docker, so we can assume it to be present in the target environment(s)? I also found the

StreamingFileSink Bug? Committing results on stream close

Hi all, Just run into a bit of a problem and I am not sure what the behavior should be and if this should be considered a bug? Or if there is some other way this should be handled? I have a streaming job with a stream that eventually closes, this job sinks to a StreamingFileSink. The problem I

[jira] [Created] (FLINK-10817) Upgrade presto dependency to support path-style access

Adam Lamar created FLINK-10817: -- Summary: Upgrade presto dependency to support path-style access Key: FLINK-10817 URL: https://issues.apache.org/jira/browse/FLINK-10817 Project: Flink Issue

[jira] [Created] (FLINK-10816) Fix LockableTypeSerializer.duplicate()

Stefan Richter created FLINK-10816: -- Summary: Fix LockableTypeSerializer.duplicate() Key: FLINK-10816 URL: https://issues.apache.org/jira/browse/FLINK-10816 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-10815) Rethink the rescale operation, can we do it async

Shimin Yang created FLINK-10815: --- Summary: Rethink the rescale operation, can we do it async Key: FLINK-10815 URL: https://issues.apache.org/jira/browse/FLINK-10815 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-27: Refactor Source Interface

Hi Becket, With my proposal, both of your examples would have to be solved by the connector and solution to both problems would be the same: Pretend that connector is never blocked (`isBlocked() { return NOT_BLOCKED; }`) and implement `getNextElement()` in blocking fashion (or semi blocking

Re: [DISCUSS] Table API Enhancement Outline

Hi Fabian, I think that the key question you raised is if we allow extra parameters in the methods map/flatMap/agg/flatAgg. I can see why allowing that may appear more convenient in some cases. However, it might also cause some confusions if we do that. For example, do we allow multiple UDFs in

Re: [DISCUSS] Enhancing the functionality and productivity of Table API

Hi Piotr: I want to clarify one thing first: I think that we will keep the interoperability between TableAPI and DataStream in any case. So user can switch between the two whenever needed. Given that, it would still be very helpful that users can use one API to achieve most of what they do.

[jira] [Created] (FLINK-10814) Kafka examples modules need scala suffix

Chesnay Schepler created FLINK-10814: Summary: Kafka examples modules need scala suffix Key: FLINK-10814 URL: https://issues.apache.org/jira/browse/FLINK-10814 Project: Flink Issue Type:

Re: JIRA assignment

Thanks Till! -Joey > On Nov 7, 2018, at 02:32, Till Rohrmann wrote: > > I've given you permissions and assigned you to this ticket. > > Cheers, > Till > >> On Tue, Nov 6, 2018 at 7:25 PM Joey Echeverria wrote: >> I submitted a PR[1] for FLINK-10213[2] awhile ago, but I couldn't assign >>

Re: [VOTE] Release 1.7.0, release candidate #1

I hereby cancel the release vote because of the Scala suffix problems. I will create the next RC in the next days. Until then, please continue testing with the current release candidate. Cheers, Till On Wed, Nov 7, 2018 at 2:39 PM Till Rohrmann wrote: > Thanks for spotting and addressing the

[jira] [Created] (FLINK-10813) Automatically seach for missing scala suffixes

Chesnay Schepler created FLINK-10813: Summary: Automatically seach for missing scala suffixes Key: FLINK-10813 URL: https://issues.apache.org/jira/browse/FLINK-10813 Project: Flink Issue

Re: [VOTE] Release 1.7.0, release candidate #1

Thanks for spotting and addressing the Scala problem Chesnay. The corresponding JIRA issue is https://issues.apache.org/jira/browse/FLINK-10811. Cheers, Till On Wed, Nov 7, 2018 at 12:36 PM Chesnay Schepler wrote: > This isn't quite correct (as test-scoped dependencies are not > transitive,

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

Hi Xuefu and all, Thanks for sharing this design document! I'm very much in favor of restructuring / reworking the catalog handling in Flink SQL as outlined in the document. Most changes described in the design document seem to be rather general and not specifically related to the Hive

[jira] [Created] (FLINK-10812) Snapshot deployment broken for master

Chesnay Schepler created FLINK-10812: Summary: Snapshot deployment broken for master Key: FLINK-10812 URL: https://issues.apache.org/jira/browse/FLINK-10812 Project: Flink Issue Type:

[jira] [Created] (FLINK-10811) Hcatalog modules needs scala suffix

Chesnay Schepler created FLINK-10811: Summary: Hcatalog modules needs scala suffix Key: FLINK-10811 URL: https://issues.apache.org/jira/browse/FLINK-10811 Project: Flink Issue Type: Bug

Re: [VOTE] Release 1.7.0, release candidate #1

This isn't quite correct (as test-scoped dependencies are not transitive, but all compile dependencies still are, even for the test-jar). But effectively this means we don't need additional rules for test-jars as compile dependencies already have to be taken care of separately from tests

[jira] [Created] (FLINK-10810) Update DataStreamUtils.reinterpretAsKeyedStream to state limitations

Till Rohrmann created FLINK-10810: - Summary: Update DataStreamUtils.reinterpretAsKeyedStream to state limitations Key: FLINK-10810 URL: https://issues.apache.org/jira/browse/FLINK-10810 Project:

Re: Replace email address destination

Hi, you can unsubscribe and resubscribe to the mailing list as described here [1]. Best, Fabian [1] https://flink.apache.org/community.html#mailing-lists Am Mi., 7. Nov. 2018 um 12:29 Uhr schrieb florent tragni < florent.tra...@precodata.com>: > Could you replace my email address by

Replace email address destination

Could you replace my email address by nore...@precodata.com ? Thanks. -- Florent Tragni

Re: [VOTE] Release 1.7.0, release candidate #1

I looked into this issue and my conclusion was that test-jars don't pull in transitive dependencies when you depend on them. I verified this with an example maven project where I also verified that a test-jar built with Scala 2.12 works on a project that uses Scala 2.11. On the hcatalog

Re: JIRA assignment

I've given you permissions and assigned you to this ticket. Cheers, Till On Tue, Nov 6, 2018 at 7:25 PM Joey Echeverria wrote: > I submitted a PR[1] for FLINK-10213[2] awhile ago, but I couldn't assign > the issue to myself or update it's status in JIRA. Is there anything I > need to do to get

Re: [DISCUSS] FLIP-27: Refactor Source Interface

Hi Piotr, I might have misunderstood you proposal. But let me try to explain my concern. I am thinking about the following case: 1. a reader has the following two interfaces, boolean isBlocked() T getNextElement() 2. the implementation of getNextElement() is non-blocking. 3. The reader is

Re: [VOTE] Release 1.7.0, release candidate #1

What was the conclusion in regards to modules requiring a scala-suffix if their test-jar depends on scala-infected modules? (Which basically affects all modules) Beyond that, the hcatalog connector has a dependency on flink-hadoop-compatibility_2.12, and should thus also have a scala suffix.

[jira] [Created] (FLINK-10809) Using DataStreamUtils.reinterpretAsKeyedStream produces corrupted keyed state after restore

Dawid Wysakowicz created FLINK-10809: Summary: Using DataStreamUtils.reinterpretAsKeyedStream produces corrupted keyed state after restore Key: FLINK-10809 URL:

Re: [DISCUSS] Table API Enhancement Outline

Hi, * Re emit: I think we should start with a well understood semantics of full replacement. This is how the other agg functions work. As was said before, there are open questions regarding an append mode (checkpointing, whether supporting retractions or not and if yes how to declare them, ...).

Re: [DISCUSS] Task speculative execution for Flink batch

Thanks so much for your all feedbacks! Yes, as mentioned above by Jin Sun, the design currently targets batch to explore the general framework and basic modules. The strategy could be also applied to stream with some extended code, for example, the result commitment. Jin Sun 于2018年11月7日周三

Re: [DISCUSS] FLIP-27: Refactor Source Interface

Hi, a) > BTW, regarding the isBlock() method, I have a few more questions. 21, Is a > method isReady() with boolean as a return value > equivalent? Personally I found it is a little bit confusing in what is > supposed to be returned when the future is completed. 22. if > the implementation of