[jira] [Created] (FLINK-6770) Add excludeSerializersInCheckpoints to ExeuctionConfig

2017-05-30 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6770: -- Summary: Add excludeSerializersInCheckpoints to ExeuctionConfig Key: FLINK-6770 URL: https://issues.apache.org/jira/browse/FLINK-6770 Project: Flink

Re: [VOTE] Release Apache Flink 1.3.0 (RC2)

2017-05-26 Thread Tzu-Li (Gordon) Tai
10 AM, Tzu-Li (Gordon) Tai wrote: > I think https://issues.apache.org/jira/browse/FLINK-6714 is most likely a > blocker. > It basically makes most of our connectors that are not part of the binary > distribution and uses Java serialization for their operator state unusable >

[jira] [Created] (FLINK-6719) Add details about fault-tolerance of timers to ProcessFunction docs

2017-05-25 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6719: -- Summary: Add details about fault-tolerance of timers to ProcessFunction docs Key: FLINK-6719 URL: https://issues.apache.org/jira/browse/FLINK-6719

Re: [VOTE] Release Apache Flink 1.3.0 (RC2)

2017-05-25 Thread Tzu-Li (Gordon) Tai
I think https://issues.apache.org/jira/browse/FLINK-6714 is most likely a blocker. It basically makes most of our connectors that are not part of the binary distribution and uses Java serialization for their operator state unusable in standalone mode. There's a pending PR for the fix: https://g

[jira] [Created] (FLINK-6714) Operator state backend should set user classloader as context classloader when snapshotting

2017-05-25 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6714: -- Summary: Operator state backend should set user classloader as context classloader when snapshotting Key: FLINK-6714 URL: https://issues.apache.org/jira/browse/FLINK

[jira] [Created] (FLINK-6713) Document how to allow multiple Kafka consumers / producers to authenticate using different credentials

2017-05-24 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6713: -- Summary: Document how to allow multiple Kafka consumers / producers to authenticate using different credentials Key: FLINK-6713 URL: https://issues.apache.org/jira

[jira] [Created] (FLINK-6712) Bump Kafka010 version to 0.10.2.0

2017-05-24 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6712: -- Summary: Bump Kafka010 version to 0.10.2.0 Key: FLINK-6712 URL: https://issues.apache.org/jira/browse/FLINK-6712 Project: Flink Issue Type

Re: [DISCUSS] Backwards compatibility policy.

2017-05-21 Thread Tzu-Li (Gordon) Tai
Hi Kostas, Thanks for bringing this up! I think it is reasonable to keep this coherent with our timely-based release model guarantees. With the timely-based release model, there is a guarantee that the current latest major version and the previous one is supported. For example, upon releasing 1

[jira] [Created] (FLINK-6653) Avoid directly serializing AWS's Shard class in Kinesis consumer's checkpoints

2017-05-21 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6653: -- Summary: Avoid directly serializing AWS's Shard class in Kinesis consumer's checkpoints Key: FLINK-6653 URL: https://issues.apache.org/jira/browse/

Re: [DISCUSS] Release 1.3.0 RC1 (Non voting, testing release candidate)

2017-05-18 Thread Tzu-Li (Gordon) Tai
Hi Robert, There is one last pending fix for the serializer upgrades feature:  https://issues.apache.org/jira/browse/FLINK-6482. Pending PR: https://github.com/apache/flink/pull/3937. I can’t say its a complete blocker, but since it will affect serialization format of checkpoints, it would be be

[jira] [Created] (FLINK-6608) Relax Kerberos login contexts parsing by trimming whitespaces in contexts list

2017-05-17 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6608: -- Summary: Relax Kerberos login contexts parsing by trimming whitespaces in contexts list Key: FLINK-6608 URL: https://issues.apache.org/jira/browse/FLINK-6608

[jira] [Created] (FLINK-6607) Confusing statement in docs that Kafka 010 consumer does not require timestamp extractors

2017-05-16 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6607: -- Summary: Confusing statement in docs that Kafka 010 consumer does not require timestamp extractors Key: FLINK-6607 URL: https://issues.apache.org/jira/browse/FLINK

[jira] [Created] (FLINK-6600) Add key serializer's config snapshot to KeyedBackendSerializationProxy

2017-05-16 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6600: -- Summary: Add key serializer's config snapshot to KeyedBackendSerializationProxy Key: FLINK-6600 URL: https://issues.apache.org/jira/browse/FLINK

[jira] [Created] (FLINK-6566) Narrow down interface for compatibility hook method in VersionedIOReadableWritable

2017-05-12 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6566: -- Summary: Narrow down interface for compatibility hook method in VersionedIOReadableWritable Key: FLINK-6566 URL: https://issues.apache.org/jira/browse/FLINK-6566

[jira] [Created] (FLINK-6565) Improve error messages for state restore failures

2017-05-12 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6565: -- Summary: Improve error messages for state restore failures Key: FLINK-6565 URL: https://issues.apache.org/jira/browse/FLINK-6565 Project: Flink

[jira] [Created] (FLINK-6520) FlinkKafkaConsumer09+ does not overwrite props to not auto commit offsets when commit mode is OffsetCommitMode.ON_CHECKPOINTS

2017-05-10 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6520: -- Summary: FlinkKafkaConsumer09+ does not overwrite props to not auto commit offsets when commit mode is OffsetCommitMode.ON_CHECKPOINTS Key: FLINK-6520 URL: https

Re: interested in volunteering at Flink ML/Stream ML Project

2017-05-09 Thread Tzu-Li (Gordon) Tai
Hi Alex, I would start searching in the Flink JIRA for tickets tagged with the component “Machine Learning Library”, and once you find a ticket you’ll like to work on, ask for the JIRA to be assigned to you on the mailing lists. From then on you can also assign any other tasks you’ll like to wo

[jira] [Created] (FLINK-6503) Refactor KinesisDataFetcher to separate concerns for shard discovery

2017-05-09 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6503: -- Summary: Refactor KinesisDataFetcher to separate concerns for shard discovery Key: FLINK-6503 URL: https://issues.apache.org/jira/browse/FLINK-6503

[jira] [Created] (FLINK-6482) Add nested serializers into configuration snapshots of composite serializers

2017-05-08 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6482: -- Summary: Add nested serializers into configuration snapshots of composite serializers Key: FLINK-6482 URL: https://issues.apache.org/jira/browse/FLINK-6482

[jira] [Created] (FLINK-6478) Add documentation on how to upgrade serializers for managed state

2017-05-07 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6478: -- Summary: Add documentation on how to upgrade serializers for managed state Key: FLINK-6478 URL: https://issues.apache.org/jira/browse/FLINK-6478 Project

Re: [DISCUSS] Feature Freeze

2017-05-02 Thread Tzu-Li (Gordon) Tai
Hi, I have 2 parts to discuss about delaying the feature freeze for a bit: 1. Allow upgrades to state serializers (FLINK-6178) The feature is split up into 2 PRs, one of which is already open and the second one coming up very soon. PR #1 reconfigurable TypeSerializers: https://github.com/apache/

[jira] [Created] (FLINK-6425) Integrate serializer reconfiguration into state restore flow to activate serializer upgrades

2017-05-02 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6425: -- Summary: Integrate serializer reconfiguration into state restore flow to activate serializer upgrades Key: FLINK-6425 URL: https://issues.apache.org/jira/browse/FLINK

[jira] [Created] (FLINK-6363) Document precedence rules of Kryo serializer registrations

2017-04-23 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6363: -- Summary: Document precedence rules of Kryo serializer registrations Key: FLINK-6363 URL: https://issues.apache.org/jira/browse/FLINK-6363 Project: Flink

[jira] [Created] (FLINK-6349) Enforce per-subtask record ordering on resharding for FlinkKinesisConsumer

2017-04-20 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6349: -- Summary: Enforce per-subtask record ordering on resharding for FlinkKinesisConsumer Key: FLINK-6349 URL: https://issues.apache.org/jira/browse/FLINK-6349

[jira] [Created] (FLINK-6348) Migrate from Java serialization for RollingSink's state

2017-04-20 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6348: -- Summary: Migrate from Java serialization for RollingSink's state Key: FLINK-6348 URL: https://issues.apache.org/jira/browse/FLINK-6348 Project:

[jira] [Created] (FLINK-6347) Migrate from Java serialization for MessageAcknowledgingSourceBase's state

2017-04-20 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6347: -- Summary: Migrate from Java serialization for MessageAcknowledgingSourceBase's state Key: FLINK-6347 URL: https://issues.apache.org/jira/browse/FLINK

[jira] [Created] (FLINK-6346) Migrate from Java serialization for GenericWriteAheadSink's state

2017-04-20 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6346: -- Summary: Migrate from Java serialization for GenericWriteAheadSink's state Key: FLINK-6346 URL: https://issues.apache.org/jira/browse/FLINK-6346 Pr

[jira] [Created] (FLINK-6345) Migrate from Java serialization for ContinuousFileReaderOperator's state

2017-04-20 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6345: -- Summary: Migrate from Java serialization for ContinuousFileReaderOperator's state Key: FLINK-6345 URL: https://issues.apache.org/jira/browse/FLINK

[jira] [Created] (FLINK-6344) Migrate from Java serialization for `BucketingSink`'s state

2017-04-20 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6344: -- Summary: Migrate from Java serialization for `BucketingSink`'s state Key: FLINK-6344 URL: https://issues.apache.org/jira/browse/FLINK-6344 Project:

[jira] [Created] (FLINK-6343) Migrate from discouraged Java serialization for all sources / sinks

2017-04-20 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6343: -- Summary: Migrate from discouraged Java serialization for all sources / sinks Key: FLINK-6343 URL: https://issues.apache.org/jira/browse/FLINK-6343

[jira] [Created] (FLINK-6324) Refine state access methods in OperatorStateStore

2017-04-18 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6324: -- Summary: Refine state access methods in OperatorStateStore Key: FLINK-6324 URL: https://issues.apache.org/jira/browse/FLINK-6324 Project: Flink

[jira] [Created] (FLINK-6316) Kinesis connector docs should mention that FlinkKinesisConsumer does not provide strong ordering guarantees on resharding

2017-04-17 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6316: -- Summary: Kinesis connector docs should mention that FlinkKinesisConsumer does not provide strong ordering guarantees on resharding Key: FLINK-6316 URL: https

Re: Jobmanager drops upon submitting a jar

2017-04-16 Thread Tzu-Li (Gordon) Tai
Hi Amir, What do you mean by “modernizing” the FlinkKafkaConsumer implementation? Could you explain a bit more? Thanks :) Chiming in just to check if there’s any issue we need to be aware of .. Cheers, Gordon On 14 April 2017 at 5:35:43 AM, amir bahmanyari (amirto...@yahoo.com.invalid) wrote:

[jira] [Created] (FLINK-6311) NPE in FlinkKinesisConsumer if source was closed before run

2017-04-16 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6311: -- Summary: NPE in FlinkKinesisConsumer if source was closed before run Key: FLINK-6311 URL: https://issues.apache.org/jira/browse/FLINK-6311 Project: Flink

Re: Possible bug in Kafka producer partitioning logic

2017-04-11 Thread Tzu-Li (Gordon) Tai
> Do you have any suggestions how to get around this? > > Gyula > > Tzu-Li (Gordon) Tai ezt írta (időpont: 2017. ápr. > 10., H, 15:57): > > I would prefer to make this a blocker for a future bugfix actually, and > not 1.2.1. > > The reason is that

Re: Possible bug in Kafka producer partitioning logic

2017-04-10 Thread Tzu-Li (Gordon) Tai
changes into releases. On April 10, 2017 at 6:46:29 AM, Gyula Fóra (gyula.f...@gmail.com) wrote: Thanks for checking this out. I would say this is definitely a blocking issue for the bugfix release, what do you think? Gyula Tzu-Li (Gordon) Tai ezt írta (időpont: 2017. ápr. 10., H, 15

Re: Possible bug in Kafka producer partitioning logic

2017-04-10 Thread Tzu-Li (Gordon) Tai
Hi Gyula, Yes, I think the semantics of the Partitioner interface is a bit off. The `numPartitions` value ideally should be the number of partitions of the `targetTopic`. Here’s a JIRA I just filed to track the issue:  https://issues.apache.org/jira/browse/FLINK-6288. Cheers, Gordon On April 1

[jira] [Created] (FLINK-6288) FlinkKafkaProducer's custom Partitioner is always invoked with number of partitions of default topic

2017-04-10 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6288: -- Summary: FlinkKafkaProducer's custom Partitioner is always invoked with number of partitions of default topic Key: FLINK-6288 URL: https://issues.apache.org

[jira] [Created] (FLINK-6258) Deprecate ListCheckpointed interface for managed operator state

2017-04-03 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6258: -- Summary: Deprecate ListCheckpointed interface for managed operator state Key: FLINK-6258 URL: https://issues.apache.org/jira/browse/FLINK-6258 Project

[jira] [Created] (FLINK-6191) Make non-primitive, internal built-in serializers reconfigurable

2017-03-26 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6191: -- Summary: Make non-primitive, internal built-in serializers reconfigurable Key: FLINK-6191 URL: https://issues.apache.org/jira/browse/FLINK-6191 Project

[jira] [Created] (FLINK-6190) Write "Serializer Configurations" metainfo along with state

2017-03-26 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6190: -- Summary: Write "Serializer Configurations" metainfo along with state Key: FLINK-6190 URL: https://issues.apache.org/jira/browse/FLINK-6190 Proj

Re: [DISCUSS] Release Flink 1.1.5 / Flink 1.2.1

2017-03-24 Thread Tzu-Li (Gordon) Tai
ase in the end. On Fri, Mar 24, 2017 at 7:02 AM, Tzu-Li (Gordon) Tai wrote: > Update for 1.2.1: > > The last fix was just merged! > > Since nobody else seems interested in managing 1.2.1, I can also help with > this one :) > I’ll create the release candi

[jira] [Created] (FLINK-6178) Allow upgrades to state serializers

2017-03-24 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6178: -- Summary: Allow upgrades to state serializers Key: FLINK-6178 URL: https://issues.apache.org/jira/browse/FLINK-6178 Project: Flink Issue Type

Re: [DISCUSS] Release Flink 1.1.5 / Flink 1.2.1

2017-03-23 Thread Tzu-Li (Gordon) Tai
1.2.1. Status: PR review pending - https://github.com/apache/flink/pull/3587 On March 22, 2017 at 12:23:03 AM, Tzu-Li (Gordon) Tai (tzuli...@apache.org) wrote: Update for Flink 1.2.1: There’s only one PR pending that is LGTM - https://issues.apache.org/jira/browse/FLINK-6084 Fix for Cassandra

[ANNOUNCE] Apache Flink 1.1.5 Released

2017-03-23 Thread Tzu-Li (Gordon) Tai
The Apache Flink community is pleased to announce the availability of Flink 1.1.5, which is the next bugfix release for the 1.1 series. The official release announcement: https://flink.apache.org/news/2017/03/23/release-1.1.5.html Release binaries: http://apache.lauf-forum.at/flink/flink-1.1.5 Fo

[RESULT] [VOTE] Release Apache Flink 1.1.5 (RC1)

2017-03-22 Thread Tzu-Li (Gordon) Tai
celling the vote, but then I realized that since Flink 0.10 >> (november 2015) we never released the module: >> http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22org. >> apache.flink%22%20AND%20a%3A%22flink-java8%22 >> >> >> On Fri, Mar 17, 2017 at

Re: [DISCUSS] Release Flink 1.1.5 / Flink 1.2.1

2017-03-21 Thread Tzu-Li (Gordon) Tai
- https://github.com/apache/flink/pull/3587 On March 22, 2017 at 12:23:03 AM, Tzu-Li (Gordon) Tai (tzuli...@apache.org) wrote: Update for Flink 1.2.1: There’s only one PR pending that is LGTM - https://issues.apache.org/jira/browse/FLINK-6084 Fix for Cassandra connector dropping metrics-core

Re: [DISCUSS] Release Flink 1.1.5 / Flink 1.2.1

2017-03-21 Thread Tzu-Li (Gordon) Tai
ompermaier wrote: > I propose to fix https://issues.apache.org/jira/browse/FLINK-6103 before > issue a release > > On Fri, Mar 17, 2017 at 8:12 AM, Ufuk Celebi wrote: > > > Cool! Thanks for taking care of this Gordon :-) > > > > On Fri, Mar 17, 2017 a

Re: [ANNOUNCE] New committer: Theodore Vasiloudis

2017-03-21 Thread Tzu-Li (Gordon) Tai
Welcome Theodore! Great to have you on board :-) On March 21, 2017 at 4:35:35 PM, Robert Metzger (rmetz...@apache.org) wrote: Hi everybody, On behalf of the PMC I am delighted to announce Theodore Vasiloudis as a new Flink committer! Theo has been a community member for a very long time and

[jira] [Created] (FLINK-6139) Documentation for building / preparing Flink for MapR

2017-03-20 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6139: -- Summary: Documentation for building / preparing Flink for MapR Key: FLINK-6139 URL: https://issues.apache.org/jira/browse/FLINK-6139 Project: Flink

[jira] [Created] (FLINK-6109) Add "consumer lag" report metric to FlinkKafkaConsumer

2017-03-18 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6109: -- Summary: Add "consumer lag" report metric to FlinkKafkaConsumer Key: FLINK-6109 URL: https://issues.apache.org/jira/browse/FLINK-6109 Proj

[VOTE] Release Apache Flink 1.1.5 (RC1)

2017-03-17 Thread Tzu-Li (Gordon) Tai
Dear Flink community,  Please vote on releasing the following candidate as Apache Flink version 1.1.5. The commit to be voted on:  ed18e97 (http://git-wip-us.apache.org/repos/asf/flink/commit/ed18e97)  Branch:  release-1.1.5-rc1  (https://git1-us-west.apache.org/repos/asf/flink/repo?p=flink.git;

Re: [DISCUSS] Release Flink 1.1.5 / Flink 1.2.1

2017-03-16 Thread Tzu-Li (Gordon) Tai
t;> Status: PR > >>>>>>> > >>>>>>> > >>>>>>>> Am 14.03.2017 um 15:15 schrieb Till Rohrmann < > trohrm...@apache.org > >>>>> : > >>>>>>>> > >>&

Re: [DISCUSS] Release Flink 1.1.5 / Flink 1.2.1

2017-03-15 Thread Tzu-Li (Gordon) Tai
annot retrieve the >>>>>> completed checkpoint from the meta data state handle retrieved from >>>>>> ZooKeeper. This can, for example, happen if the meta data is >> deleted. >>>>>> Checkpoints with unretrievable state handles are

[DISCUSS] Release Flink 1.1.5 / Flink 1.2.1

2017-03-14 Thread Tzu-Li (Gordon) Tai
Hi all! I would like to start a discussion for the next bugfix release for 1.1.x and 1.2.x. There’s been quite a few critical fixes for bugs in both the releases recently, and I think they deserve a bugfix release soon. Most of the bugs were reported by users. I’m starting the discussion for bo

[jira] [Created] (FLINK-6038) Add links to Apache Bahir Flink streaming connectors

2017-03-14 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6038: -- Summary: Add links to Apache Bahir Flink streaming connectors Key: FLINK-6038 URL: https://issues.apache.org/jira/browse/FLINK-6038 Project: Flink

[jira] [Created] (FLINK-6025) User code ClassLoader not used when KryoSerializer fallbacks to serialization for copying

2017-03-11 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6025: -- Summary: User code ClassLoader not used when KryoSerializer fallbacks to serialization for copying Key: FLINK-6025 URL: https://issues.apache.org/jira/browse/FLINK

[jira] [Created] (FLINK-6006) Kafka Consumer can lose state if queried partition list is incomplete on restore

2017-03-09 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6006: -- Summary: Kafka Consumer can lose state if queried partition list is incomplete on restore Key: FLINK-6006 URL: https://issues.apache.org/jira/browse/FLINK-6006

[jira] [Created] (FLINK-6004) Allow FlinkKinesisConsumer to skip corrupted messages

2017-03-08 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6004: -- Summary: Allow FlinkKinesisConsumer to skip corrupted messages Key: FLINK-6004 URL: https://issues.apache.org/jira/browse/FLINK-6004 Project: Flink

[jira] [Created] (FLINK-5991) Expose Broadcast Operator State through public APIs

2017-03-07 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-5991: -- Summary: Expose Broadcast Operator State through public APIs Key: FLINK-5991 URL: https://issues.apache.org/jira/browse/FLINK-5991 Project: Flink

[jira] [Created] (FLINK-5949) Flink on YARN checks for Kerberos credentials for non-Kerberos authentication methods

2017-03-01 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-5949: -- Summary: Flink on YARN checks for Kerberos credentials for non-Kerberos authentication methods Key: FLINK-5949 URL: https://issues.apache.org/jira/browse/FLINK-5949

[jira] [Created] (FLINK-5939) Wrong version in README.md for several Apache Flink extensions

2017-02-28 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-5939: -- Summary: Wrong version in README.md for several Apache Flink extensions Key: FLINK-5939 URL: https://issues.apache.org/jira/browse/FLINK-5939 Project

Re: [DISCUSS] Per-key event time

2017-02-28 Thread Tzu-Li (Gordon) Tai
Throwing in some thoughts: When a source determines that no more data will come for a key (which  in itself is a bit of a tricky problem) then it should signal to downstream  operations to take the key out of watermark calculations, that is that we  can release some space.  I don’t think this is p

[jira] [Created] (FLINK-5849) Kafka Consumer checkpointed state may contain undefined offsets

2017-02-20 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-5849: -- Summary: Kafka Consumer checkpointed state may contain undefined offsets Key: FLINK-5849 URL: https://issues.apache.org/jira/browse/FLINK-5849 Project

Re: Wish to Contribute - Andrea Spina

2017-02-16 Thread Tzu-Li (Gordon) Tai
Hi Andrea! Welcome to the Flink community! Very happy to see that you’re interested in contributing. Just let us know when you’ve came across an issue on JIRA you’d like to start working on, and also your JIRA id. We can assign you contributor permissions on JIRA so that you can assign issues

Re: Running multiple streaming jobs in same cluster

2017-02-12 Thread Tzu-Li (Gordon) Tai
Hi Ozan, From your description, it seems like your original huge job can be broken down into smaller disconnected graphs, with only some of the graphs requiring checkpointing / snapshots. In general, it would be a good practice to split disconnected graphs of the execution graph into multiple j

Re: [ANNOUNCE] Welcome Stefan Richter as a new committer

2017-02-10 Thread Tzu-Li (Gordon) Tai
Great news! Welcome Stefan :-D On February 10, 2017 at 11:36:14 PM, Aljoscha Krettek (aljos...@apache.org) wrote: Welcome! :-) On Fri, 10 Feb 2017 at 16:10 Till Rohrmann wrote: > Great to have you on board as a committer Stefan :-) > > On Fri, Feb 10, 2017 at 3:32 PM, Greg Hogan wro

[jira] [Created] (FLINK-5772) Instability with embedded Elasticsearch node in ElasticsearchSink test

2017-02-10 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-5772: -- Summary: Instability with embedded Elasticsearch node in ElasticsearchSink test Key: FLINK-5772 URL: https://issues.apache.org/jira/browse/FLINK-5772

Re: Fwd: Setup Flink on windows link not accessible

2017-02-09 Thread Tzu-Li (Gordon) Tai
Hi Kush! The link is broken, thank you for reporting this. This is the correct link:  https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/flink_on_windows.html Cheers, Gordon On February 10, 2017 at 12:54:01 PM, kush batra (kush.ba...@gmail.com) wrote: Hi, In Quickstart section

Re: [ANNOUNCE] Welcome Jark Wu and Kostas Kloudas as committers

2017-02-07 Thread Tzu-Li (Gordon) Tai
Great news, congratulations! Thank you Jark and Kostas for all your contribution to the Flink community so far, and really looking forward seeing it grow even more in the future with you being aboard :-D Cheers, Gordon On February 8, 2017 at 11:35:08 AM, Jark Wu (wuchong...@alibaba-inc.com) wr

[jira] [Created] (FLINK-5728) FlinkKafkaProducer should flush on checkpoint by default

2017-02-06 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-5728: -- Summary: FlinkKafkaProducer should flush on checkpoint by default Key: FLINK-5728 URL: https://issues.apache.org/jira/browse/FLINK-5728 Project: Flink

[jira] [Created] (FLINK-5719) Let LatencyMarkers completely bypass operators / chains

2017-02-06 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-5719: -- Summary: Let LatencyMarkers completely bypass operators / chains Key: FLINK-5719 URL: https://issues.apache.org/jira/browse/FLINK-5719 Project: Flink

[jira] [Created] (FLINK-5716) Make streaming SourceContexts aware of source idleness

2017-02-06 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-5716: -- Summary: Make streaming SourceContexts aware of source idleness Key: FLINK-5716 URL: https://issues.apache.org/jira/browse/FLINK-5716 Project: Flink

[jira] [Created] (FLINK-5704) Deprecate FlinkKafkaConsumer constructors in favor of improvements to decoupling from Kafka offset committing

2017-02-03 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-5704: -- Summary: Deprecate FlinkKafkaConsumer constructors in favor of improvements to decoupling from Kafka offset committing Key: FLINK-5704 URL: https://issues.apache.org

[jira] [Created] (FLINK-5702) Kafka Producer docs should warn if using setLogFailuresOnly, at-least-once is compromised

2017-02-02 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-5702: -- Summary: Kafka Producer docs should warn if using setLogFailuresOnly, at-least-once is compromised Key: FLINK-5702 URL: https://issues.apache.org/jira/browse/FLINK

[jira] [Created] (FLINK-5701) FlinkKafkaPrdocuer violates at-least-once by not handling failed records

2017-02-02 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-5701: -- Summary: FlinkKafkaPrdocuer violates at-least-once by not handling failed records Key: FLINK-5701 URL: https://issues.apache.org/jira/browse/FLINK-5701

[jira] [Created] (FLINK-5697) Add per-shard watermarks for FlinkKinesisConsumer

2017-02-02 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-5697: -- Summary: Add per-shard watermarks for FlinkKinesisConsumer Key: FLINK-5697 URL: https://issues.apache.org/jira/browse/FLINK-5697 Project: Flink

Re: [VOTE] Release Apache Flink 1.2.0 (RC3)

2017-01-30 Thread Tzu-Li (Gordon) Tai
+1 (non-binding) - Tested TaskManager failures on Mesos / Standalone with exactly-once guarantees - Above tests also done against Kafka 0.8 / 0.9 / 0.10, offsets committed correctly back to ZK (manual check for 0.8 due to FLINK-4822) - Tested Kafka 0.10 server-side timestamps - Verified Async I/O

Re: es v5 support

2017-01-30 Thread Tzu-Li (Gordon) Tai
Hi Sree, The expected Flink version for support for Elasticsearch 5 is Flink 1.3.0. The release ETA for 1.3.0 is near the end of May 2017, based on the recently announced timely-based release schedules. The pull requests for Elasticsearch 5 support just needs to be reviewed before they are merge

[jira] [Created] (FLINK-5581) Improve Kerberos security related documentation

2017-01-19 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-5581: -- Summary: Improve Kerberos security related documentation Key: FLINK-5581 URL: https://issues.apache.org/jira/browse/FLINK-5581 Project: Flink

[jira] [Created] (FLINK-5580) Kerberos keytabs not working for YARN deployment mode

2017-01-19 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-5580: -- Summary: Kerberos keytabs not working for YARN deployment mode Key: FLINK-5580 URL: https://issues.apache.org/jira/browse/FLINK-5580 Project: Flink

[jira] [Created] (FLINK-5579) Kerberos not working for Kafka connector using ticket cache

2017-01-19 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-5579: -- Summary: Kerberos not working for Kafka connector using ticket cache Key: FLINK-5579 URL: https://issues.apache.org/jira/browse/FLINK-5579 Project: Flink

Re: [DISCUSS] Time-based releases in Flink

2017-01-18 Thread Tzu-Li (Gordon) Tai
Hi Robert, Thanks for bringing up the discussion. I like the proposal. Regarding some of the downsides mentioned in the wiki: 1. Features that don’t make it in time with the feature freeze: I think that’s ok, as long as we’re consistent with the schedules for the next release. This way users wa

[jira] [Created] (FLINK-5512) RabbitMQ documentation should inform that exactly-once holds for RMQSource only when parallelism is 1

2017-01-16 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-5512: -- Summary: RabbitMQ documentation should inform that exactly-once holds for RMQSource only when parallelism is 1 Key: FLINK-5512 URL: https://issues.apache.org/jira

[jira] [Created] (FLINK-5487) Proper at-least-once support for ElasticsearchSink

2017-01-13 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-5487: -- Summary: Proper at-least-once support for ElasticsearchSink Key: FLINK-5487 URL: https://issues.apache.org/jira/browse/FLINK-5487 Project: Flink

[jira] [Created] (FLINK-5483) Link to "linking modules not contained in binary distribution" broken in all connector documentations

2017-01-13 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-5483: -- Summary: Link to "linking modules not contained in binary distribution" broken in all connector documentations Key: FLINK-5483 URL: https://issues.apach

[jira] [Created] (FLINK-5479) Per-partition watermarks in FlinkKafkaConsumer should consider idle partitions

2017-01-13 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-5479: -- Summary: Per-partition watermarks in FlinkKafkaConsumer should consider idle partitions Key: FLINK-5479 URL: https://issues.apache.org/jira/browse/FLINK-5479

Re: [DISCUSS] ElasticSearch in Flink Strategy

2017-01-09 Thread Tzu-Li (Gordon) Tai
K-5353 should be > > solved sooner or later... > > > > Best, > > Flavio > > > > On 9 Jan 2017 12:24, "Tzu-Li (Gordon) Tai" wrote: > > > > > Thanks for starting the discussion Robert. > > > > > > I agree

Re: [DISCUSS] ElasticSearch in Flink Strategy

2017-01-09 Thread Tzu-Li (Gordon) Tai
Thanks for starting the discussion Robert. I agree that we should try to treat the Elasticsearch connectors in a similar fashion to the multiple Kafka connector versions. +1 for flink-connector-elasticsearch-base +1 to consolidating the documents +1 for keeping the ES connector in Flink I’d als

Re: [VOTE] Release Apache Flink 1.1.4 (RC4)

2016-12-21 Thread Tzu-Li (Gordon) Tai
+1 (unbinding) - Assured Kinesis connector works properly across different / older AWS EMR versions with simple job (randomly tested in version 5.0.3, 4.8.2, 4.7.2, 4.5) - Ran Java & Scala quickstart examples (in local IDE mode, cluster mode and multi-node cluster) On December 21, 2016 at 9:38

[jira] [Created] (FLINK-5128) Get Kafka partitions in FlinkKafkaProducer only if a partitioner is set

2016-11-22 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-5128: -- Summary: Get Kafka partitions in FlinkKafkaProducer only if a partitioner is set Key: FLINK-5128 URL: https://issues.apache.org/jira/browse/FLINK-5128

[jira] [Created] (FLINK-5075) Kinesis consumer incorrectly determines shards as newly discovered when tested against Kinesalite

2016-11-15 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-5075: -- Summary: Kinesis consumer incorrectly determines shards as newly discovered when tested against Kinesalite Key: FLINK-5075 URL: https://issues.apache.org/jira/browse

Re: [VOTE] Release Apache Flink 1.1.4 (RC1)

2016-11-15 Thread Tzu-Li (Gordon) Tai
+1 for cancelling RC1. I would also like to merge FLINK-5013 (Kinesis connector not working on old EMR versions) to be included in 1.1.4. The PR (https://github.com/apache/flink/pull/2787) has been reviewed and can be merged soon. Best, Gordon On November 15, 2016 at 6:56:42 PM, Stefan Richte

Re: [DISCUSS] Proposed updates to Flink project site

2016-11-08 Thread Tzu-Li (Gordon) Tai
Hi Mike, I really like the idea! Especially that you pointed out we need a substantial rework on the introduction page to let users that are new to modern streaming frameworks understand the space first. With this content as a prelude, once the detailed Flink contents kick in, it will definitel

[jira] [Created] (FLINK-5018) User configurable source idle timeout to work with WatermarkStatus emitting

2016-11-04 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-5018: -- Summary: User configurable source idle timeout to work with WatermarkStatus emitting Key: FLINK-5018 URL: https://issues.apache.org/jira/browse/FLINK-5018

[jira] [Created] (FLINK-5017) Introduce WatermarkStatus stream element to allow for temporarily idle streaming sources

2016-11-04 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-5017: -- Summary: Introduce WatermarkStatus stream element to allow for temporarily idle streaming sources Key: FLINK-5017 URL: https://issues.apache.org/jira/browse/FLINK

[jira] [Created] (FLINK-4821) Implement rescalable non-partitioned state for Kinesis Connector

2016-10-13 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-4821: -- Summary: Implement rescalable non-partitioned state for Kinesis Connector Key: FLINK-4821 URL: https://issues.apache.org/jira/browse/FLINK-4821 Project

[jira] [Created] (FLINK-4753) Kafka 0.8 connector's Periodic Offset Committer should synchronize on checkpoint lock

2016-10-06 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-4753: -- Summary: Kafka 0.8 connector's Periodic Offset Committer should synchronize on checkpoint lock Key: FLINK-4753 URL: https://issues.apache.org/jira/browse/FLINK

Re: KafkaProducer can not be instantiated

2016-10-05 Thread Tzu-Li (Gordon) Tai
Sorry, correction to my last statements: On the consumer side I think the instantiation was already removed from the constructor in a recent commit. On October 5, 2016 at 11:37:41 PM, Tzu-Li (Gordon) Tai (tzuli...@apache.org) wrote: This matters on the consumer side, yes. Moving the

Re: KafkaProducer can not be instantiated

2016-10-05 Thread Tzu-Li (Gordon) Tai
fetching subtask the partition list is identical? On 05.10.2016 17:17, Tzu-Li (Gordon) Tai wrote: > Hi Timo, > > I haven’t had the chance to look at the producer side too much yet, but after > a look in the code, > I think it’s reasonable to remove the instantiation fr

<    3   4   5   6   7   8   9   >