Re: [DISCUSS] FLIP-14: Loops API and Termination

2016-11-11 Thread Paris Carbone
Hi Shi, Naiad/Timely Dataflow and other projects use global coordination which is very convenient for asynchronous progress tracking in general but it has some downsides in a production systems that count on in-flight transactional control mechanisms and rollback recovery guarantees. This is

Re: [DISCUSS] FLIP-14: Loops API and Termination

2016-11-11 Thread SHI Xiaogang
Hi, Fouad Thank you for the explanation. Now the centralized method seems correct to me. The passing of StatusUpdate events will lead to synchronous iterations and we are using the information in each iterations to terminate the computation. Actually, i prefer the centralized method because in

Re: RocksDB IO error

2016-11-11 Thread Aljoscha Krettek
Hi Zhenhao, does this happen reproducibly? What happens after the failure? Will it retry restoring and then succeed? I have a suspicion that Yarn could be cleaning up some files that RocksDB expects to be there while restoring. Cheers, Aljoscha On Thu, 10 Nov 2016 at 14:24

RE: [FLINK-3848] Add ProjectableTableSource

2016-11-11 Thread Anton Solovev
I have a question about type info, it looks like one in calcite mail you showed in jira, but I catch CodeGenException when checking predefined table field types with output types from source Flink cannot generate conversion because of row arity. We cannot reduce number of table fields, but we

[jira] [Created] (FLINK-5056) BucketingSink deletes valid data when checkpoint notification is slow.

2016-11-11 Thread Kostas Kloudas (JIRA)
Kostas Kloudas created FLINK-5056: - Summary: BucketingSink deletes valid data when checkpoint notification is slow. Key: FLINK-5056 URL: https://issues.apache.org/jira/browse/FLINK-5056 Project:

Re: Flink using Yarn on MapR

2016-11-11 Thread vijikarthi
Flink security context gets initialized during the application start phase. As part of the initialization, the UserGroupInformation (UGI) instance is bootstrapped using the Hadoop configuration files (read: HADOOP_CONF_DIR or YARN_CONF_DIR environment variable is set). If the hadoop configuration

[jira] [Created] (FLINK-5055) Security feature crashes JM for certain Hadoop versions even though using no Kerberos

2016-11-11 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-5055: Summary: Security feature crashes JM for certain Hadoop versions even though using no Kerberos Key: FLINK-5055 URL: https://issues.apache.org/jira/browse/FLINK-5055

Re: Flink using Yarn on MapR

2016-11-11 Thread Till Rohrmann
Hi Naveen, I could reproduce your problem with the given Hadoop version (2.7.0-mapr-1607). It seems to me as if this version always tries to use Kerberos even though I selected the AuthenticationMethod.SIMPLE (no Kerberos activated). I've also tested it with vanilla Hadoop 2.7.3 and there it

[jira] [Created] (FLINK-5054) Make the BucketingSink rescalable.

2016-11-11 Thread Kostas Kloudas (JIRA)
Kostas Kloudas created FLINK-5054: - Summary: Make the BucketingSink rescalable. Key: FLINK-5054 URL: https://issues.apache.org/jira/browse/FLINK-5054 Project: Flink Issue Type: Improvement

Re: [DISCUSS] FLIP-14: Loops API and Termination

2016-11-11 Thread Fouad ALi
Hi Shi, It seems that you are referring to the centralized algorithm which is no longer the proposed version. In the decentralized version (check last doc) there is no master node or global coordination involved. Let us keep this discussion to the decentralized one if possible. To answer your

[jira] [Created] (FLINK-5053) Incremental / lightweight snapshots for checkpoints

2016-11-11 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-5053: - Summary: Incremental / lightweight snapshots for checkpoints Key: FLINK-5053 URL: https://issues.apache.org/jira/browse/FLINK-5053 Project: Flink Issue

[jira] [Created] (FLINK-5052) Changing the maximum parallelism (number of key groups) of a job

2016-11-11 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-5052: - Summary: Changing the maximum parallelism (number of key groups) of a job Key: FLINK-5052 URL: https://issues.apache.org/jira/browse/FLINK-5052 Project: Flink

[jira] [Created] (FLINK-5051) Backwards compatibility for serializers in backend state

2016-11-11 Thread Stefan Richter (JIRA)
Stefan Richter created FLINK-5051: - Summary: Backwards compatibility for serializers in backend state Key: FLINK-5051 URL: https://issues.apache.org/jira/browse/FLINK-5051 Project: Flink

RE: [FLINK-4541] Support for SQL NOT IN operator

2016-11-11 Thread Alexander Shoshin
Yes, I would like to continue my work with this issue. It is already assigned to me. I will follow the approach that you suggested. Thanks, Alexander > -Original Message- > From: Fabian Hueske [mailto:fhue...@gmail.com] > Sent: Friday, November 11, 2016 11:49 AM > To:

[VOTE] Release Apache Flink 1.1.4 (RC1)

2016-11-11 Thread Ufuk Celebi
Dear Flink community, Please vote on releasing the following candidate as Apache Flink version 1.1.4. The commit to be voted on: 3c1024a (http://git-wip-us.apache.org/repos/asf/flink/commit/3c1024a) Branch: release-1.1.4-rc1

Re: [FLINK-4541] Support for SQL NOT IN operator

2016-11-11 Thread Fabian Hueske
Hi Alexander, I had a look at the logical plan generated by Calcite for NOT IN again and noticed that the CROSS JOIN is only used to join a single record (the result of a global aggregation) to all records of the other table. IMO, this is a special case, that we can easily support and efficiently

RE: [FLINK-4541] Support for SQL NOT IN operator

2016-11-11 Thread Alexander Shoshin
Hi Fabian, Should we close this issue then? Or I could just leave the comment why we can't repair NOT IN at the moment. So no one else will do the same research again. Perhaps the Calcite team will change a logical plan for NOT IN and we will be back to this issue. Regards, Alexander