[jira] [Commented] (FLINK-5167) StreamExecutionEnvironment's set function return `this` instead of void

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15697465#comment-15697465
 ] 

ASF GitHub Bot commented on FLINK-5167:
---

GitHub user shijinkui opened a pull request:

https://github.com/apache/flink/pull/2874

[FLINK-5167] StreamExecutionEnvironment set function return `this` in…

StreamExecutionEnvironment's set function return `this` instead of void

for example :

public void setNumberOfExecutionRetries(int numberOfExecutionRetries) { 
  config.setNumberOfExecutionRetries(numberOfExecutionRetries); 
}

change to:

public StreamExecutionEnvironment setNumberOfExecutionRetries(int 
numberOfExecutionRetries) { 
  config.setNumberOfExecutionRetries(numberOfExecutionRetries); 
  return this; 
}

- [X] General
  - The pull request references the related JIRA issue ("[FLINK-5167] 
StreamExecutionEnvironment's set function return `this` instead of void")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [X] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [X] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shijinkui/flink FLINK-5166

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2874.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2874


commit b28992af876c33e00f86d27e1dd6ef783fefa3a0
Author: shijinkui 
Date:   2016-11-26T06:47:07Z

[FLINK-5167] StreamExecutionEnvironment set function return `this` instead 
of void




> StreamExecutionEnvironment's set function return `this` instead of void
> ---
>
> Key: FLINK-5167
> URL: https://issues.apache.org/jira/browse/FLINK-5167
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Reporter: shijinkui
>
> for example :
> public void setNumberOfExecutionRetries(int numberOfExecutionRetries) {
>   config.setNumberOfExecutionRetries(numberOfExecutionRetries);
> }
> change to:
> public StreamExecutionEnvironment setNumberOfExecutionRetries(int 
> numberOfExecutionRetries) {
>   config.setNumberOfExecutionRetries(numberOfExecutionRetries);
>   return this;
> }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2874: [FLINK-5167] StreamExecutionEnvironment set functi...

[shijinkui]

GitHub user shijinkui opened a pull request:

https://github.com/apache/flink/pull/2874

[FLINK-5167] StreamExecutionEnvironment set function return `this` in…

StreamExecutionEnvironment's set function return `this` instead of void

for example :

public void setNumberOfExecutionRetries(int numberOfExecutionRetries) { 
  config.setNumberOfExecutionRetries(numberOfExecutionRetries); 
}

change to:

public StreamExecutionEnvironment setNumberOfExecutionRetries(int 
numberOfExecutionRetries) { 
  config.setNumberOfExecutionRetries(numberOfExecutionRetries); 
  return this; 
}

- [X] General
  - The pull request references the related JIRA issue ("[FLINK-5167] 
StreamExecutionEnvironment's set function return `this` instead of void")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [X] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [X] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shijinkui/flink FLINK-5166

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2874.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2874


commit b28992af876c33e00f86d27e1dd6ef783fefa3a0
Author: shijinkui 
Date:   2016-11-26T06:47:07Z

[FLINK-5167] StreamExecutionEnvironment set function return `this` instead 
of void




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2509: [FLINK-4280][kafka-connector] Explicit start posit...

[tzulitai]

Github user tzulitai commented on a diff in the pull request:

https://github.com/apache/flink/pull/2509#discussion_r89664876
  
--- Diff: 
flink-streaming-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/Kafka09Fetcher.java
 ---
@@ -218,26 +221,57 @@ public void run() {
}
}
 
-   // seek the consumer to the initial offsets
+   List partitionsWithNoOffset = new 
ArrayList<>();
for (KafkaTopicPartitionState partition 
: subscribedPartitions()) {
if (partition.isOffsetDefined()) {
LOG.info("Partition {} has restored 
initial offsets {} from checkpoint / savepoint; seeking the consumer " +
"to position {}", 
partition.getKafkaPartitionHandle(), partition.getOffset(), 
partition.getOffset() + 1);
 

consumer.seek(partition.getKafkaPartitionHandle(), partition.getOffset() + 1);
} else {
-   // for partitions that do not have 
offsets restored from a checkpoint/savepoint,
-   // we need to define our internal 
offset state for them using the initial offsets retrieved from Kafka
-   // by the KafkaConsumer, so that they 
are correctly checkpointed and committed on the next checkpoint
+   
partitionsWithNoOffset.add(partition.getKafkaTopicPartition());
+   }
+   }
 
-   long fetchedOffset = 
consumer.position(partition.getKafkaPartitionHandle());
+   if (partitionsWithNoOffset.size() == 
subscribedPartitions().length) {
+   // if all partitions have no initial offsets, 
that means we're starting fresh
+   switch (startupMode) {
+   case EARLIEST:
+   LOG.info("Setting starting 
point as earliest offset for partitions {}", partitionsWithNoOffset);
+
+   
seekPartitionsToBeginning(consumer, 
convertKafkaPartitions(subscribedPartitions()));
--- End diff --

The problem with this one is that the `seekToBeginning` method broke 
compatibility from 0.8 to 0.9+.
In 0.8, it's `seekToBeginning(TopicPartition...)` while in 0.9+ it's 
`seekToBeginning(Collection)`.

I'll integrate these seek methods into the `KafkaConsumerCallBridge` 
introduced in a recent PR. I'll be inevitable that we must redundantly do the 
Array -> List conversion because our `subscribedPartitions` is an Array, while 
0.9+ methods take an API. For the 0.8 methods, instead of converting the list 
back to an array, I'll just iterate over the list and call 
`seekPartitionsToBeginning` for each one.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-5167) StreamExecutionEnvironment's set function return `this` instead of void

[shijinkui (JIRA)]

shijinkui created FLINK-5167:


 Summary: StreamExecutionEnvironment's set function return `this` 
instead of void
 Key: FLINK-5167
 URL: https://issues.apache.org/jira/browse/FLINK-5167
 Project: Flink
  Issue Type: Bug
  Components: DataStream API
Reporter: shijinkui


for example :

public void setNumberOfExecutionRetries(int numberOfExecutionRetries) {
config.setNumberOfExecutionRetries(numberOfExecutionRetries);
}

change to:

public StreamExecutionEnvironment setNumberOfExecutionRetries(int 
numberOfExecutionRetries) {
config.setNumberOfExecutionRetries(numberOfExecutionRetries);
return this;
}





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5166) TextInputFormatTest.testNestedFileRead

[shijinkui (JIRA)]

shijinkui created FLINK-5166:


 Summary: TextInputFormatTest.testNestedFileRead
 Key: FLINK-5166
 URL: https://issues.apache.org/jira/browse/FLINK-5166
 Project: Flink
  Issue Type: Bug
  Components: Build System
Reporter: shijinkui


mvn clean package -P \!scala-2.11,scala-2.11  -U

Failed tests:
  TextInputFormatTest.testNestedFileRead:140 Test erroneous

Tests run: 846, Failures: 1, Errors: 0, Skipped: 0





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3551) Sync Scala and Java Streaming Examples

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696363#comment-15696363
 ] 

ASF GitHub Bot commented on FLINK-3551:
---

Github user thvasilo commented on a diff in the pull request:

https://github.com/apache/flink/pull/2761#discussion_r89645498
  
--- Diff: 
flink-examples/flink-examples-streaming/src/main/scala/org/apache/flink/streaming/scala/examples/kafka/ReadFromKafka.scala
 ---
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.scala.examples.kafka
+
+import org.apache.flink.api.common.restartstrategy.RestartStrategies
+import org.apache.flink.api.java.utils.ParameterTool
+import org.apache.flink.streaming.api.scala._
+import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer08
+import org.apache.flink.streaming.util.serialization.SimpleStringSchema
+
+/**
+  * Read Strings from Kafka and print them to standard out.
+  * Note: On a cluster, DataStream.print() will print to the TaskManager's 
.out file!
+  *
+  * Please pass the following arguments to run the example:
+  * --topic test
+  * --bootstrap.servers localhost:9092
+  * --zookeeper.connect localhost:2181
+  * --group.id myconsumer
+  *
+  */
+object ReadFromKafka {
+
+  def main(args: Array[String]): Unit = {
+
+// parse input arguments
+val params = ParameterTool.fromArgs(args)
+
+if (params.getNumberOfParameters < 4) {
+  println("Missing parameters!\nUsage: Kafka --topic  " +
+"--bootstrap.servers  --zookeeper.connect  --group.id ")
+  return
+}
+
+val env = StreamExecutionEnvironment.getExecutionEnvironment
+env.getConfig.disableSysoutLogging
+
env.getConfig.setRestartStrategy(RestartStrategies.fixedDelayRestart(4, 1))
+// create a checkpoint every 5 seconds
+env.enableCheckpointing(5000)
+// make parameters available in the web interface
+env.getConfig.setGlobalJobParameters(params)
+
+// create a Kafka streaming source consumer for Kafka 0.8.x
+val kafkaConsumer = new FlinkKafkaConsumer08(
--- End diff --

I would suggest to keep it as is for now unless a project committer 
suggests otherwise. 

The linked issue would indicate that the intention is to have different 
versions, but I guess that would affect the release as well, since different 
jar files would need to be generated.


> Sync Scala and Java Streaming Examples
> --
>
> Key: FLINK-3551
> URL: https://issues.apache.org/jira/browse/FLINK-3551
> Project: Flink
>  Issue Type: Sub-task
>  Components: Examples
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Lim Chee Hau
> Fix For: 1.0.1
>
>
> The Scala Examples lack behind the Java Examples



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2761: [FLINK-3551] [examples] Sync Scala Streaiming Exam...

[thvasilo]

Github user thvasilo commented on a diff in the pull request:

https://github.com/apache/flink/pull/2761#discussion_r89645498
  
--- Diff: 
flink-examples/flink-examples-streaming/src/main/scala/org/apache/flink/streaming/scala/examples/kafka/ReadFromKafka.scala
 ---
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.scala.examples.kafka
+
+import org.apache.flink.api.common.restartstrategy.RestartStrategies
+import org.apache.flink.api.java.utils.ParameterTool
+import org.apache.flink.streaming.api.scala._
+import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer08
+import org.apache.flink.streaming.util.serialization.SimpleStringSchema
+
+/**
+  * Read Strings from Kafka and print them to standard out.
+  * Note: On a cluster, DataStream.print() will print to the TaskManager's 
.out file!
+  *
+  * Please pass the following arguments to run the example:
+  * --topic test
+  * --bootstrap.servers localhost:9092
+  * --zookeeper.connect localhost:2181
+  * --group.id myconsumer
+  *
+  */
+object ReadFromKafka {
+
+  def main(args: Array[String]): Unit = {
+
+// parse input arguments
+val params = ParameterTool.fromArgs(args)
+
+if (params.getNumberOfParameters < 4) {
+  println("Missing parameters!\nUsage: Kafka --topic  " +
+"--bootstrap.servers  --zookeeper.connect  --group.id ")
+  return
+}
+
+val env = StreamExecutionEnvironment.getExecutionEnvironment
+env.getConfig.disableSysoutLogging
+
env.getConfig.setRestartStrategy(RestartStrategies.fixedDelayRestart(4, 1))
+// create a checkpoint every 5 seconds
+env.enableCheckpointing(5000)
+// make parameters available in the web interface
+env.getConfig.setGlobalJobParameters(params)
+
+// create a Kafka streaming source consumer for Kafka 0.8.x
+val kafkaConsumer = new FlinkKafkaConsumer08(
--- End diff --

I would suggest to keep it as is for now unless a project committer 
suggests otherwise. 

The linked issue would indicate that the intention is to have different 
versions, but I guess that would affect the release as well, since different 
jar files would need to be generated.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5146) Improved resource cleanup in RocksDB keyed state backend

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696350#comment-15696350
 ] 

ASF GitHub Bot commented on FLINK-5146:
---

Github user StefanRRichter commented on the issue:

https://github.com/apache/flink/pull/2857
  
I introduced the planned change to the test. You could take another look if 
you want :-)


> Improved resource cleanup in RocksDB keyed state backend
> 
>
> Key: FLINK-5146
> URL: https://issues.apache.org/jira/browse/FLINK-5146
> Project: Flink
>  Issue Type: Improvement
>  Components: State Backends, Checkpointing
>Reporter: Stefan Richter
>Assignee: Stefan Richter
>Priority: Blocker
>
> Currently, the resources such as taken snapshots or iterators are not always 
> cleaned up in the RocksDB state backend. In particular, not starting the 
> runnable future will leave taken snapshots unreleased.
> We should improve the releases of all resources allocated through the RocksDB 
> JNI bridge.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2857: [FLINK-5146] Improved resource cleanup in RocksDB keyed s...

[StefanRRichter]

Github user StefanRRichter commented on the issue:

https://github.com/apache/flink/pull/2857
  
I introduced the planned change to the test. You could take another look if 
you want :-)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3551) Sync Scala and Java Streaming Examples

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696338#comment-15696338
 ] 

ASF GitHub Bot commented on FLINK-3551:
---

Github user ch33hau commented on a diff in the pull request:

https://github.com/apache/flink/pull/2761#discussion_r89644293
  
--- Diff: 
flink-examples/flink-examples-streaming/src/main/scala/org/apache/flink/streaming/scala/examples/kafka/ReadFromKafka.scala
 ---
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.scala.examples.kafka
+
+import org.apache.flink.api.common.restartstrategy.RestartStrategies
+import org.apache.flink.api.java.utils.ParameterTool
+import org.apache.flink.streaming.api.scala._
+import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer08
+import org.apache.flink.streaming.util.serialization.SimpleStringSchema
+
+/**
+  * Read Strings from Kafka and print them to standard out.
+  * Note: On a cluster, DataStream.print() will print to the TaskManager's 
.out file!
+  *
+  * Please pass the following arguments to run the example:
+  * --topic test
+  * --bootstrap.servers localhost:9092
+  * --zookeeper.connect localhost:2181
+  * --group.id myconsumer
+  *
+  */
+object ReadFromKafka {
+
+  def main(args: Array[String]): Unit = {
+
+// parse input arguments
+val params = ParameterTool.fromArgs(args)
+
+if (params.getNumberOfParameters < 4) {
+  println("Missing parameters!\nUsage: Kafka --topic  " +
+"--bootstrap.servers  --zookeeper.connect  --group.id ")
+  return
+}
+
+val env = StreamExecutionEnvironment.getExecutionEnvironment
+env.getConfig.disableSysoutLogging
+
env.getConfig.setRestartStrategy(RestartStrategies.fixedDelayRestart(4, 1))
+// create a checkpoint every 5 seconds
+env.enableCheckpointing(5000)
+// make parameters available in the web interface
+env.getConfig.setGlobalJobParameters(params)
+
+// create a Kafka streaming source consumer for Kafka 0.8.x
+val kafkaConsumer = new FlinkKafkaConsumer08(
--- End diff --

Actually I saw that there is a related issue [FLINK-4286 Have Kafka 
examples that use the Kafka 0.9 
connector](https://issues.apache.org/jira/browse/FLINK-4286) which basically 
requests to have examples for different Kafka versions (eg, Kafka08.jar, 
Kafka09.jar). 
So I just synced exactly same as Java's examples and thinking of let 
FLINK-4286 to take care about the different versions of Kafka.

However, if use the more recent one make more sense to you, I am happy to 
update it


> Sync Scala and Java Streaming Examples
> --
>
> Key: FLINK-3551
> URL: https://issues.apache.org/jira/browse/FLINK-3551
> Project: Flink
>  Issue Type: Sub-task
>  Components: Examples
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Lim Chee Hau
> Fix For: 1.0.1
>
>
> The Scala Examples lack behind the Java Examples



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2761: [FLINK-3551] [examples] Sync Scala Streaiming Exam...

[ch33hau]

Github user ch33hau commented on a diff in the pull request:

https://github.com/apache/flink/pull/2761#discussion_r89644293
  
--- Diff: 
flink-examples/flink-examples-streaming/src/main/scala/org/apache/flink/streaming/scala/examples/kafka/ReadFromKafka.scala
 ---
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.scala.examples.kafka
+
+import org.apache.flink.api.common.restartstrategy.RestartStrategies
+import org.apache.flink.api.java.utils.ParameterTool
+import org.apache.flink.streaming.api.scala._
+import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer08
+import org.apache.flink.streaming.util.serialization.SimpleStringSchema
+
+/**
+  * Read Strings from Kafka and print them to standard out.
+  * Note: On a cluster, DataStream.print() will print to the TaskManager's 
.out file!
+  *
+  * Please pass the following arguments to run the example:
+  * --topic test
+  * --bootstrap.servers localhost:9092
+  * --zookeeper.connect localhost:2181
+  * --group.id myconsumer
+  *
+  */
+object ReadFromKafka {
+
+  def main(args: Array[String]): Unit = {
+
+// parse input arguments
+val params = ParameterTool.fromArgs(args)
+
+if (params.getNumberOfParameters < 4) {
+  println("Missing parameters!\nUsage: Kafka --topic  " +
+"--bootstrap.servers  --zookeeper.connect  --group.id ")
+  return
+}
+
+val env = StreamExecutionEnvironment.getExecutionEnvironment
+env.getConfig.disableSysoutLogging
+
env.getConfig.setRestartStrategy(RestartStrategies.fixedDelayRestart(4, 1))
+// create a checkpoint every 5 seconds
+env.enableCheckpointing(5000)
+// make parameters available in the web interface
+env.getConfig.setGlobalJobParameters(params)
+
+// create a Kafka streaming source consumer for Kafka 0.8.x
+val kafkaConsumer = new FlinkKafkaConsumer08(
--- End diff --

Actually I saw that there is a related issue [FLINK-4286 Have Kafka 
examples that use the Kafka 0.9 
connector](https://issues.apache.org/jira/browse/FLINK-4286) which basically 
requests to have examples for different Kafka versions (eg, Kafka08.jar, 
Kafka09.jar). 
So I just synced exactly same as Java's examples and thinking of let 
FLINK-4286 to take care about the different versions of Kafka.

However, if use the more recent one make more sense to you, I am happy to 
update it


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4913) Per-job Yarn clusters: include user jar in system class loader

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696330#comment-15696330
 ] 

ASF GitHub Bot commented on FLINK-4913:
---

Github user uce closed the pull request at:

https://github.com/apache/flink/pull/2795


> Per-job Yarn clusters: include user jar in system class loader 
> ---
>
> Key: FLINK-4913
> URL: https://issues.apache.org/jira/browse/FLINK-4913
> Project: Flink
>  Issue Type: Improvement
>  Components: Client, YARN Client
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
> Fix For: 1.2.0
>
>
> Including the jar directly in the system classloader avoids loading it for 
> every instantiation of the ExecutionGraph and every Task execution. Note, 
> this is only possible for per-job clusters (i.e. Yarn/Mesos).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2795: Revert "[FLINK-4913][yarn] include user jars in sy...

[uce]

Github user uce closed the pull request at:

https://github.com/apache/flink/pull/2795


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2754: [FLINK-4975] [checkpointing] Add a limit for how m...

[StephanEwen]

Github user StephanEwen closed the pull request at:

https://github.com/apache/flink/pull/2754


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4975) Add a limit for how much data may be buffered during checkpoint alignment

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696288#comment-15696288
 ] 

ASF GitHub Bot commented on FLINK-4975:
---

Github user StephanEwen closed the pull request at:

https://github.com/apache/flink/pull/2754


> Add a limit for how much data may be buffered during checkpoint alignment
> -
>
> Key: FLINK-4975
> URL: https://issues.apache.org/jira/browse/FLINK-4975
> Project: Flink
>  Issue Type: Improvement
>  Components: State Backends, Checkpointing
>Affects Versions: 1.1.3
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 1.2.0, 1.1.4
>
>
> During checkpoint alignment, data may be buffered/spilled.
> We should introduce an upper limit for the spilled data volume. After 
> exceeding that limit, the checkpoint alignment should abort and the 
> checkpoint be canceled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (FLINK-4565) Support for SQL IN operator

[Nikolay Vasilishin (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696107#comment-15696107
 ] 

Nikolay Vasilishin edited comment on FLINK-4565 at 11/25/16 4:38 PM:
-

[~fhueske], I've finally opened the PR, 
[https://github.com/apache/flink/pull/2870].

And what's about subqueries.
As I've seen above, there are couple of ways to implement it: 
 - create table with inner join somewhere at the beginning
 - try to use calcite's IN operator, when constructing RexNode.

I tried to implement the second approach. It looks as it has to be 
([example|https://github.com/NickolayVasilishin/flink/commit/792a10440eede233260d3b2157fbd93af9e3572f#diff-7c7c5dd5a5723b84b8d45424f04b2be5R68]).
In that case we need to [overload in 
method|https://github.com/NickolayVasilishin/flink/commit/792a10440eede233260d3b2157fbd93af9e3572f#diff-04d1bca648d7ee47ab9ce787c8d944a6R108]
 with Table argument, then somehow construct proper rex node in [toRexNode 
method|https://github.com/NickolayVasilishin/flink/commit/792a10440eede233260d3b2157fbd93af9e3572f#diff-1b0f642e7f9b75bde5062b89b0b873e8R28]
 [1] (now it constructs not-working plan, I'll show it below). Then it will be 
passed to 
[CodeGenerator|https://github.com/NickolayVasilishin/flink/commit/792a10440eede233260d3b2157fbd93af9e3572f#diff-c0ee691580cf6752e6cb186ca4f0260dR980],
 where I have to implement visitSubquery method. Actually, it's not hard to 
generate code for all subquery's nodes, but I don't know, what code should be 
ganarated to link subquery with the rest query.

RexNode constructed whis way has this logical plan:
{noformat}
LogicalFilter(condition=[IN(IN($2, {
LogicalProject(c=[$2])
  LogicalFilter(condition=[=($1, 6)])
LogicalTableScan(table=[[_DataSetTable_0]])
}))])
  LogicalTableScan(table=[[_DataSetTable_0]])
{noformat}
There is duplicating IN call.

Logical plan generating for similar query in SQL API looks like this:
{noformat}
LogicalProject(a=[$0], c=[$2])
  LogicalJoin(condition=[=($1, $3)], joinType=[inner])
LogicalTableScan(table=[[_DataSetTable_0]])
LogicalAggregate(group=[{0}])
  LogicalProject(b=[$1])
LogicalFilter(condition=[AND(>=($1, 6), <=($1, 9))])
  LogicalTableScan(table=[[_DataSetTable_0]])
{noformat}
So, it's implemented via the first approach.

In the first approach it's not clear for me where we will get reference on 
first (left) table, as we invoke IN method on expressions like 'column. But I 
didn't thought about it well yet.

[1] I'm sorry, I forgot to change code to this:
{noformat}
val in: RexSubQuery = RexSubQuery.in(table.getRelNode, new 
ImmutableList.Builder[RexNode]().add(children.map(_.toRexNode): _*).build())
relBuilder.call(SqlStdOperatorTable.IN, in)
{noformat}
In this case there will be generated plan shown above


was (Author: nvasilishin):
[~fhueske], I've finally opened the PR, 
[https://github.com/apache/flink/pull/2870].

And what's about subqueries.
As I've seen above, there are couple of ways to implement it: 
 - create table with inner join somewhere at the beginning
 - try to use calcite's IN operator, when constructing RexNode.

I tried to implement the second approach. It looks as it has to be 
([example|https://github.com/NickolayVasilishin/flink/commit/792a10440eede233260d3b2157fbd93af9e3572f#diff-7c7c5dd5a5723b84b8d45424f04b2be5R68]).
In that case we need to [overload in 
method|https://github.com/NickolayVasilishin/flink/commit/792a10440eede233260d3b2157fbd93af9e3572f#diff-04d1bca648d7ee47ab9ce787c8d944a6R108]
 with Table argument, then somehow construct proper rex node in [toRexNode 
method|https://github.com/NickolayVasilishin/flink/commit/792a10440eede233260d3b2157fbd93af9e3572f#diff-1b0f642e7f9b75bde5062b89b0b873e8R28]
 [1] (now it constructs not-working plan, I'll show it below). Then it will be 
passed to 
[CodeGenerator|https://github.com/NickolayVasilishin/flink/commit/792a10440eede233260d3b2157fbd93af9e3572f#diff-c0ee691580cf6752e6cb186ca4f0260dR980],
 where I have to implement visitSubquery method. Actually, it's not hard to 
generate code for all subquery's nodes, but I don't know, what code should be 
ganarated to link subquery with the rest query.

RexNode constructed whis way has this logical plan:
{noformat}
LogicalFilter(condition=[IN(IN($2, {
LogicalProject(c=[$2])
  LogicalFilter(condition=[=($1, 6)])
LogicalTableScan(table=[[_DataSetTable_0]])
}))])
  LogicalTableScan(table=[[_DataSetTable_0]])
{noformat}
There is duplicating IN call.

Logical plan generating for similar query in SQL API looks like this:
{noformat}
LogicalProject(a=[$0], c=[$2])
  LogicalJoin(condition=[=($1, $3)], joinType=[inner])
LogicalTableScan(table=[[_DataSetTable_0]])
LogicalAggregate(group=[{0}])
  LogicalProject(b=[$1])
LogicalFilter(condition=[AND(>=($1, 6), <=($1, 9))])
  LogicalTableScan(table=[[_DataSetTable_0]])

[jira] [Commented] (FLINK-4712) Implementing ranking predictions for ALS

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696250#comment-15696250
 ] 

ASF GitHub Bot commented on FLINK-4712:
---

Github user thvasilo commented on the issue:

https://github.com/apache/flink/pull/2838
  
> The problem is not with the evaluate(test: TestType): DataSet[Double] but 
rather with evaluate(test: TestType): DataSet[(Prediction,Prediction)].

Completely agree there, I advocated for removing/renaming the evaluate 
function, we considered using a `score` function for a more sklearn-like 
approach before, see e.g. #902. Having _some_ function that returns a 
`DataSet[(truth: Prediction,pred: Prediction)]` is useful and probably 
necessary, but we should look at alternatives as the current state is confusing.
I think I like the approach you are suggesting, so feel free to come up 
with an alternative in the WIP PRs.

Getting rid of the Pipeline requirements for recommendation algorithms 
would simplify some things. In that case we'll have to re-evaluate if it makes 
sense for them to implement the `Predictor` interface at all, or maybe we have 
`ChainablePredictors` but I think our hierarchy is deep enough already.


> Implementing ranking predictions for ALS
> 
>
> Key: FLINK-4712
> URL: https://issues.apache.org/jira/browse/FLINK-4712
> Project: Flink
>  Issue Type: New Feature
>  Components: Machine Learning Library
>Reporter: Domokos Miklós Kelen
>Assignee: Gábor Hermann
>
> We started working on implementing ranking predictions for recommender 
> systems. Ranking prediction means that beside predicting scores for user-item 
> pairs, the recommender system is able to recommend a top K list for the users.
> Details:
> In practice, this would mean finding the K items for a particular user with 
> the highest predicted rating. It should be possible also to specify whether 
> to exclude the already seen items from a particular user's toplist. (See for 
> example the 'exclude_known' setting of [Graphlab Create's ranking 
> factorization 
> recommender|https://turi.com/products/create/docs/generated/graphlab.recommender.ranking_factorization_recommender.RankingFactorizationRecommender.recommend.html#graphlab.recommender.ranking_factorization_recommender.RankingFactorizationRecommender.recommend]
>  ).
> The output of the topK recommendation function could be in the form of 
> {{DataSet[(Int,Int,Int)]}}, meaning (user, item, rank), similar to Graphlab 
> Create's output. However, this is arguable: follow up work includes 
> implementing ranking recommendation evaluation metrics (such as precision@k, 
> recall@k, ndcg@k), similar to [Spark's 
> implementations|https://spark.apache.org/docs/1.5.0/mllib-evaluation-metrics.html#ranking-systems].
>  It would be beneficial if we were able to design the API such that it could 
> be included in the proposed evaluation framework (see 
> [5157|https://issues.apache.org/jira/browse/FLINK-2157]), which makes it 
> neccessary to consider the possible output type {{DataSet[(Int, 
> Array[Int])]}} or {{DataSet[(Int, Array[(Int,Double)])]}} meaning (user, 
> array of items), possibly including the predicted scores as well. See 
> [4713|https://issues.apache.org/jira/browse/FLINK-4713] for details.
> Another question arising is whether to provide this function as a member of 
> the ALS class, as a switch-kind of parameter to the ALS implementation 
> (meaning the model is either a rating or a ranking recommender model) or in 
> some other way.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2838: [FLINK-4712] [FLINK-4713] [ml] Ranking recommendation & e...

[thvasilo]

Github user thvasilo commented on the issue:

https://github.com/apache/flink/pull/2838
  
> The problem is not with the evaluate(test: TestType): DataSet[Double] but 
rather with evaluate(test: TestType): DataSet[(Prediction,Prediction)].

Completely agree there, I advocated for removing/renaming the evaluate 
function, we considered using a `score` function for a more sklearn-like 
approach before, see e.g. #902. Having _some_ function that returns a 
`DataSet[(truth: Prediction,pred: Prediction)]` is useful and probably 
necessary, but we should look at alternatives as the current state is confusing.
I think I like the approach you are suggesting, so feel free to come up 
with an alternative in the WIP PRs.

Getting rid of the Pipeline requirements for recommendation algorithms 
would simplify some things. In that case we'll have to re-evaluate if it makes 
sense for them to implement the `Predictor` interface at all, or maybe we have 
`ChainablePredictors` but I think our hierarchy is deep enough already.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2873: [backport] [FLINK-5158] [ckPtCoord] Handle excepti...

[tillrohrmann]

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/2873

[backport] [FLINK-5158] [ckPtCoord] Handle exceptions from 
CompletedCheckpointStore in CheckpointCoordinator

Backport of the #2872 for the release 1.1 branch.

Handle exceptions from the CompletedCheckpointStore properly in the 
CheckpointCoordinator. This
means that in case of an exception, the completed checkpoint will be 
properly cleaned up and also
the triggering of subsequent checkpoints will be started.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink 
backportFixCheckpointCoordinatorExceptionHandling

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2873.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2873


commit c68c08f7b478f354a5c432f8640a344dcf553190
Author: Till Rohrmann 
Date:   2016-11-24T17:16:28Z

[FLINK-5158] [ckPtCoord] Handle exceptions from CompletedCheckpointStore in 
CheckpointCoordinator

Handle exceptions from the CompletedCheckpointStore properly in the 
CheckpointCoordinator. This
means that in case of an exception, the completed checkpoint will be 
properly cleaned up and also
the triggering of subsequent checkpoints will be started.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (FLINK-4565) Support for SQL IN operator

[Nikolay Vasilishin (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696107#comment-15696107
 ] 

Nikolay Vasilishin edited comment on FLINK-4565 at 11/25/16 4:10 PM:
-

[~fhueske], I've finally opened the PR, 
[https://github.com/apache/flink/pull/2870].

And what's about subqueries.
As I've seen above, there are couple of ways to implement it: 
 - create table with inner join somewhere at the beginning
 - try to use calcite's IN operator, when constructing RexNode.

I tried to implement the second approach. It looks as it has to be 
([example|https://github.com/NickolayVasilishin/flink/commit/792a10440eede233260d3b2157fbd93af9e3572f#diff-7c7c5dd5a5723b84b8d45424f04b2be5R68]).
In that case we need to [overload in 
method|https://github.com/NickolayVasilishin/flink/commit/792a10440eede233260d3b2157fbd93af9e3572f#diff-04d1bca648d7ee47ab9ce787c8d944a6R108]
 with Table argument, then somehow construct proper rex node in [toRexNode 
method|https://github.com/NickolayVasilishin/flink/commit/792a10440eede233260d3b2157fbd93af9e3572f#diff-1b0f642e7f9b75bde5062b89b0b873e8R28]
 [1] (now it constructs not-working plan, I'll show it below). Then it will be 
passed to 
[CodeGenerator|https://github.com/NickolayVasilishin/flink/commit/792a10440eede233260d3b2157fbd93af9e3572f#diff-c0ee691580cf6752e6cb186ca4f0260dR980],
 where I have to implement visitSubquery method. Actually, it's not hard to 
generate code for all subquery's nodes, but I don't know, what code should be 
ganarated to link subquery with the rest query.

RexNode constructed whis way has this logical plan:
{noformat}
LogicalFilter(condition=[IN(IN($2, {
LogicalProject(c=[$2])
  LogicalFilter(condition=[=($1, 6)])
LogicalTableScan(table=[[_DataSetTable_0]])
}))])
  LogicalTableScan(table=[[_DataSetTable_0]])
{noformat}
There is duplicating IN call.

Logical plan generating for similar query in SQL API looks like this:
{noformat}
LogicalProject(a=[$0], c=[$2])
  LogicalJoin(condition=[=($1, $3)], joinType=[inner])
LogicalTableScan(table=[[_DataSetTable_0]])
LogicalAggregate(group=[{0}])
  LogicalProject(b=[$1])
LogicalFilter(condition=[AND(>=($1, 6), <=($1, 9))])
  LogicalTableScan(table=[[_DataSetTable_0]])
{noformat}
So, it's implemented via the first approach.

In the first approach it's not clear for me where we will get reference on 
first (left) table, as we invoke IN method on expressions like 'column. But I 
didn't thought about it well yet.

[1] I'm sorry, I forgot to change code to this:
{noformat}
val in: RexSubQuery = RexSubQuery.in(table.getRelNode, new 
ImmutableList.Builder[RexNode]().add(children.map(_.toRexNode): _*).build())
relBuilder.call(SqlStdOperatorTable.IN, in)
{noformat}
In this case there will be generated plan shown above


was (Author: nvasilishin):
[~fhueske], I've finally opened the PR, 
[https://github.com/apache/flink/pull/2870].

And what's about subqueries.
As I've seen above, there are couple of ways to implement it: 
 - create table with inner join somewhere at the beginning
 - try to use calcite's IN operator, when constructing RexNode.

I tried to implement the second approach. It looks as it has to be 
([example|https://github.com/NickolayVasilishin/flink/commit/792a10440eede233260d3b2157fbd93af9e3572f#diff-7c7c5dd5a5723b84b8d45424f04b2be5R68]).
In that case we need to [overload in 
method|https://github.com/NickolayVasilishin/flink/commit/792a10440eede233260d3b2157fbd93af9e3572f#diff-04d1bca648d7ee47ab9ce787c8d944a6R108]
 with Table argument, then somehow construct proper rex node in [toRexNode 
method|https://github.com/NickolayVasilishin/flink/commit/792a10440eede233260d3b2157fbd93af9e3572f#diff-04d1bca648d7ee47ab9ce787c8d944a6R108]
 (now it constructs not-working plan, I'll show it below). Then it will be 
passed to 
[CodeGenerator|https://github.com/NickolayVasilishin/flink/commit/792a10440eede233260d3b2157fbd93af9e3572f#diff-c0ee691580cf6752e6cb186ca4f0260dR980],
 where I have to implement visitSubquery method. Actually, it's not hard to 
generate code for all subquery's nodes, but I don't know, what code should be 
ganarated to link subquery with the rest query.

RexNode constructed whis way has this logical plan:
{noformat}
LogicalFilter(condition=[IN(IN($2, {
LogicalProject(c=[$2])
  LogicalFilter(condition=[=($1, 6)])
LogicalTableScan(table=[[_DataSetTable_0]])
}))])
  LogicalTableScan(table=[[_DataSetTable_0]])
{noformat}
There is duplicating IN call.

Logical plan generating for similar query in SQL API looks like this:
{noformat}
LogicalProject(a=[$0], c=[$2])
  LogicalJoin(condition=[=($1, $3)], joinType=[inner])
LogicalTableScan(table=[[_DataSetTable_0]])
LogicalAggregate(group=[{0}])
  LogicalProject(b=[$1])
LogicalFilter(condition=[AND(>=($1, 6), <=($1, 9))])
  LogicalTableScan(table=[[_DataSetTable_0]])

[GitHub] flink issue #2843: FLINK-4798] CEPITCase.testSimpleKeyedPatternCEP test fail...

[BorisOsipov]

Github user BorisOsipov commented on the issue:

https://github.com/apache/flink/pull/2843
  
@zentol this one: https://issues.apache.org/jira/browse/FLINK-2020


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4798) CEPITCase.testSimpleKeyedPatternCEP test failure

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696191#comment-15696191
 ] 

ASF GitHub Bot commented on FLINK-4798:
---

Github user BorisOsipov commented on the issue:

https://github.com/apache/flink/pull/2843
  
@zentol this one: https://issues.apache.org/jira/browse/FLINK-2020


> CEPITCase.testSimpleKeyedPatternCEP test failure
> 
>
> Key: FLINK-4798
> URL: https://issues.apache.org/jira/browse/FLINK-4798
> Project: Flink
>  Issue Type: Bug
>  Components: CEP
>Affects Versions: 1.2.0
>Reporter: Robert Metzger
>Assignee: Boris Osipov
>  Labels: test-stability
>
> {code}
> ---
>  T E S T S
> ---
> Running org.apache.flink.cep.CEPITCase
> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.627 sec <<< 
> FAILURE! - in org.apache.flink.cep.CEPITCase
> testSimpleKeyedPatternCEP(org.apache.flink.cep.CEPITCase)  Time elapsed: 
> 0.312 sec  <<< FAILURE!
> java.lang.AssertionError: Different number of lines in expected and obtained 
> result. expected:<3> but was:<1>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:645)
>   at 
> org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:316)
>   at 
> org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:302)
>   at org.apache.flink.cep.CEPITCase.after(CEPITCase.java:61)
> {code}
> in https://api.travis-ci.org/jobs/166676733/log.txt?deansi=true



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLINK-4680) Add SlidingRow row-windows for streaming tables

[sunjincheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sunjincheng reassigned FLINK-4680:
--

Assignee: sunjincheng

> Add SlidingRow row-windows for streaming tables
> ---
>
> Key: FLINK-4680
> URL: https://issues.apache.org/jira/browse/FLINK-4680
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API & SQL
>Affects Versions: 1.2.0
>Reporter: Fabian Hueske
>Assignee: sunjincheng
>
> Add SlideRow row-windows for streaming tables as described in 
> [FLIP-11|https://cwiki.apache.org/confluence/display/FLINK/FLIP-11%3A+Table+API+Stream+Aggregations].
>  
> This task requires to implement a custom stream operator and integrate it 
> with checkpointing and timestamp / watermark logic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5158) Handle ZooKeeperCompletedCheckpointStore exceptions in CheckpointCoordinator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696175#comment-15696175
 ] 

ASF GitHub Bot commented on FLINK-5158:
---

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/2872

[FLINK-5158] [ckPtCoord] Handle exceptions from CompletedCheckpointStore in 
CheckpointCoordinator

Handle exceptions from the CompletedCheckpointStore properly in the 
CheckpointCoordinator. This
means that in case of an exception, the completed checkpoint will be 
properly cleaned up and also
the triggering of subsequent checkpoints will be started.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink 
fixCheckpointCoordinatorExceptionHandling

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2872.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2872


commit 063a696b4eb5a259c714818c0b0ba5cc883a596d
Author: Till Rohrmann 
Date:   2016-11-24T17:16:28Z

[FLINK-5158] [ckPtCoord] Handle exceptions from CompletedCheckpointStore in 
CheckpointCoordinator

Handle exceptions from the CompletedCheckpointStore properly in the 
CheckpointCoordinator. This
means that in case of an exception, the completed checkpoint will be 
properly cleaned up and also
the triggering of subsequent checkpoints will be started.




> Handle ZooKeeperCompletedCheckpointStore exceptions in CheckpointCoordinator
> 
>
> Key: FLINK-5158
> URL: https://issues.apache.org/jira/browse/FLINK-5158
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
> Fix For: 1.2.0, 1.1.4
>
>
> The checkpoint coordinator does not properly handle exceptions when trying to 
> store completed checkpoints. As a result, completed checkpoints are not 
> properly cleaned up and even worse, the {{CheckpointCoordinator}} might get 
> stuck stopping triggering checkpoints.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5163) Make the production functions rescalable (apart from the Rolling/Bucketing Sinks)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696174#comment-15696174
 ] 

ASF GitHub Bot commented on FLINK-5163:
---

GitHub user kl0u opened a pull request:

https://github.com/apache/flink/pull/2871

[FLINK-5163] Ports the production functions to the new state abstraction.

This includes the following functions:
1) `StatefulSequenceSource`
2) `MessageAcknowledgingSourceBase`
3) `FromElementsFunction`
4) `ContinuousFileMonitoringFunction`

Each of them is a separate commit, for ease of reviewing. 
Most of the functions assume parallelism of 1. The only exception is the 
`StatefulSequenceSource`.

R @aljoscha 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kl0u/flink dop1-source-rescaling

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2871.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2871


commit 3a436c5ef7d0123f0f244700f8a62bb325fac118
Author: kl0u 
Date:   2016-11-17T13:54:08Z

[FLINK-5163] Port the ContinuousFileMonitoringFunction to the new state 
abstractions.

commit a49fce791553c94dc866140c7616f52354d788fc
Author: kl0u 
Date:   2016-11-17T15:52:50Z

[FLINK-5163] Port the FromElementsFunction to the new state abstractions.

commit ecee88819bc7be09681047fc0f8c2e347ddbdd06
Author: kl0u 
Date:   2016-11-18T15:07:45Z

[FLINK-5163] Port the MessageAcknowledgingSourceBase to the new state 
abstractions.

commit 1dde7c7d7b978cfac086ee93eb775069763b7788
Author: kl0u 
Date:   2016-11-21T17:50:30Z

[FLINK-5163] Port the StatefulSequenceSource to the new state abstractions.




> Make the production functions rescalable (apart from the Rolling/Bucketing 
> Sinks)
> -
>
> Key: FLINK-5163
> URL: https://issues.apache.org/jira/browse/FLINK-5163
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.2.0
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
> Fix For: 1.2.0
>
>
> This issue targets porting all the functions in the production code to the 
> new state abstractions. These functions are:
> 1) StatefulSequenceSource
> 2) MessageAcknowledgingSourceBase
> 3) FromElementsFunction
> 4) ContinuousFileMonitoringFunction



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2871: [FLINK-5163] Ports the production functions to the...

[kl0u]

GitHub user kl0u opened a pull request:

https://github.com/apache/flink/pull/2871

[FLINK-5163] Ports the production functions to the new state abstraction.

This includes the following functions:
1) `StatefulSequenceSource`
2) `MessageAcknowledgingSourceBase`
3) `FromElementsFunction`
4) `ContinuousFileMonitoringFunction`

Each of them is a separate commit, for ease of reviewing. 
Most of the functions assume parallelism of 1. The only exception is the 
`StatefulSequenceSource`.

R @aljoscha 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kl0u/flink dop1-source-rescaling

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2871.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2871


commit 3a436c5ef7d0123f0f244700f8a62bb325fac118
Author: kl0u 
Date:   2016-11-17T13:54:08Z

[FLINK-5163] Port the ContinuousFileMonitoringFunction to the new state 
abstractions.

commit a49fce791553c94dc866140c7616f52354d788fc
Author: kl0u 
Date:   2016-11-17T15:52:50Z

[FLINK-5163] Port the FromElementsFunction to the new state abstractions.

commit ecee88819bc7be09681047fc0f8c2e347ddbdd06
Author: kl0u 
Date:   2016-11-18T15:07:45Z

[FLINK-5163] Port the MessageAcknowledgingSourceBase to the new state 
abstractions.

commit 1dde7c7d7b978cfac086ee93eb775069763b7788
Author: kl0u 
Date:   2016-11-21T17:50:30Z

[FLINK-5163] Port the StatefulSequenceSource to the new state abstractions.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2811: [FLINK-5159] Improve perfomance of inner joins wit...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/2811#discussion_r89635696
  
--- Diff: 
flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/runtime/RichMapRunner.scala
 ---
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.table.runtime
+
+import org.apache.flink.api.common.functions.RichMapFunction
+import org.apache.flink.api.common.typeinfo.TypeInformation
+import org.apache.flink.api.java.typeutils.ResultTypeQueryable
+import org.apache.flink.configuration.Configuration
+import org.slf4j.LoggerFactory
+
+class RichMapRunner[IN, OUT](
+name: String,
+code: String,
+@transient returnType: TypeInformation[OUT])
+  extends RichMapFunction[IN, OUT]
+  with ResultTypeQueryable[OUT]
+  with FunctionCompiler[RichMapFunction[IN, OUT]] {
+
+  val LOG = LoggerFactory.getLogger(this.getClass)
+
+  private var function: RichMapFunction[IN, OUT] = null
+
+  override def open(parameters: Configuration): Unit = {
--- End diff --

You are right. When I wrote the comment, I had not thought about wrapping a 
`FlatJoinFunction` yet but was still assuming a wrapped `RichFlatMapFunction` 
which has `open()`.

So we do not need to forward `close()` and `open()` to the 
`FlatJoinFunction`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5159) Improve perfomance of inner joins with a single row input

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696167#comment-15696167
 ] 

ASF GitHub Bot commented on FLINK-5159:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/2811#discussion_r89635696
  
--- Diff: 
flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/runtime/RichMapRunner.scala
 ---
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.table.runtime
+
+import org.apache.flink.api.common.functions.RichMapFunction
+import org.apache.flink.api.common.typeinfo.TypeInformation
+import org.apache.flink.api.java.typeutils.ResultTypeQueryable
+import org.apache.flink.configuration.Configuration
+import org.slf4j.LoggerFactory
+
+class RichMapRunner[IN, OUT](
+name: String,
+code: String,
+@transient returnType: TypeInformation[OUT])
+  extends RichMapFunction[IN, OUT]
+  with ResultTypeQueryable[OUT]
+  with FunctionCompiler[RichMapFunction[IN, OUT]] {
+
+  val LOG = LoggerFactory.getLogger(this.getClass)
+
+  private var function: RichMapFunction[IN, OUT] = null
+
+  override def open(parameters: Configuration): Unit = {
--- End diff --

You are right. When I wrote the comment, I had not thought about wrapping a 
`FlatJoinFunction` yet but was still assuming a wrapped `RichFlatMapFunction` 
which has `open()`.

So we do not need to forward `close()` and `open()` to the 
`FlatJoinFunction`.


> Improve perfomance of inner joins with a single row input
> -
>
> Key: FLINK-5159
> URL: https://issues.apache.org/jira/browse/FLINK-5159
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Alexander Shoshin
>Assignee: Alexander Shoshin
>Priority: Minor
>
> All inner joins (including a cross join) can be implemented as a 
> {{MapFunction}} if one of their inputs is a single row. This row can be 
> passed to a {{MapFunction}} as a {{BroadcastSet}}.
> This approach is going to be more lightweight than the other current 
> strategies.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3848) Add ProjectableTableSource interface and translation rule

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696157#comment-15696157
 ] 

ASF GitHub Bot commented on FLINK-3848:
---

Github user fhueske commented on the issue:

https://github.com/apache/flink/pull/2810
  
Hi @tonycox, I'll have a look over the weekend or early next week as well.
Thanks, Fabian


> Add ProjectableTableSource interface and translation rule
> -
>
> Key: FLINK-3848
> URL: https://issues.apache.org/jira/browse/FLINK-3848
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Reporter: Fabian Hueske
>Assignee: Anton Solovev
>
> Add a {{ProjectableTableSource}} interface for {{TableSource}} implementation 
> that support projection push-down.
> The interface could look as follows
> {code}
> def trait ProjectableTableSource {
>   def setProjection(fields: Array[String]): Unit
> }
> {code}
> In addition we need Calcite rules to push a projection into a TableScan that 
> refers to a {{ProjectableTableSource}}. We might need to tweak the cost model 
> as well to push the optimizer in the right direction.
> Moreover, the {{CsvTableSource}} could be extended to implement 
> {{ProjectableTableSource}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2810: [FLINK-3848] Add ProjectableTableSource interface and tra...

[fhueske]

Github user fhueske commented on the issue:

https://github.com/apache/flink/pull/2810
  
Hi @tonycox, I'll have a look over the weekend or early next week as well.
Thanks, Fabian


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (FLINK-4667) Yarn Session CLI not listening on correct ZK namespace when HA is enabled to use ZooKeeper backend

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels resolved FLINK-4667.
---
   Resolution: Fixed
Fix Version/s: 1.2.0

Fixed via 5bd47012e50516d45a6b50d47f347e6802dfde4c.

> Yarn Session CLI not listening on correct ZK namespace when HA is enabled to 
> use ZooKeeper backend
> --
>
> Key: FLINK-4667
> URL: https://issues.apache.org/jira/browse/FLINK-4667
> Project: Flink
>  Issue Type: Bug
>  Components: Security, YARN Client
>Reporter: Vijay Srinivasaraghavan
>Assignee: Vijay Srinivasaraghavan
>Priority: Minor
> Fix For: 1.2.0
>
>
> In Yarn mode, when Flink is configured for HA using ZooKeeper backend, the 
> leader election listener does not provide correct JM/leader info and will 
> timeout since the listener is waiting on default ZK namespace instead of the 
> application specific (Application ID)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4798) CEPITCase.testSimpleKeyedPatternCEP test failure

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696132#comment-15696132
 ] 

ASF GitHub Bot commented on FLINK-4798:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2843
  
Which "known issue" are you referring to?


> CEPITCase.testSimpleKeyedPatternCEP test failure
> 
>
> Key: FLINK-4798
> URL: https://issues.apache.org/jira/browse/FLINK-4798
> Project: Flink
>  Issue Type: Bug
>  Components: CEP
>Affects Versions: 1.2.0
>Reporter: Robert Metzger
>Assignee: Boris Osipov
>  Labels: test-stability
>
> {code}
> ---
>  T E S T S
> ---
> Running org.apache.flink.cep.CEPITCase
> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.627 sec <<< 
> FAILURE! - in org.apache.flink.cep.CEPITCase
> testSimpleKeyedPatternCEP(org.apache.flink.cep.CEPITCase)  Time elapsed: 
> 0.312 sec  <<< FAILURE!
> java.lang.AssertionError: Different number of lines in expected and obtained 
> result. expected:<3> but was:<1>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:645)
>   at 
> org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:316)
>   at 
> org.apache.flink.test.util.TestBaseUtils.compareResultsByLinesInMemory(TestBaseUtils.java:302)
>   at org.apache.flink.cep.CEPITCase.after(CEPITCase.java:61)
> {code}
> in https://api.travis-ci.org/jobs/166676733/log.txt?deansi=true



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2843: FLINK-4798] CEPITCase.testSimpleKeyedPatternCEP test fail...

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2843
  
Which "known issue" are you referring to?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (FLINK-3903) Homebrew Installation

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-3903:
--
Labels: starter  (was: )

> Homebrew Installation
> -
>
> Key: FLINK-3903
> URL: https://issues.apache.org/jira/browse/FLINK-3903
> Project: Flink
>  Issue Type: Task
>  Components: Documentation, release
>Reporter: Eron Wright 
>Assignee: Ufuk Celebi
>Priority: Minor
>  Labels: starter
>
> Recently I submitted a formula for apache-flink to the 
> [homebrew|http://brew.sh/] project.   Homebrew simplifies installation on Mac:
> {code}
> $ brew install apache-flink
> ...
> $ flink --version
> Version: 1.0.2, Commit ID: d39af15
> {code}
> Updates to the formula are adhoc at the moment.  I opened this issue to 
> formalize updating homebrew into the release process.  I drafted a procedure 
> doc here:
> [https://gist.github.com/EronWright/b62bd3b192a15be4c200a2542f7c9376]
>  
> Please also consider updating the website documentation to suggest homebrew 
> as an alternate installation method for Mac users.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3903) Homebrew Installation

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696129#comment-15696129
 ] 

Maximilian Michels commented on FLINK-3903:
---

Thanks for the formula! Updating the documentation is a good starter issue for 
new contributors.

> Homebrew Installation
> -
>
> Key: FLINK-3903
> URL: https://issues.apache.org/jira/browse/FLINK-3903
> Project: Flink
>  Issue Type: Task
>  Components: Documentation, release
>Reporter: Eron Wright 
>Assignee: Ufuk Celebi
>Priority: Minor
>  Labels: starter
>
> Recently I submitted a formula for apache-flink to the 
> [homebrew|http://brew.sh/] project.   Homebrew simplifies installation on Mac:
> {code}
> $ brew install apache-flink
> ...
> $ flink --version
> Version: 1.0.2, Commit ID: d39af15
> {code}
> Updates to the formula are adhoc at the moment.  I opened this issue to 
> formalize updating homebrew into the release process.  I drafted a procedure 
> doc here:
> [https://gist.github.com/EronWright/b62bd3b192a15be4c200a2542f7c9376]
>  
> Please also consider updating the website documentation to suggest homebrew 
> as an alternate installation method for Mac users.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3883) Rename flink clients for inclusion on system path

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696125#comment-15696125
 ] 

Maximilian Michels commented on FLINK-3883:
---

+1 for the proposed renaming.

All scripts which import {{config.sh}} automatically switch to the symlink 
resolved directory. So you can use {{pwd}} or look it up in the 
{{SYMLINK_RESOLVED_BIN}} variable.

> Rename flink clients for inclusion on system path
> -
>
> Key: FLINK-3883
> URL: https://issues.apache.org/jira/browse/FLINK-3883
> Project: Flink
>  Issue Type: Bug
>  Components: Client
>Reporter: Eron Wright 
>Priority: Minor
>
> I ran into some trouble in preparing a Homebrew formula to install Flink on 
> Mac ([homebrew-core#968|https://github.com/Homebrew/homebrew-core/pull/968]). 
>   Homebrew can install bin scripts into the system path, e.g. 
> `/usr/local/bin/flink`:
> {code}
> $ cat /usr/local/bin/flink
> #!/bin/bash
> exec "/usr/local/Cellar/apache-flink/1.0.2/libexec/bin/flink" "$@"
> {code}
> It would be nice to install the various Flink shells too, but their names 
> don't seem ideal for inclusion on the system path.   I propose that they be 
> renamed or aliased to have the following names (old -> new):
> {code}
> "pyflink2.sh" -> "pyflink2"
> "pyflink3.sh" -> "pyflink3"
> "start-scala-shell.sh" -> "flink-scala-shell"   (note: updated based on 
> feedback)
> {code}
> A related issue is, the shell scripts don't correctly determine their install 
> location when called via a symlink.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-5091) Formalize the AppMaster environment for docker compability

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-5091:
--
Fix Version/s: 1.2.0

> Formalize the AppMaster environment for docker compability
> --
>
> Key: FLINK-5091
> URL: https://issues.apache.org/jira/browse/FLINK-5091
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management, Mesos
>Reporter: Eron Wright 
>Assignee: Eron Wright 
> Fix For: 1.2.0
>
>
> For scenarios where the AppMaster is launched from a docker image, it would 
> be ideal to use the installed Flink rather than rely on a special file layout 
> in the sandbox directory.
> This is related to DCOS integration, which (in 1.2) will launch the AppMaster 
> via Marathon (as a top-level DCOS service).  The existing code assumed that 
> only the dispatcher (coming in 1.3) would launch the AppMaster.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-5091) Formalize the AppMaster environment for docker compability

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-5091:
--
Component/s: Mesos

> Formalize the AppMaster environment for docker compability
> --
>
> Key: FLINK-5091
> URL: https://issues.apache.org/jira/browse/FLINK-5091
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management, Mesos
>Reporter: Eron Wright 
>Assignee: Eron Wright 
> Fix For: 1.2.0
>
>
> For scenarios where the AppMaster is launched from a docker image, it would 
> be ideal to use the installed Flink rather than rely on a special file layout 
> in the sandbox directory.
> This is related to DCOS integration, which (in 1.2) will launch the AppMaster 
> via Marathon (as a top-level DCOS service).  The existing code assumed that 
> only the dispatcher (coming in 1.3) would launch the AppMaster.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4922) Write documentation

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-4922:
--
Fix Version/s: 1.2.0

> Write documentation
> ---
>
> Key: FLINK-4922
> URL: https://issues.apache.org/jira/browse/FLINK-4922
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management, Mesos
>Reporter: Eron Wright 
> Fix For: 1.2.0
>
>
> Document the new config settings and provide a conceptual overview.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-5029) Implement KvState SSL

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-5029:
--
Component/s: Security

> Implement KvState SSL
> -
>
> Key: FLINK-5029
> URL: https://issues.apache.org/jira/browse/FLINK-5029
> Project: Flink
>  Issue Type: Sub-task
>  Components: Security
>Reporter: Eron Wright 
> Fix For: 1.2.0
>
>
> The KVState endpoint is new to 1.2 and should support SSL as the others do.
> Note that, with FLINK-4898, the SSL support code is decoupled from the 
> NettyClient/NettyServer, so can be used by the KvState code by simply 
> installing the `SSLProtocolHandler`.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4922) Write documentation

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-4922:
--
Component/s: Mesos

> Write documentation
> ---
>
> Key: FLINK-4922
> URL: https://issues.apache.org/jira/browse/FLINK-4922
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management, Mesos
>Reporter: Eron Wright 
> Fix For: 1.2.0
>
>
> Document the new config settings and provide a conceptual overview.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4921) Upgrade to Mesos 1.0.1

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-4921:
--
Component/s: Mesos

> Upgrade to Mesos 1.0.1
> --
>
> Key: FLINK-4921
> URL: https://issues.apache.org/jira/browse/FLINK-4921
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management, Mesos
>Reporter: Eron Wright 
>Assignee: Eron Wright 
> Fix For: 1.2.0
>
>
> Upgrade the client library to 1.0.1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-5030) Support hostname verification

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-5030:
--
Component/s: Security

> Support hostname verification
> -
>
> Key: FLINK-5030
> URL: https://issues.apache.org/jira/browse/FLINK-5030
> Project: Flink
>  Issue Type: Sub-task
>  Components: Security
>Reporter: Eron Wright 
> Fix For: 1.2.0
>
>
> _See [Dangerous Code|http://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf] and 
> [further 
> commentary|https://tersesystems.com/2014/03/23/fixing-hostname-verification/] 
> for useful background._
> When hostname verification is performed, it should use the hostname (not IP 
> address) to match the certificate.   The current code is wrongly using the 
> address.
> In technical terms, ensure that calls to `SSLContext::createSSLEngine` supply 
> the expected hostname, not host address.
> Please audit all SSL setup code as to whether hostname verification is 
> enabled, and file follow-ups where necessary.   For example, Akka 2.4 
> supports it but 2.3 doesn't 
> ([ref|http://doc.akka.io/docs/akka/2.4.4/scala/http/client-side/https-support.html#Hostname_verification]).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4921) Upgrade to Mesos 1.0.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696116#comment-15696116
 ] 

ASF GitHub Bot commented on FLINK-4921:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2827
  
Looks good to me. Do we lose backwards-capability for Mesos installations < 
1.0?


> Upgrade to Mesos 1.0.1
> --
>
> Key: FLINK-4921
> URL: https://issues.apache.org/jira/browse/FLINK-4921
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management, Mesos
>Reporter: Eron Wright 
>Assignee: Eron Wright 
> Fix For: 1.2.0
>
>
> Upgrade the client library to 1.0.1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2827: [FLINK-4921] Upgrade to Mesos 1.0.1

[mxm]

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2827
  
Looks good to me. Do we lose backwards-capability for Mesos installations < 
1.0?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (FLINK-4921) Upgrade to Mesos 1.0.1

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-4921:
--
Fix Version/s: 1.2.0

> Upgrade to Mesos 1.0.1
> --
>
> Key: FLINK-4921
> URL: https://issues.apache.org/jira/browse/FLINK-4921
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Eron Wright 
>Assignee: Eron Wright 
> Fix For: 1.2.0
>
>
> Upgrade the client library to 1.0.1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4918) Add SSL support to Mesos artifact server

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-4918:
--
Component/s: Mesos

> Add SSL support to Mesos artifact server
> 
>
> Key: FLINK-4918
> URL: https://issues.apache.org/jira/browse/FLINK-4918
> Project: Flink
>  Issue Type: Task
>  Components: Mesos, Security
>Reporter: Vijay Srinivasaraghavan
>Assignee: Vijay Srinivasaraghavan
> Fix For: 1.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4919) Add secure cookie support for the cluster deployed in Mesos environment

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-4919:
--
Fix Version/s: 1.2.0

> Add secure cookie support for the cluster deployed in Mesos environment
> ---
>
> Key: FLINK-4919
> URL: https://issues.apache.org/jira/browse/FLINK-4919
> Project: Flink
>  Issue Type: Task
>  Components: Mesos, Security
>Reporter: Vijay Srinivasaraghavan
>Assignee: Vijay Srinivasaraghavan
> Fix For: 1.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4918) Add SSL support to Mesos artifact server

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-4918:
--
Fix Version/s: 1.2.0

> Add SSL support to Mesos artifact server
> 
>
> Key: FLINK-4918
> URL: https://issues.apache.org/jira/browse/FLINK-4918
> Project: Flink
>  Issue Type: Task
>  Components: Mesos, Security
>Reporter: Vijay Srinivasaraghavan
>Assignee: Vijay Srinivasaraghavan
> Fix For: 1.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4919) Add secure cookie support for the cluster deployed in Mesos environment

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-4919:
--
Component/s: Mesos

> Add secure cookie support for the cluster deployed in Mesos environment
> ---
>
> Key: FLINK-4919
> URL: https://issues.apache.org/jira/browse/FLINK-4919
> Project: Flink
>  Issue Type: Task
>  Components: Mesos, Security
>Reporter: Vijay Srinivasaraghavan
>Assignee: Vijay Srinivasaraghavan
> Fix For: 1.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4901) Build DCOS Docker image

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-4901:
--
Fix Version/s: 1.2.0

> Build DCOS Docker image
> ---
>
> Key: FLINK-4901
> URL: https://issues.apache.org/jira/browse/FLINK-4901
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management, Mesos
>Reporter: Eron Wright 
> Fix For: 1.2.0
>
>
> For DCOS integration, a Docker image is needed with Flink preinstalled.
> Be aware, the Docker images for many packages of DCOS are built atop a 
> DCOS-specific base image.   Also clarify which repository the image 
> definition should be stored in.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4900) Implement Docker image support

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-4900:
--
Component/s: Mesos

> Implement Docker image support
> --
>
> Key: FLINK-4900
> URL: https://issues.apache.org/jira/browse/FLINK-4900
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management, Mesos
>Reporter: Eron Wright 
>Assignee: Mischa Krüger
>  Labels: release
> Fix For: 1.2.0
>
>
> Support the use of a docker image, with both the unified containerizer and 
> the Docker containerizer.
> Use a configuration setting to explicitly configure which image and 
> containerizer to use.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4901) Build DCOS Docker image

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-4901:
--
Component/s: Mesos

> Build DCOS Docker image
> ---
>
> Key: FLINK-4901
> URL: https://issues.apache.org/jira/browse/FLINK-4901
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management, Mesos
>Reporter: Eron Wright 
> Fix For: 1.2.0
>
>
> For DCOS integration, a Docker image is needed with Flink preinstalled.
> Be aware, the Docker images for many packages of DCOS are built atop a 
> DCOS-specific base image.   Also clarify which repository the image 
> definition should be stored in.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4899) Implement DCOS package

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-4899:
--
Component/s: Mesos

> Implement DCOS package
> --
>
> Key: FLINK-4899
> URL: https://issues.apache.org/jira/browse/FLINK-4899
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management, Mesos
>Reporter: Eron Wright 
>Assignee: Eron Wright 
> Fix For: 1.2.0
>
>
> Implement a DCOS package to install Flink.   Some implementation goals:
> - Install a traditional Flink cluster as a DCOS service (i.e. a single 
> appmaster leveraging the Mesos resource manager).
> - Ensure the service name is customizable, to allow for numerous clusters.  
> Use the service name (default Flink) in the `high-availability.namespace` 
> setting.
> - Provide the web UI via the DCOS admin router.
> - Expose the JM endpoint (and other endpoints) as service ports (e.g. 
> `$PORT0`).
> - Prepare and use a Docker image, i.e `dcos-flink`, as the basis for the JM 
> and the TM.
> - Configure the ZK connection info as normal for a DCOS service.
> No further DCOS CLI integration is expected in the initial release.   The 
> normal `flink run` command (operating in 'standalone' mode) should suffice to 
> use the service.Note that the command will probably work only from within 
> the cluster (since it must connect to ZK and to the JM service port).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4899) Implement DCOS package

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-4899:
--
Fix Version/s: 1.2.0

> Implement DCOS package
> --
>
> Key: FLINK-4899
> URL: https://issues.apache.org/jira/browse/FLINK-4899
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management, Mesos
>Reporter: Eron Wright 
>Assignee: Eron Wright 
> Fix For: 1.2.0
>
>
> Implement a DCOS package to install Flink.   Some implementation goals:
> - Install a traditional Flink cluster as a DCOS service (i.e. a single 
> appmaster leveraging the Mesos resource manager).
> - Ensure the service name is customizable, to allow for numerous clusters.  
> Use the service name (default Flink) in the `high-availability.namespace` 
> setting.
> - Provide the web UI via the DCOS admin router.
> - Expose the JM endpoint (and other endpoints) as service ports (e.g. 
> `$PORT0`).
> - Prepare and use a Docker image, i.e `dcos-flink`, as the basis for the JM 
> and the TM.
> - Configure the ZK connection info as normal for a DCOS service.
> No further DCOS CLI integration is expected in the initial release.   The 
> normal `flink run` command (operating in 'standalone' mode) should suffice to 
> use the service.Note that the command will probably work only from within 
> the cluster (since it must connect to ZK and to the JM service port).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4826) Add keytab based kerberos support for Mesos environment

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-4826:
--
Fix Version/s: 1.2.0

> Add keytab based kerberos support for Mesos environment
> ---
>
> Key: FLINK-4826
> URL: https://issues.apache.org/jira/browse/FLINK-4826
> Project: Flink
>  Issue Type: Task
>  Components: Security
>Reporter: Vijay Srinivasaraghavan
>Assignee: Vijay Srinivasaraghavan
> Fix For: 1.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-3932) Implement State Backend Security

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-3932:
--
Fix Version/s: 1.2.0

> Implement State Backend Security
> 
>
> Key: FLINK-3932
> URL: https://issues.apache.org/jira/browse/FLINK-3932
> Project: Flink
>  Issue Type: New Feature
>  Components: Security
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: security
> Fix For: 1.2.0
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Flink should protect its HA, checkpoint, and savepoint state against 
> unauthorized access.
> As described in the design doc, implement:
> - ZooKeeper authentication w/ Kerberos
> - ZooKeeper authorization (i.e. znode ACLs)
> - Checkpoint/savepoint data protection



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4635) Implement Data Transfer Authentication using shared secret configuration

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-4635:
--
Fix Version/s: 1.2.0

> Implement Data Transfer Authentication using shared secret configuration
> 
>
> Key: FLINK-4635
> URL: https://issues.apache.org/jira/browse/FLINK-4635
> Project: Flink
>  Issue Type: Sub-task
>  Components: Security
>Reporter: Vijay Srinivasaraghavan
>Assignee: Vijay Srinivasaraghavan
> Fix For: 1.2.0
>
>
> The data transfer authentication (TM/Netty) requirement was not addressed as 
> part of FLINK-3930 and this JIRA is created to track the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4637) Address Yarn proxy incompatibility with Flink Web UI when service level authorization is enabled

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-4637:
--
Fix Version/s: 1.2.0

> Address Yarn proxy incompatibility with Flink Web UI when service level 
> authorization is enabled
> 
>
> Key: FLINK-4637
> URL: https://issues.apache.org/jira/browse/FLINK-4637
> Project: Flink
>  Issue Type: Task
>  Components: Security
>Reporter: Vijay Srinivasaraghavan
>Assignee: Vijay Srinivasaraghavan
> Fix For: 1.2.0
>
>
> When service level authorization is enabled (FLINK-3930), the tracking URL 
> (Yarn RM Proxy) is not forwarding the secure cookie and as a result, the 
> Flink Web UI cannot be accessed through the proxy layer. Current workaround 
> is to use the direct Flink Web URL instead of navigating through proxy. This 
> JIRA should address the Yarn proxy/secure cookie navigation issue. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4565) Support for SQL IN operator

[Nikolay Vasilishin (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696107#comment-15696107
 ] 

Nikolay Vasilishin commented on FLINK-4565:
---

[~fhueske], I've finally opened the PR, 
[https://github.com/apache/flink/pull/2870].

And what's about subqueries.
As I've seen above, there are couple of ways to implement it: 
 - create table with inner join somewhere at the beginning
 - try to use calcite's IN operator, when constructing RexNode.

I tried to implement the second approach. It looks as it has to be 
([example|https://github.com/NickolayVasilishin/flink/commit/792a10440eede233260d3b2157fbd93af9e3572f#diff-7c7c5dd5a5723b84b8d45424f04b2be5R68]).
In that case we need to [overload in 
method|https://github.com/NickolayVasilishin/flink/commit/792a10440eede233260d3b2157fbd93af9e3572f#diff-04d1bca648d7ee47ab9ce787c8d944a6R108]
 with Table argument, then somehow construct proper rex node in [toRexNode 
method|https://github.com/NickolayVasilishin/flink/commit/792a10440eede233260d3b2157fbd93af9e3572f#diff-04d1bca648d7ee47ab9ce787c8d944a6R108]
 (now it constructs not-working plan, I'll show it below). Then it will be 
passed to 
[CodeGenerator|https://github.com/NickolayVasilishin/flink/commit/792a10440eede233260d3b2157fbd93af9e3572f#diff-c0ee691580cf6752e6cb186ca4f0260dR980],
 where I have to implement visitSubquery method. Actually, it's not hard to 
generate code for all subquery's nodes, but I don't know, what code should be 
ganarated to link subquery with the rest query.

RexNode constructed whis way has this logical plan:
{noformat}
LogicalFilter(condition=[IN(IN($2, {
LogicalProject(c=[$2])
  LogicalFilter(condition=[=($1, 6)])
LogicalTableScan(table=[[_DataSetTable_0]])
}))])
  LogicalTableScan(table=[[_DataSetTable_0]])
{noformat}
There is duplicating IN call.

Logical plan generating for similar query in SQL API looks like this:
{noformat}
LogicalProject(a=[$0], c=[$2])
  LogicalJoin(condition=[=($1, $3)], joinType=[inner])
LogicalTableScan(table=[[_DataSetTable_0]])
LogicalAggregate(group=[{0}])
  LogicalProject(b=[$1])
LogicalFilter(condition=[AND(>=($1, 6), <=($1, 9))])
  LogicalTableScan(table=[[_DataSetTable_0]])
{noformat}
So, it's implemented via the first approach.

In the first approach it's not clear for me where we will get reference on 
first (left) table, as we invoke IN method on expressions like 'column. But I 
didn't thought about it well yet.

> Support for SQL IN operator
> ---
>
> Key: FLINK-4565
> URL: https://issues.apache.org/jira/browse/FLINK-4565
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Nikolay Vasilishin
>
> It seems that Flink SQL supports the uncorrelated sub-query IN operator. But 
> it should also be available in the Table API and tested.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-3931) Implement Transport Encryption (SSL/TLS)

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-3931:
--
Component/s: Security

> Implement Transport Encryption (SSL/TLS)
> 
>
> Key: FLINK-3931
> URL: https://issues.apache.org/jira/browse/FLINK-3931
> Project: Flink
>  Issue Type: New Feature
>  Components: Security
>Reporter: Eron Wright 
>Assignee: Eron Wright 
>  Labels: security
> Fix For: 1.2.0
>
>   Original Estimate: 1,008h
>  Remaining Estimate: 1,008h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> To assure privacy and data integrity between Flink components, enable TLS for 
> all communication channels.  As described in the design doc:
> - Accept a configured certificate or generate a certificate.
> - Enable Akka SSL
> - Implement Data Transfer SSL
> - Implement Blob Server SSL
> - Implement Web UI HTTPS



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-3932) Implement State Backend Security

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-3932:
--
Component/s: Security

> Implement State Backend Security
> 
>
> Key: FLINK-3932
> URL: https://issues.apache.org/jira/browse/FLINK-3932
> Project: Flink
>  Issue Type: New Feature
>  Components: Security
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: security
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Flink should protect its HA, checkpoint, and savepoint state against 
> unauthorized access.
> As described in the design doc, implement:
> - ZooKeeper authentication w/ Kerberos
> - ZooKeeper authorization (i.e. znode ACLs)
> - Checkpoint/savepoint data protection



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-3929:
--
Component/s: Security

> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>  Components: Security
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
> Fix For: 1.2.0
>
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4950) Add support to include multiple Yarn application entries in Yarn properties file

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696100#comment-15696100
 ] 

Maximilian Michels commented on FLINK-4950:
---

Could you backport your changes part of the pull request 
https://github.com/apache/flink/pull/2425?

> Add support to include multiple Yarn application entries in Yarn properties 
> file
> 
>
> Key: FLINK-4950
> URL: https://issues.apache.org/jira/browse/FLINK-4950
> Project: Flink
>  Issue Type: Task
>  Components: YARN Client
>Reporter: Vijay Srinivasaraghavan
>Assignee: Vijay Srinivasaraghavan
>Priority: Minor
> Fix For: 1.2.0
>
>
> When deploying Flink on Yarn using CLI, Yarn properties file is created in 
> /tmp directory and persisted with the application ID along with few other 
> properties. 
> This JIRA addresses two changes to the current implementation.
> 1) The properties file should be created in the user home directory so that 
> the configurations are not leaked
> 2) Support multiple application entries in the properties file



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4913) Per-job Yarn clusters: include user jar in system class loader

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696102#comment-15696102
 ] 

ASF GitHub Bot commented on FLINK-4913:
---

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/2795
  
@uce can you close the PR? I forgot that pushes to a branch different then 
master don't close a PR.


> Per-job Yarn clusters: include user jar in system class loader 
> ---
>
> Key: FLINK-4913
> URL: https://issues.apache.org/jira/browse/FLINK-4913
> Project: Flink
>  Issue Type: Improvement
>  Components: Client, YARN Client
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
> Fix For: 1.2.0
>
>
> Including the jar directly in the system classloader avoids loading it for 
> every instantiation of the ExecutionGraph and every Task execution. Note, 
> this is only possible for per-job clusters (i.e. Yarn/Mesos).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4950) Add support to include multiple Yarn application entries in Yarn properties file

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-4950:
--
Fix Version/s: 1.2.0

> Add support to include multiple Yarn application entries in Yarn properties 
> file
> 
>
> Key: FLINK-4950
> URL: https://issues.apache.org/jira/browse/FLINK-4950
> Project: Flink
>  Issue Type: Task
>  Components: YARN Client
>Reporter: Vijay Srinivasaraghavan
>Assignee: Vijay Srinivasaraghavan
>Priority: Minor
> Fix For: 1.2.0
>
>
> When deploying Flink on Yarn using CLI, Yarn properties file is created in 
> /tmp directory and persisted with the application ID along with few other 
> properties. 
> This JIRA addresses two changes to the current implementation.
> 1) The properties file should be created in the user home directory so that 
> the configurations are not leaked
> 2) Support multiple application entries in the properties file



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4635) Implement Data Transfer Authentication using shared secret configuration

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-4635:
--
Component/s: Security

> Implement Data Transfer Authentication using shared secret configuration
> 
>
> Key: FLINK-4635
> URL: https://issues.apache.org/jira/browse/FLINK-4635
> Project: Flink
>  Issue Type: Sub-task
>  Components: Security
>Reporter: Vijay Srinivasaraghavan
>Assignee: Vijay Srinivasaraghavan
>
> The data transfer authentication (TM/Netty) requirement was not addressed as 
> part of FLINK-3930 and this JIRA is created to track the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4637) Address Yarn proxy incompatibility with Flink Web UI when service level authorization is enabled

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-4637:
--
Component/s: Security

> Address Yarn proxy incompatibility with Flink Web UI when service level 
> authorization is enabled
> 
>
> Key: FLINK-4637
> URL: https://issues.apache.org/jira/browse/FLINK-4637
> Project: Flink
>  Issue Type: Task
>  Components: Security
>Reporter: Vijay Srinivasaraghavan
>Assignee: Vijay Srinivasaraghavan
>
> When service level authorization is enabled (FLINK-3930), the tracking URL 
> (Yarn RM Proxy) is not forwarding the secure cookie and as a result, the 
> Flink Web UI cannot be accessed through the proxy layer. Current workaround 
> is to use the direct Flink Web URL instead of navigating through proxy. This 
> JIRA should address the Yarn proxy/secure cookie navigation issue. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-4913) Per-job Yarn clusters: include user jar in system class loader

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger resolved FLINK-4913.
---
Resolution: Fixed

I removed fix for version 1.1.4 again.

> Per-job Yarn clusters: include user jar in system class loader 
> ---
>
> Key: FLINK-4913
> URL: https://issues.apache.org/jira/browse/FLINK-4913
> Project: Flink
>  Issue Type: Improvement
>  Components: Client, YARN Client
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
> Fix For: 1.2.0
>
>
> Including the jar directly in the system classloader avoids loading it for 
> every instantiation of the ExecutionGraph and every Task execution. Note, 
> this is only possible for per-job clusters (i.e. Yarn/Mesos).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4913) Per-job Yarn clusters: include user jar in system class loader

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger updated FLINK-4913:
--
Fix Version/s: (was: 1.1.4)

> Per-job Yarn clusters: include user jar in system class loader 
> ---
>
> Key: FLINK-4913
> URL: https://issues.apache.org/jira/browse/FLINK-4913
> Project: Flink
>  Issue Type: Improvement
>  Components: Client, YARN Client
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
> Fix For: 1.2.0
>
>
> Including the jar directly in the system classloader avoids loading it for 
> every instantiation of the ExecutionGraph and every Task execution. Note, 
> this is only possible for per-job clusters (i.e. Yarn/Mesos).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4667) Yarn Session CLI not listening on correct ZK namespace when HA is enabled to use ZooKeeper backend

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated FLINK-4667:
--
Component/s: Security

> Yarn Session CLI not listening on correct ZK namespace when HA is enabled to 
> use ZooKeeper backend
> --
>
> Key: FLINK-4667
> URL: https://issues.apache.org/jira/browse/FLINK-4667
> Project: Flink
>  Issue Type: Bug
>  Components: Security, YARN Client
>Reporter: Vijay Srinivasaraghavan
>Assignee: Vijay Srinivasaraghavan
>Priority: Minor
>
> In Yarn mode, when Flink is configured for HA using ZooKeeper backend, the 
> leader election listener does not provide correct JM/leader info and will 
> timeout since the listener is waiting on default ZK namespace instead of the 
> application specific (Application ID)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (FLINK-4913) Per-job Yarn clusters: include user jar in system class loader

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger reopened FLINK-4913:
---

Reverted in 1.1.4: http://git-wip-us.apache.org/repos/asf/flink/commit/3b5d3c6f

> Per-job Yarn clusters: include user jar in system class loader 
> ---
>
> Key: FLINK-4913
> URL: https://issues.apache.org/jira/browse/FLINK-4913
> Project: Flink
>  Issue Type: Improvement
>  Components: Client, YARN Client
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
> Fix For: 1.2.0
>
>
> Including the jar directly in the system classloader avoids loading it for 
> every instantiation of the ExecutionGraph and every Task execution. Note, 
> this is only possible for per-job clusters (i.e. Yarn/Mesos).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2795: Revert "[FLINK-4913][yarn] include user jars in system cl...

[rmetzger]

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/2795
  
Merging ...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4913) Per-job Yarn clusters: include user jar in system class loader

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696084#comment-15696084
 ] 

ASF GitHub Bot commented on FLINK-4913:
---

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/2795
  
Merging ...


> Per-job Yarn clusters: include user jar in system class loader 
> ---
>
> Key: FLINK-4913
> URL: https://issues.apache.org/jira/browse/FLINK-4913
> Project: Flink
>  Issue Type: Improvement
>  Components: Client, YARN Client
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
> Fix For: 1.2.0, 1.1.4
>
>
> Including the jar directly in the system classloader avoids loading it for 
> every instantiation of the ExecutionGraph and every Task execution. Note, 
> this is only possible for per-job clusters (i.e. Yarn/Mesos).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4565) Support for SQL IN operator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696057#comment-15696057
 ] 

ASF GitHub Bot commented on FLINK-4565:
---

GitHub user NickolayVasilishin opened a pull request:

https://github.com/apache/flink/pull/2870

[FLINK-4565] Support for SQL IN operator

[FLINK-4565] Support for SQL IN operator

This PR is a part of work on SQL IN operator in Table API, which 
implements IN for literals.
Two cases are covered: less and great then 20 literals.

Also I have some questions:
- converting all numeric types to BigDecimal isn't ok? I decided to 
make so to simplify use of hashset.
- validation isn't really good. It forces to use operator with same 
type literals. Should I rework it or maybe just add more cases?

expressionDsl.scala:
entry point for IN operator in scala API
ScalarOperators.scala:
1) All numeric types are upcasting to BigDecimal for using in hashset, 
other types are unchanged in castNumeric
2) valuesInitialization used for 2 cases: when we have more then 20 
operands (then we use hashset, initialized in constructor, descibed below) and 
less then 20 operands (then we initialize operands in method's body and use 
them in conjunction)
3) comparison also covers described above cases. In first case we use 
callback to declare and initialize hashset with all operands. Otherwise we just 
put all operands in conjunction.
4) Final code is built up with these code snippets.
CodeGenerator.scala:
passes arguments and callback to declare and init hashet
FunctionCatalog.scala:
registers "in" as method
InITCase:
some use cases


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/NickolayVasilishin/flink FLINK-4565

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2870.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2870


commit e495f96c0016f53195398d78f757857eb8546917
Author: nikolay_vasilishin 
Date:   2016-11-25T14:18:20Z

[FLINK-4565] Support for SQL IN operator

expressionDsl.scala:
entry point for IN operator in scala API
ScalarOperators.scala:
1) All numeric types are upcasting to BigDecimal for using in hashset, 
other types are unchanged in castNumeric
2) valuesInitialization used for 2 cases: when we have more then 20 
operands (then we use hashset, initialized in constructor, descibed below) and 
less then 20 operands (then we initialize operands in method's body and use 
them in conjunction)
3) comparison also covers described above cases. In first case we use 
callback to declare and initialize hashset with all operands. Otherwise we just 
put all operands in conjunction.
4) Final code is built up with these code snippets.
CodeGenerator.scala:
passes arguments and callback to declare and init hashet
FunctionCatalog.scala:
registers "in" as method
InITCase:
some use cases




> Support for SQL IN operator
> ---
>
> Key: FLINK-4565
> URL: https://issues.apache.org/jira/browse/FLINK-4565
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Nikolay Vasilishin
>
> It seems that Flink SQL supports the uncorrelated sub-query IN operator. But 
> it should also be available in the Table API and tested.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2870: [FLINK-4565] Support for SQL IN operator

[NickolayVasilishin]

GitHub user NickolayVasilishin opened a pull request:

https://github.com/apache/flink/pull/2870

[FLINK-4565] Support for SQL IN operator

[FLINK-4565] Support for SQL IN operator

This PR is a part of work on SQL IN operator in Table API, which 
implements IN for literals.
Two cases are covered: less and great then 20 literals.

Also I have some questions:
- converting all numeric types to BigDecimal isn't ok? I decided to 
make so to simplify use of hashset.
- validation isn't really good. It forces to use operator with same 
type literals. Should I rework it or maybe just add more cases?

expressionDsl.scala:
entry point for IN operator in scala API
ScalarOperators.scala:
1) All numeric types are upcasting to BigDecimal for using in hashset, 
other types are unchanged in castNumeric
2) valuesInitialization used for 2 cases: when we have more then 20 
operands (then we use hashset, initialized in constructor, descibed below) and 
less then 20 operands (then we initialize operands in method's body and use 
them in conjunction)
3) comparison also covers described above cases. In first case we use 
callback to declare and initialize hashset with all operands. Otherwise we just 
put all operands in conjunction.
4) Final code is built up with these code snippets.
CodeGenerator.scala:
passes arguments and callback to declare and init hashet
FunctionCatalog.scala:
registers "in" as method
InITCase:
some use cases


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/NickolayVasilishin/flink FLINK-4565

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2870.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2870


commit e495f96c0016f53195398d78f757857eb8546917
Author: nikolay_vasilishin 
Date:   2016-11-25T14:18:20Z

[FLINK-4565] Support for SQL IN operator

expressionDsl.scala:
entry point for IN operator in scala API
ScalarOperators.scala:
1) All numeric types are upcasting to BigDecimal for using in hashset, 
other types are unchanged in castNumeric
2) valuesInitialization used for 2 cases: when we have more then 20 
operands (then we use hashset, initialized in constructor, descibed below) and 
less then 20 operands (then we initialize operands in method's body and use 
them in conjunction)
3) comparison also covers described above cases. In first case we use 
callback to declare and initialize hashset with all operands. Otherwise we just 
put all operands in conjunction.
4) Final code is built up with these code snippets.
CodeGenerator.scala:
passes arguments and callback to declare and init hashet
FunctionCatalog.scala:
registers "in" as method
InITCase:
some use cases




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5159) Improve perfomance of inner joins with a single row input

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696048#comment-15696048
 ] 

ASF GitHub Bot commented on FLINK-5159:
---

Github user AlexanderShoshin commented on a diff in the pull request:

https://github.com/apache/flink/pull/2811#discussion_r89628629
  
--- Diff: 
flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/runtime/RichMapRunner.scala
 ---
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.table.runtime
+
+import org.apache.flink.api.common.functions.RichMapFunction
+import org.apache.flink.api.common.typeinfo.TypeInformation
+import org.apache.flink.api.java.typeutils.ResultTypeQueryable
+import org.apache.flink.configuration.Configuration
+import org.slf4j.LoggerFactory
+
+class RichMapRunner[IN, OUT](
+name: String,
+code: String,
+@transient returnType: TypeInformation[OUT])
+  extends RichMapFunction[IN, OUT]
+  with ResultTypeQueryable[OUT]
+  with FunctionCompiler[RichMapFunction[IN, OUT]] {
+
+  val LOG = LoggerFactory.getLogger(this.getClass)
+
+  private var function: RichMapFunction[IN, OUT] = null
+
+  override def open(parameters: Configuration): Unit = {
--- End diff --

Do we really need it?
FlatJoinFunction that we are going to use doesn't have a close() method. So 
if we create a close() method here it will look like this:
{{override def close(): Unit = super.close()}}
And {{super.close()}} is also empty.


> Improve perfomance of inner joins with a single row input
> -
>
> Key: FLINK-5159
> URL: https://issues.apache.org/jira/browse/FLINK-5159
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Alexander Shoshin
>Assignee: Alexander Shoshin
>Priority: Minor
>
> All inner joins (including a cross join) can be implemented as a 
> {{MapFunction}} if one of their inputs is a single row. This row can be 
> passed to a {{MapFunction}} as a {{BroadcastSet}}.
> This approach is going to be more lightweight than the other current 
> strategies.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2811: [FLINK-5159] Improve perfomance of inner joins wit...

[AlexanderShoshin]

Github user AlexanderShoshin commented on a diff in the pull request:

https://github.com/apache/flink/pull/2811#discussion_r89628629
  
--- Diff: 
flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/runtime/RichMapRunner.scala
 ---
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.table.runtime
+
+import org.apache.flink.api.common.functions.RichMapFunction
+import org.apache.flink.api.common.typeinfo.TypeInformation
+import org.apache.flink.api.java.typeutils.ResultTypeQueryable
+import org.apache.flink.configuration.Configuration
+import org.slf4j.LoggerFactory
+
+class RichMapRunner[IN, OUT](
+name: String,
+code: String,
+@transient returnType: TypeInformation[OUT])
+  extends RichMapFunction[IN, OUT]
+  with ResultTypeQueryable[OUT]
+  with FunctionCompiler[RichMapFunction[IN, OUT]] {
+
+  val LOG = LoggerFactory.getLogger(this.getClass)
+
+  private var function: RichMapFunction[IN, OUT] = null
+
+  override def open(parameters: Configuration): Unit = {
--- End diff --

Do we really need it?
FlatJoinFunction that we are going to use doesn't have a close() method. So 
if we create a close() method here it will look like this:
{{override def close(): Unit = super.close()}}
And {{super.close()}} is also empty.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2734: Keytab & TLS support for Flink on Mesos Setup

[mxm]

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2734
  
Sorry this is taking so long. You you please rebase to the latest master?

@EronWright Could you take a look at the changes?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4872) Type erasure problem exclusively on cluster execution

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696012#comment-15696012
 ] 

ASF GitHub Bot commented on FLINK-4872:
---

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/2823
  
Thanks for the hint @StephanEwen. I improved the PR.


> Type erasure problem exclusively on cluster execution
> -
>
> Key: FLINK-4872
> URL: https://issues.apache.org/jira/browse/FLINK-4872
> Project: Flink
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 1.1.2
>Reporter: Martin Junghanns
>Assignee: Timo Walther
>
> The following codes runs fine on local and collection execution environment 
> but fails when executed on a cluster.
> {code:title=Problem.java}
> import org.apache.flink.api.common.functions.MapFunction;
> import org.apache.flink.api.java.DataSet;
> import org.apache.flink.api.java.ExecutionEnvironment;
> import org.apache.flink.api.java.tuple.Tuple1;
> import java.lang.reflect.Array;
> public class Problem {
>   public static class Pojo {
>   }
>   public static class Foo extends Tuple1 {
>   }
>   public static class Bar extends Tuple1 {
>   }
>   public static class UDF implements MapFunction {
> private final Class clazz;
> public UDF(Class clazz) {
>   this.clazz = clazz;
> }
> @Override
> public Bar map(Foo value) throws Exception {
>   Bar bar = new Bar<>();
>   //noinspection unchecked
>   bar.f0 = (T[]) Array.newInstance(clazz, 10);
>   return bar;
> }
>   }
>   public static void main(String[] args) throws Exception {
> // runs in local, collection and cluster execution
> withLong();
> // runs in local and collection execution, fails on cluster execution
> withPojo();
>   }
>   public static void withLong() throws Exception {
> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
> Foo foo = new Foo<>();
> foo.f0 = 42L;
> DataSet barDataSource = env.fromElements(foo);
> DataSet map = barDataSource.map(new UDF<>(Long.class));
> map.print();
>   }
>   public static void withPojo() throws Exception {
> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
> Foo foo = new Foo<>();
> foo.f0 = new Pojo();
> DataSet barDataSource = env.fromElements(foo);
> DataSet map = barDataSource.map(new UDF<>(Pojo.class));
> map.print();
>   }
> }
> {code}
> {code:title=ProblemTest.java}
> import org.apache.flink.test.util.MultipleProgramsTestBase;
> import org.junit.Test;
> import org.junit.runner.RunWith;
> import org.junit.runners.Parameterized;
> @RunWith(Parameterized.class)
> public class ProblemTest extends MultipleProgramsTestBase {
>   public ProblemTest(TestExecutionMode mode) {
> super(mode);
>   }
>   @Test
>   public void testWithLong() throws Exception {
> Problem.withLong();
>   }
>   @Test
>   public void testWithPOJO() throws Exception {
> Problem.withPojo();
>   }
> }
> {code}
> Exception:
> {code}
> The return type of function 'withPojo(Problem.java:58)' could not be 
> determined automatically, due to type erasure. You can give type information 
> hints by using the returns(...) method on the result of the transformation 
> call, or by letting your function implement the 'ResultTypeQueryable' 
> interface.
> org.apache.flink.api.java.DataSet.getType(DataSet.java:178)
> org.apache.flink.api.java.DataSet.collect(DataSet.java:407)
> org.apache.flink.api.java.DataSet.print(DataSet.java:1605)
> Problem.withPojo(Problem.java:60)
> Problem.main(Problem.java:38) 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2823: [FLINK-4872] [types] Type erasure problem exclusively on ...

[twalthr]

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/2823
  
Thanks for the hint @StephanEwen. I improved the PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3848) Add ProjectableTableSource interface and translation rule

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15695994#comment-15695994
 ] 

ASF GitHub Bot commented on FLINK-3848:
---

Github user wuchong commented on the issue:

https://github.com/apache/flink/pull/2810
  
Hi @tonycox , the overall change looks good to me. I will do more thorough 
review in this weekend.


> Add ProjectableTableSource interface and translation rule
> -
>
> Key: FLINK-3848
> URL: https://issues.apache.org/jira/browse/FLINK-3848
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Reporter: Fabian Hueske
>Assignee: Anton Solovev
>
> Add a {{ProjectableTableSource}} interface for {{TableSource}} implementation 
> that support projection push-down.
> The interface could look as follows
> {code}
> def trait ProjectableTableSource {
>   def setProjection(fields: Array[String]): Unit
> }
> {code}
> In addition we need Calcite rules to push a projection into a TableScan that 
> refers to a {{ProjectableTableSource}}. We might need to tweak the cost model 
> as well to push the optimizer in the right direction.
> Moreover, the {{CsvTableSource}} could be extended to implement 
> {{ProjectableTableSource}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2810: [FLINK-3848] Add ProjectableTableSource interface and tra...

[wuchong]

Github user wuchong commented on the issue:

https://github.com/apache/flink/pull/2810
  
Hi @tonycox , the overall change looks good to me. I will do more thorough 
review in this weekend.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3848) Add ProjectableTableSource interface and translation rule

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15695980#comment-15695980
 ] 

ASF GitHub Bot commented on FLINK-3848:
---

Github user tonycox commented on the issue:

https://github.com/apache/flink/pull/2810
  
@wuchong could you review my changes?
cc @fhueske @StephanEwen 


> Add ProjectableTableSource interface and translation rule
> -
>
> Key: FLINK-3848
> URL: https://issues.apache.org/jira/browse/FLINK-3848
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Reporter: Fabian Hueske
>Assignee: Anton Solovev
>
> Add a {{ProjectableTableSource}} interface for {{TableSource}} implementation 
> that support projection push-down.
> The interface could look as follows
> {code}
> def trait ProjectableTableSource {
>   def setProjection(fields: Array[String]): Unit
> }
> {code}
> In addition we need Calcite rules to push a projection into a TableScan that 
> refers to a {{ProjectableTableSource}}. We might need to tweak the cost model 
> as well to push the optimizer in the right direction.
> Moreover, the {{CsvTableSource}} could be extended to implement 
> {{ProjectableTableSource}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2810: [FLINK-3848] Add ProjectableTableSource interface and tra...

[tonycox]

Github user tonycox commented on the issue:

https://github.com/apache/flink/pull/2810
  
@wuchong could you review my changes?
cc @fhueske @StephanEwen 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-5164) Hadoop-compat IOFormat tests fail on Windows

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-5164:
---

 Summary: Hadoop-compat IOFormat tests fail on Windows
 Key: FLINK-5164
 URL: https://issues.apache.org/jira/browse/FLINK-5164
 Project: Flink
  Issue Type: Bug
  Components: Tests
Affects Versions: 1.1.3
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.2.0


The HaddopMapredITCase fails on windows with the following exception:
{code}
java.lang.NullPointerException
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:445)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at 
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
at 
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:631)
at 
org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:468)
at 
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:456)
at 
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:424)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:907)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:800)
at 
org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)
at 
org.apache.flink.api.java.hadoop.mapred.HadoopOutputFormatBase.open(HadoopOutputFormatBase.java:145)
at 
org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:178)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:654)
at java.lang.Thread.run(Thread.java:745)
{code}

I suggest to disable the test on windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5055) Security feature crashes JM for certain Hadoop versions even though using no Kerberos

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15695931#comment-15695931
 ] 

ASF GitHub Bot commented on FLINK-5055:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2864


> Security feature crashes JM for certain Hadoop versions even though using no 
> Kerberos
> -
>
> Key: FLINK-5055
> URL: https://issues.apache.org/jira/browse/FLINK-5055
> Project: Flink
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Maximilian Michels
>Priority: Critical
> Fix For: 1.2.0
>
>
> A user reported [1] that the {{JobManager}} does not start when using Flink 
> with Hadoop-2.7.0-mapr-1607 and no security activated because of 
> {code}
> javax.security.auth.login.LoginException: Unable to obtain Principal Name for 
> authentication
> at 
> com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:841)
> at 
> com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:704)
> at 
> com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
> {code}
> It seems that this Hadoop version always tries to login via Kerberos even 
> though the user did not activate it and, thus, should use 
> {{AuthenticationMode.SIMPLE}}.
> I'm not really familiar with the security feature, but my understanding is 
> that it should not have any effect on Flink when not activated. I might be 
> wrong here, but if not, then we should fix this problem for 1.2.0 because it 
> prevents people from using Flink.
> [1] 
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Flink-using-Yarn-on-MapR-td14484.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2864: [FLINK-5055][security] skip Hadoop UGI login if un...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2864


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3555) Web interface does not render job information properly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15695905#comment-15695905
 ] 

ASF GitHub Bot commented on FLINK-3555:
---

GitHub user bitchelov opened a pull request:

https://github.com/apache/flink/pull/2869

[FLINK-3555] Web interface does not render job information properly

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [ ] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [ ] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [ ] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bitchelov/flink FLINK-3555

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2869.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2869


commit cd3d79cc704f45d41c494173abe16bc98d818b86
Author: sergey_sokur 
Date:   2016-11-25T11:45:08Z

[FLINK-3555] Web interface does not render job information properly




> Web interface does not render job information properly
> --
>
> Key: FLINK-3555
> URL: https://issues.apache.org/jira/browse/FLINK-3555
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Sergey Sokur
>Priority: Minor
> Attachments: Chrome.png, Safari.png
>
>
> In Chrome and Safari, the different tabs of the detailed job view are not 
> properly rendered. The text goes beyond the surrounding box. I would guess 
> that this is some kind of css issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2868: [FLINK-4565] Support for SQL IN operator

[NickolayVasilishin]

Github user NickolayVasilishin closed the pull request at:

https://github.com/apache/flink/pull/2868


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4565) Support for SQL IN operator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15695886#comment-15695886
 ] 

ASF GitHub Bot commented on FLINK-4565:
---

Github user NickolayVasilishin closed the pull request at:

https://github.com/apache/flink/pull/2868


> Support for SQL IN operator
> ---
>
> Key: FLINK-4565
> URL: https://issues.apache.org/jira/browse/FLINK-4565
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Nikolay Vasilishin
>
> It seems that Flink SQL supports the uncorrelated sub-query IN operator. But 
> it should also be available in the Table API and tested.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3160) Aggregate operator statistics by TaskManager

[Ufuk Celebi (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15695879#comment-15695879
 ] 

Ufuk Celebi commented on FLINK-3160:


I just noticed that we overlooked something: currently both the subtasks _and_ 
the taskmanagers view sort by host. This should have been only added for the 
taskmanagers tab.

Ideally, sorting by a different column should not require us to add a separate 
tab. Would that be a good alternative to solve this, Greg? For now I would like 
to revert the change in {{web/partials/jobs/job.plan.node.subtasks.html}} and 
remove the {{orderBy: host}}.

> Aggregate operator statistics by TaskManager
> 
>
> Key: FLINK-3160
> URL: https://issues.apache.org/jira/browse/FLINK-3160
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
> Fix For: 1.0.0
>
>
> The web client job info page presents a table of the following per task 
> statistics: start time, end time, duration, bytes received, records received, 
> bytes sent, records sent, attempt, host, status.
> Flink supports clusters with thousands of slots and a job setting a high 
> parallelism renders this job info page unwieldy and difficult to analyze in 
> real-time.
> It would be helpful to optionally or automatically aggregate statistics by 
> TaskManager. These rows could then be expanded to reveal the current per task 
> statistics.
> Start time, end time, duration, and attempt are not applicable to a 
> TaskManager since new tasks for repeated attempts may be started. Bytes 
> received, records received, bytes sent, and records sent are summed. Any 
> throughput metrics can be averaged over the total task time or time window. 
> Status could reference the number of running tasks on the TaskManager or an 
> idle state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-5163) Make the production functions rescalable (apart from the Rolling/Bucketing Sinks)

[Kostas Kloudas (JIRA)]

Kostas Kloudas created FLINK-5163:
-

 Summary: Make the production functions rescalable (apart from the 
Rolling/Bucketing Sinks)
 Key: FLINK-5163
 URL: https://issues.apache.org/jira/browse/FLINK-5163
 Project: Flink
  Issue Type: Improvement
Affects Versions: 1.2.0
Reporter: Kostas Kloudas
Assignee: Kostas Kloudas
 Fix For: 1.2.0


This issue targets porting all the functions in the production code to the new 
state abstractions. These functions are:
1) StatefulSequenceSource
2) MessageAcknowledgingSourceBase
3) FromElementsFunction
4) ContinuousFileMonitoringFunction



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2868: [FLINK-4565] Support for SQL IN operator

[NickolayVasilishin]

GitHub user NickolayVasilishin opened a pull request:

https://github.com/apache/flink/pull/2868

[FLINK-4565] Support for SQL IN operator

This PR is a part of work on SQL IN operator in Table API, which implements 
IN for literals.
Two cases are covered: less and great then 20 literals.

Also I have some questions:
- converting all numeric types to BigDecimal isn't ok? I decided to make so 
to simplify use of hashset.
- validation isn't really good. It forces to use operator with same type 
literals. Should I rework it or maybe just add more cases?

expressionDsl.scala:
entry point for IN operator in scala API
ScalarOperators.scala:
1) All numeric types are upcasting to BigDecimal for using in hashset, 
other types are unchanged in castNumeric
2) valuesInitialization used for 2 cases: when we have more then 20 
operands (then we use hashset,  initialized in constructor, descibed below) 
and less then 20 operands (then we initialize operands in   method's body and 
use them in conjunction)
3) comparison also covers described above cases. In first case we use 
callback to declare and initializehashset with all operands. Otherwise 
we just put all operands in conjunction.
4) Final code is built up with these code snippets.
CodeGenerator.scala:
passes arguments and callback to declare and init hashet
FunctionCatalog.scala:
registers "in" as method
InITCase:
some use cases

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/NickolayVasilishin/flink FLINK-4565

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2868.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2868


commit 7699c37a840dcb62c4984e0d63f87813d355d1f8
Author: nikolay_vasilishin 
Date:   2016-11-25T12:44:53Z

[FLINK-4565] Support for SQL IN operator

expressionDsl.scala:
entry point for IN operator in scala API
ScalarOperators.scala:
1) All numeric types are upcasting to BigDecimal for using in hashset, 
other types are unchanged in castNumeric
2) valuesInitialization used for 2 cases: when we have more then 20 
operands (then we use hashset,  initialized in constructor, descibed below) 
and less then 20 operands (then we initialize operands in   method's body and 
use them in conjunction)
3) comparison also covers described above cases. In first case we use 
callback to declare and initializehashset with all operands. Otherwise 
we just put all operands in conjunction.
4) Final code is built up with these code snippets.
CodeGenerator.scala:
passes arguments and callback to declare and init hashet
FunctionCatalog.scala:
registers "in" as method
InITCase:
some use cases




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4565) Support for SQL IN operator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15695836#comment-15695836
 ] 

ASF GitHub Bot commented on FLINK-4565:
---

GitHub user NickolayVasilishin opened a pull request:

https://github.com/apache/flink/pull/2868

[FLINK-4565] Support for SQL IN operator

This PR is a part of work on SQL IN operator in Table API, which implements 
IN for literals.
Two cases are covered: less and great then 20 literals.

Also I have some questions:
- converting all numeric types to BigDecimal isn't ok? I decided to make so 
to simplify use of hashset.
- validation isn't really good. It forces to use operator with same type 
literals. Should I rework it or maybe just add more cases?

expressionDsl.scala:
entry point for IN operator in scala API
ScalarOperators.scala:
1) All numeric types are upcasting to BigDecimal for using in hashset, 
other types are unchanged in castNumeric
2) valuesInitialization used for 2 cases: when we have more then 20 
operands (then we use hashset,  initialized in constructor, descibed below) 
and less then 20 operands (then we initialize operands in   method's body and 
use them in conjunction)
3) comparison also covers described above cases. In first case we use 
callback to declare and initializehashset with all operands. Otherwise 
we just put all operands in conjunction.
4) Final code is built up with these code snippets.
CodeGenerator.scala:
passes arguments and callback to declare and init hashet
FunctionCatalog.scala:
registers "in" as method
InITCase:
some use cases

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/NickolayVasilishin/flink FLINK-4565

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2868.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2868


commit 7699c37a840dcb62c4984e0d63f87813d355d1f8
Author: nikolay_vasilishin 
Date:   2016-11-25T12:44:53Z

[FLINK-4565] Support for SQL IN operator

expressionDsl.scala:
entry point for IN operator in scala API
ScalarOperators.scala:
1) All numeric types are upcasting to BigDecimal for using in hashset, 
other types are unchanged in castNumeric
2) valuesInitialization used for 2 cases: when we have more then 20 
operands (then we use hashset,  initialized in constructor, descibed below) 
and less then 20 operands (then we initialize operands in   method's body and 
use them in conjunction)
3) comparison also covers described above cases. In first case we use 
callback to declare and initializehashset with all operands. Otherwise 
we just put all operands in conjunction.
4) Final code is built up with these code snippets.
CodeGenerator.scala:
passes arguments and callback to declare and init hashet
FunctionCatalog.scala:
registers "in" as method
InITCase:
some use cases




> Support for SQL IN operator
> ---
>
> Key: FLINK-4565
> URL: https://issues.apache.org/jira/browse/FLINK-4565
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Nikolay Vasilishin
>
> It seems that Flink SQL supports the uncorrelated sub-query IN operator. But 
> it should also be available in the Table API and tested.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2845: [FLINK-5096] Make the RollingSink rescalable.

[kl0u]

Github user kl0u commented on the issue:

https://github.com/apache/flink/pull/2845
  
Thanks!



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5096) Make the RollingSink rescalable.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15695816#comment-15695816
 ] 

ASF GitHub Bot commented on FLINK-5096:
---

Github user kl0u commented on the issue:

https://github.com/apache/flink/pull/2845
  
Thanks!



> Make the RollingSink rescalable.
> 
>
> Key: FLINK-5096
> URL: https://issues.apache.org/jira/browse/FLINK-5096
> Project: Flink
>  Issue Type: Improvement
>  Components: filesystem-connector
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
> Fix For: 1.2.0
>
>
> Integrate the RollingSink with the new state abstractions so that its 
> parallelism can change after restoring from a savepoint.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5096) Make the RollingSink rescalable.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15695815#comment-15695815
 ] 

ASF GitHub Bot commented on FLINK-5096:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2845
  
Great, will start merging this.


> Make the RollingSink rescalable.
> 
>
> Key: FLINK-5096
> URL: https://issues.apache.org/jira/browse/FLINK-5096
> Project: Flink
>  Issue Type: Improvement
>  Components: filesystem-connector
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
> Fix For: 1.2.0
>
>
> Integrate the RollingSink with the new state abstractions so that its 
> parallelism can change after restoring from a savepoint.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2845: [FLINK-5096] Make the RollingSink rescalable.

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2845
  
Great, will start merging this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5092) Add integration with Sonarqube and code coverage

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15695811#comment-15695811
 ] 

ASF GitHub Bot commented on FLINK-5092:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2836
  
It does not: https://github.com/jacoco/jacoco/wiki/MavenMultiModule

That's a bummer but apparently there are some workarounds:
http://www.thinkcode.se/blog/2012/02/18/test-coverage-in-a-
multi-module-maven-project

On Fri, Nov 25, 2016 at 1:38 PM, zentol  wrote:

> The plugin cannot detect cross-module coverage, correct? as in, all the
> tests in flink-tests will not contribute in any way to the coverage of
> other modules?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



> Add integration with Sonarqube and code coverage
> 
>
> Key: FLINK-5092
> URL: https://issues.apache.org/jira/browse/FLINK-5092
> Project: Flink
>  Issue Type: Improvement
>  Components: Tests
>Reporter: Boris Osipov
>Assignee: Boris Osipov
>
> It would be good to have the opportunity to generate test coverage reports 
> for Flink and analyze code by SonarQube.
> Parts of tasks:
> - add generate test coverage reports for Flink with new maven profile
> - implement integration with https://analysis.apache.org/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)