[jira] [Created] (FLINK-7394) Implement basic InputChannel for credit-based logic

[zhijiang (JIRA)]

zhijiang created FLINK-7394:
---

 Summary: Implement basic InputChannel for credit-based logic
 Key: FLINK-7394
 URL: https://issues.apache.org/jira/browse/FLINK-7394
 Project: Flink
  Issue Type: Sub-task
  Components: Core
Reporter: zhijiang
Assignee: zhijiang
 Fix For: 1.4.0


This is a part of work for credit-based network flow control. 

The basic works are:

* The exclusive buffers per channel are assigned to {{RemoteInputChannel}} 
directly during registering task.
* {{RemoteInputChannel}} implements {{BufferRecycler}} interface to manage the 
exclusive buffers itself.
* {{RemoteInputChannel}} implements {{BufferPoolListener}} interface to be 
notified available floating buffers from buffer pool.
* {{RemoteInputChannel}} maintains unannounced credit and current sender 
backlog.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6988) Add Apache Kafka 0.11 connector

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119268#comment-16119268
 ] 

ASF GitHub Bot commented on FLINK-6988:
---

Github user rangadi commented on the issue:

https://github.com/apache/flink/pull/4239
  
How does exactly-once sink handle large gap between `preCommit()` and 
`recoverAndCommit()` in case of a recovery? The server seems to abort a 
transaction after a timeout `max.transaction.timeout.ms`. 


> Add Apache Kafka 0.11 connector
> ---
>
> Key: FLINK-6988
> URL: https://issues.apache.org/jira/browse/FLINK-6988
> Project: Flink
>  Issue Type: Improvement
>  Components: Kafka Connector
>Affects Versions: 1.3.1
>Reporter: Piotr Nowojski
>Assignee: Piotr Nowojski
>
> Kafka 0.11 (it will be released very soon) add supports for transactions. 
> Thanks to that, Flink might be able to implement Kafka sink supporting 
> "exactly-once" semantic. API changes and whole transactions support is 
> described in 
> [KIP-98|https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging].
> The goal is to mimic implementation of existing BucketingSink. New 
> FlinkKafkaProducer011 would 
> * upon creation begin transaction, store transaction identifiers into the 
> state and would write all incoming data to an output Kafka topic using that 
> transaction
> * on `snapshotState` call, it would flush the data and write in state 
> information that current transaction is pending to be committed
> * on `notifyCheckpointComplete` we would commit this pending transaction
> * in case of crash between `snapshotState` and `notifyCheckpointComplete` we 
> either abort this pending transaction (if not every participant successfully 
> saved the snapshot) or restore and commit it. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4239: [FLINK-6988] flink-connector-kafka-0.11 with exactly-once...

[rangadi]

Github user rangadi commented on the issue:

https://github.com/apache/flink/pull/4239
  
How does exactly-once sink handle large gap between `preCommit()` and 
`recoverAndCommit()` in case of a recovery? The server seems to abort a 
transaction after a timeout `max.transaction.timeout.ms`. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7223) Increase DEFAULT_SHARD_DISCOVERY_INTERVAL_MILLIS for Flink-kinesis-connector

[Bowen Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119222#comment-16119222
 ] 

Bowen Li commented on FLINK-7223:
-

I'm actually fine with the default value, since we are already aware of this 
issue, and we override the default value ourself. But from a enterprise user's 
point of view, the assumption of 'this means running up to 50 Flink jobs per 
account' is not practical at all for a big AWS enterprise customer. Here's why:

In an enterprise AWS account, lots of other services are contributing to 
saturating the 10requests/sec throughput. In our (OfferUp, inc) prod AWS 
account, we are hitting that cap even before having Flink. Having more and more 
Flink jobs makes things worse, and breaks other services. Our Flink jobs are 
not more important than other services, so we can't either allocate resources 
solely for Flink or compete with other services. Thus we set Flink's discovery 
interval to be {{1 hour}}. 

It's probably not a topic worths a long discussion :) If you guys feel the 
default value is ok, I'll good with it

> Increase DEFAULT_SHARD_DISCOVERY_INTERVAL_MILLIS for Flink-kinesis-connector
> 
>
> Key: FLINK-7223
> URL: https://issues.apache.org/jira/browse/FLINK-7223
> Project: Flink
>  Issue Type: Improvement
>  Components: Kinesis Connector
>Affects Versions: 1.3.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Minor
> Fix For: 1.4.0
>
>
> Background: {{DEFAULT_SHARD_DISCOVERY_INTERVAL_MILLIS}} in 
> {{org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants}}
>  is the default value for Flink to call Kinesis's {{describeStream()}} API.
> Problem: Right now, its value is 10,000millis (10sec), which is too short. We 
> ran into problems that Flink-kinesis-connector's call of {{describeStream()}} 
> exceeds Kinesis rate limit, and broken Flink taskmanager.
> According to 
> http://docs.aws.amazon.com/kinesis/latest/APIReference/API_DescribeStream.html,
>  
> "This operation has a limit of 10 transactions per second per account.". What 
> it means is that the 10transaction/account is a limit on a single 
> organization's AWS account..:(  We contacted AWS Support, and confirmed 
> this. If you have more applications (either other Flink apps or non-Flink 
> apps) competing aggressively with your Flink app on this API, your Flink app 
> breaks. 
> I propose increasing the value DEFAULT_SHARD_DISCOVERY_INTERVAL_MILLIS from 
> 10,000millis(10sec) to preferably 300,000 (5min). Or at least 60,000 (1min) 
> if anyone has a solid reason arguing that 5min is too long, 
> This is also related to https://issues.apache.org/jira/browse/FLINK-6365



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7367) Parameterize more configs for FlinkKinesisProducer (RecordMaxBufferedTime, MaxConnections, RequestTimeout, etc)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119163#comment-16119163
 ] 

ASF GitHub Bot commented on FLINK-7367:
---

Github user bowenli86 commented on the issue:

https://github.com/apache/flink/pull/4473
  
Trying to understand the keys. Are keys like 
"aws.producer.collectionMaxCount" specified in flink-conf.yaml?



> Parameterize more configs for FlinkKinesisProducer (RecordMaxBufferedTime, 
> MaxConnections, RequestTimeout, etc)
> ---
>
> Key: FLINK-7367
> URL: https://issues.apache.org/jira/browse/FLINK-7367
> Project: Flink
>  Issue Type: Improvement
>  Components: Kinesis Connector
>Affects Versions: 1.3.0
>Reporter: Bowen Li
>Assignee: Bowen Li
> Fix For: 1.3.3
>
>
> Right now, FlinkKinesisProducer only expose two configs for the underlying 
> KinesisProducer:
> - AGGREGATION_MAX_COUNT
> - COLLECTION_MAX_COUNT
> Well, according to [AWS 
> doc|http://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-config.html] 
> and [their sample on 
> github|https://github.com/awslabs/amazon-kinesis-producer/blob/master/java/amazon-kinesis-producer-sample/default_config.properties],
>  developers can set more to make the max use of KinesisProducer, and make it 
> fault-tolerant (e.g. by increasing timeout).
> I select a few more configs that we need when using Flink with Kinesis:
> - MAX_CONNECTIONS
> - RATE_LIMIT
> - RECORD_MAX_BUFFERED_TIME
> - RECORD_TIME_TO_LIVE
> - REQUEST_TIMEOUT
> We need to parameterize FlinkKinesisProducer to pass in the above params, in 
> order to cater to our need



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4473: [FLINK-7367][kinesis connector] Parameterize more configs...

[bowenli86]

Github user bowenli86 commented on the issue:

https://github.com/apache/flink/pull/4473
  
Trying to understand the keys. Are keys like 
"aws.producer.collectionMaxCount" specified in flink-conf.yaml?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7240) Externalized RocksDB can fail with stackoverflow

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119050#comment-16119050
 ] 

ASF GitHub Bot commented on FLINK-7240:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/4497
  
I actually think that this does not fully stabilize the test case. The 
problem is the following: Waiting for all `Tasks` to be in state `RUNNING` is 
not sufficient but only necessary. The `StreamTask` can still be not running 
and thus the checkpoint gets rejected. I propose to add a `CountDownLatch` to 
the source function to signal when all sources are really running. Only then we 
will trigger the checkpoint.


> Externalized RocksDB can fail with stackoverflow
> 
>
> Key: FLINK-7240
> URL: https://issues.apache.org/jira/browse/FLINK-7240
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing, Tests
>Affects Versions: 1.3.1, 1.4.0
> Environment: https://travis-ci.org/zentol/flink/jobs/255760513
>Reporter: Chesnay Schepler
>Assignee: Till Rohrmann
>Priority: Critical
>  Labels: test-stability
>
> {code}
> testExternalizedFullRocksDBCheckpointsStandalone(org.apache.flink.test.checkpointing.ExternalizedCheckpointITCase)
>   Time elapsed: 146.894 sec  <<< ERROR!
> java.lang.StackOverflowError: null
>   at java.util.Hashtable.get(Hashtable.java:363)
>   at java.util.Properties.getProperty(Properties.java:969)
>   at java.lang.System.getProperty(System.java:720)
>   at sun.security.action.GetPropertyAction.run(GetPropertyAction.java:84)
>   at sun.security.action.GetPropertyAction.run(GetPropertyAction.java:49)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.io.PrintWriter.(PrintWriter.java:116)
>   at java.io.PrintWriter.(PrintWriter.java:100)
>   at 
> org.apache.log4j.DefaultThrowableRenderer.render(DefaultThrowableRenderer.java:58)
>   at 
> org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:87)
>   at 
> org.apache.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:413)
>   at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:313)
>   at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
>   at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
>   at 
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
>   at org.apache.log4j.Category.callAppenders(Category.java:206)
>   at org.apache.log4j.Category.forcedLog(Category.java:391)
>   at org.apache.log4j.Category.log(Category.java:856)
>   at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:381)
>   at 
> org.apache.flink.runtime.testingUtils.TestingCluster.requestCheckpoint(TestingCluster.scala:392)
>   at 
> org.apache.flink.runtime.testingUtils.TestingCluster.requestCheckpoint(TestingCluster.scala:394)
>   at 
> org.apache.flink.runtime.testingUtils.TestingCluster.requestCheckpoint(TestingCluster.scala:394)
>   at 
> org.apache.flink.runtime.testingUtils.TestingCluster.requestCheckpoint(TestingCluster.scala:394)
>   at 
> org.apache.flink.runtime.testingUtils.TestingCluster.requestCheckpoint(TestingCluster.scala:394)
>   at 
> org.apache.flink.runtime.testingUtils.TestingCluster.requestCheckpoint(TestingCluster.scala:394)
>   at 
> org.apache.flink.runtime.testingUtils.TestingCluster.requestCheckpoint(TestingCluster.scala:394)
>   at 
> org.apache.flink.runtime.testingUtils.TestingCluster.requestCheckpoint(TestingCluster.scala:394)
> ...
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4497: [FLINK-7240] [tests] Stabilize ExternalizedCheckpointITCa...

[tillrohrmann]

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/4497
  
I actually think that this does not fully stabilize the test case. The 
problem is the following: Waiting for all `Tasks` to be in state `RUNNING` is 
not sufficient but only necessary. The `StreamTask` can still be not running 
and thus the checkpoint gets rejected. I propose to add a `CountDownLatch` to 
the source function to signal when all sources are really running. Only then we 
will trigger the checkpoint.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7026) Add shaded asm dependency

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119032#comment-16119032
 ] 

ASF GitHub Bot commented on FLINK-7026:
---

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/4494
  
+1 LGTM


> Add shaded asm dependency
> -
>
> Key: FLINK-7026
> URL: https://issues.apache.org/jira/browse/FLINK-7026
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4494: [FLINK-7026] Introduce flink-shaded-asm-5

[greghogan]

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/4494
  
+1 LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6281) Create TableSink for JDBC

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118969#comment-16118969
 ] 

ASF GitHub Bot commented on FLINK-6281:
---

Github user haohui commented on a diff in the pull request:

https://github.com/apache/flink/pull/3712#discussion_r132022891
  
--- Diff: 
flink-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCAppendTableSink.java
 ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.java.io.jdbc;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.table.sinks.AppendStreamTableSink;
+import org.apache.flink.table.sinks.BatchTableSink;
+import org.apache.flink.table.sinks.TableSink;
+import org.apache.flink.types.Row;
+
+/**
+ * An at-least-once Table sink for JDBC.
+ */
+public class JDBCAppendTableSink implements AppendStreamTableSink, 
BatchTableSink {
+   private final JDBCSinkFunction sink;
+
+   private String[] fieldNames;
+   private TypeInformation[] fieldTypes;
+
+   JDBCAppendTableSink(JDBCOutputFormat outputFormat) {
+   this.sink = new JDBCSinkFunction(outputFormat);
+   }
+
+   public static JDBCAppendTableSinkBuilder builder() {
+   return new JDBCAppendTableSinkBuilder();
+   }
+
+   @Override
+   public void emitDataStream(DataStream dataStream) {
+   dataStream.addSink(sink);
+   }
+
+   @Override
+   public void emitDataSet(DataSet dataSet) {
+   dataSet.output(sink.outputFormat);
+   }
+
+   @Override
+   public TypeInformation getOutputType() {
+   return new RowTypeInfo(fieldTypes, fieldNames);
+   }
+
+   @Override
+   public String[] getFieldNames() {
+   return fieldNames;
+   }
+
+   @Override
+   public TypeInformation[] getFieldTypes() {
+   return fieldTypes;
+   }
+
+   @Override
+   public TableSink configure(String[] fieldNames, 
TypeInformation[] fieldTypes) {
+   JDBCAppendTableSink copy = new 
JDBCAppendTableSink(sink.outputFormat);
--- End diff --

The `JDBCOutputFormat` now is only constructed via 
`JDBCAppendableSinkBuilder`, thus the types should always match, but it is a 
good idea to add the checks to catch potential bugs.


> Create TableSink for JDBC
> -
>
> Key: FLINK-6281
> URL: https://issues.apache.org/jira/browse/FLINK-6281
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>
> It would be nice to integrate the table APIs with the JDBC connectors so that 
> the rows in the tables can be directly pushed into JDBC.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #3712: [FLINK-6281] Create TableSink for JDBC.

[haohui]

Github user haohui commented on a diff in the pull request:

https://github.com/apache/flink/pull/3712#discussion_r132022891
  
--- Diff: 
flink-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCAppendTableSink.java
 ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.java.io.jdbc;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.table.sinks.AppendStreamTableSink;
+import org.apache.flink.table.sinks.BatchTableSink;
+import org.apache.flink.table.sinks.TableSink;
+import org.apache.flink.types.Row;
+
+/**
+ * An at-least-once Table sink for JDBC.
+ */
+public class JDBCAppendTableSink implements AppendStreamTableSink, 
BatchTableSink {
+   private final JDBCSinkFunction sink;
+
+   private String[] fieldNames;
+   private TypeInformation[] fieldTypes;
+
+   JDBCAppendTableSink(JDBCOutputFormat outputFormat) {
+   this.sink = new JDBCSinkFunction(outputFormat);
+   }
+
+   public static JDBCAppendTableSinkBuilder builder() {
+   return new JDBCAppendTableSinkBuilder();
+   }
+
+   @Override
+   public void emitDataStream(DataStream dataStream) {
+   dataStream.addSink(sink);
+   }
+
+   @Override
+   public void emitDataSet(DataSet dataSet) {
+   dataSet.output(sink.outputFormat);
+   }
+
+   @Override
+   public TypeInformation getOutputType() {
+   return new RowTypeInfo(fieldTypes, fieldNames);
+   }
+
+   @Override
+   public String[] getFieldNames() {
+   return fieldNames;
+   }
+
+   @Override
+   public TypeInformation[] getFieldTypes() {
+   return fieldTypes;
+   }
+
+   @Override
+   public TableSink configure(String[] fieldNames, 
TypeInformation[] fieldTypes) {
+   JDBCAppendTableSink copy = new 
JDBCAppendTableSink(sink.outputFormat);
--- End diff --

The `JDBCOutputFormat` now is only constructed via 
`JDBCAppendableSinkBuilder`, thus the types should always match, but it is a 
good idea to add the checks to catch potential bugs.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-7393) Refactor FlinkKinesisConsumerTest

[Bowen Li (JIRA)]

Bowen Li created FLINK-7393:
---

 Summary: Refactor FlinkKinesisConsumerTest
 Key: FLINK-7393
 URL: https://issues.apache.org/jira/browse/FLINK-7393
 Project: Flink
  Issue Type: Test
  Components: Kinesis Connector
Affects Versions: 1.3.2
Reporter: Bowen Li
Assignee: Bowen Li
Priority: Minor
 Fix For: 1.4.0, 1.3.3


Right now, 
[{{FlinkKinesisConsumerTest}}|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/test/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumerTest.java]
 has lots of tests that actually should belong to {{KinesisConfigUtil}}, e.g. 
all the {{validateXxxConfiguration()}}

We need to move those tests out to a new file {{KinesisConfigUtilTest}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (FLINK-7159) Semantics of OVERLAPS in Table API diverge from the SQL standard

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved FLINK-7159.
---
Resolution: Fixed

This has been resolved as a part of FLINK-6429.

> Semantics of OVERLAPS in Table API diverge from the SQL standard
> 
>
> Key: FLINK-7159
> URL: https://issues.apache.org/jira/browse/FLINK-7159
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>
> According to http://web.cecs.pdx.edu/~len/sql1999.pdf
> ISO/IEC 9075-2:1999 (E) ©ISO/IEC, 8.12 
> {noformat}
> The result of the  is the result of the following 
> expression:
> ( S1 > S2 AND NOT ( S1 >= T2 AND T1 >= T2 ) )
> OR
> ( S2 > S1 AND NOT ( S2 >= T1 AND T2 >= T1 ) )
> OR
> ( S1 = S2 AND ( T1 <> T2 OR T1 = T2 ) )
> {noformat}
> The Table API diverges from this semantic.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7245) Enhance the operators to support holding back watermarks

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118649#comment-16118649
 ] 

Aljoscha Krettek commented on FLINK-7245:
-

Right now, this sounds more like an implementation detail that could be 
implemented in a custom operator, yes.

For this to be in the public API we have to know that this is also useful for 
other cases/users because we then have the burden of supporting it.

> Enhance the operators to support holding back watermarks
> 
>
> Key: FLINK-7245
> URL: https://issues.apache.org/jira/browse/FLINK-7245
> Project: Flink
>  Issue Type: New Feature
>  Components: DataStream API
>Reporter: Xingcan Cui
>Assignee: Xingcan Cui
>
> Currently the watermarks are applied and emitted by the 
> {{AbstractStreamOperator}} instantly. 
> {code:java}
> public void processWatermark(Watermark mark) throws Exception {
>   if (timeServiceManager != null) {
>   timeServiceManager.advanceWatermark(mark);
>   }
>   output.emitWatermark(mark);
> }
> {code}
> Some calculation results (with timestamp fields) triggered by these 
> watermarks (e.g., join or aggregate results) may be regarded as delayed by 
> the downstream operators since their timestamps must be less than or equal to 
> the corresponding triggers. 
> This issue aims to add another "working mode", which supports holding back 
> watermarks, to current operators. These watermarks should be blocked and 
> stored by the operators until all the corresponding new generated results are 
> emitted.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (FLINK-7051) Bump up Calcite version to 1.14

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai reassigned FLINK-7051:
-

Assignee: Haohui Mai

> Bump up Calcite version to 1.14
> ---
>
> Key: FLINK-7051
> URL: https://issues.apache.org/jira/browse/FLINK-7051
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Haohui Mai
>
> This is an umbrella issue for all tasks that need to be done once Apache 
> Calcite 1.14 is released.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7361) flink-web doesn't build with ruby 2.4

[Chesnay Schepler (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118596#comment-16118596
 ] 

Chesnay Schepler commented on FLINK-7361:
-

Let's add it as a direct dependency, we did the same for the flink docs in 
FLINK-6302.

> flink-web doesn't build with ruby 2.4
> -
>
> Key: FLINK-7361
> URL: https://issues.apache.org/jira/browse/FLINK-7361
> Project: Flink
>  Issue Type: Bug
>  Components: Project Website
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>
> The dependencies pulled in by the old jekyll version do not build with ruby 
> 2.4 and fail with something like
> {code}
> yajl_ext.c:881:22: error: 'rb_cFixnum' undeclared (first use in this 
> function); did you mean 'rb_isalnum'?
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7392) Enable more predicate push-down in joins

[Haohui Mai (JIRA)]

Haohui Mai created FLINK-7392:
-

 Summary: Enable more predicate push-down in joins
 Key: FLINK-7392
 URL: https://issues.apache.org/jira/browse/FLINK-7392
 Project: Flink
  Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai


This is a follow-up of FLINK-6429.

As a quick workaround to prevent pushing down projections for time indicators, 
FLINK-6429 reverts the behavior of {{ProjectJoinTransposeRule}} back to the one 
in Calcite 1.12.

As [~jark] suggested in FLINK-6429, we can selectively disable the push down 
for time indicators in {{ProjectJoinTransposeRule}}. This jira tracks the 
effort of implement the suggestion.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7361) flink-web doesn't build with ruby 2.4

[Nico Kruber (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118571#comment-16118571
 ] 

Nico Kruber commented on FLINK-7361:


I did some more digging and found out, that `yajl-ruby` 1.2.2 from 5th April 
2017 solves this issue as well and it looks like if you start from scratch, 
this version will be pulled instead of 1.2.1 which was pulled before.
At least, now, I cannot reproduce the issue any more starting from a clean 
checkout.

I assume, we could also set this as a direct dependency to be sure...

> flink-web doesn't build with ruby 2.4
> -
>
> Key: FLINK-7361
> URL: https://issues.apache.org/jira/browse/FLINK-7361
> Project: Flink
>  Issue Type: Bug
>  Components: Project Website
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>
> The dependencies pulled in by the old jekyll version do not build with ruby 
> 2.4 and fail with something like
> {code}
> yajl_ext.c:881:22: error: 'rb_cFixnum' undeclared (first use in this 
> function); did you mean 'rb_isalnum'?
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (FLINK-7373) Using URLClassLoader to load UDF triggers HepPlanner unexpected ClassNotFoundException

[Rong Rong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rong Rong reassigned FLINK-7373:


Assignee: Rong Rong

> Using URLClassLoader to load UDF triggers HepPlanner unexpected 
> ClassNotFoundException
> --
>
> Key: FLINK-7373
> URL: https://issues.apache.org/jira/browse/FLINK-7373
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Affects Versions: 1.3.1
>Reporter: Rong Rong
>Assignee: Rong Rong
>
> Using URLClassLoader to load, say from Artifactory, and instantiate UDF 
> instances will cause some Rule failed during runHepPlanner or 
> runVolcanoPlanner.
> One example could add an ITCase in:
> {code:title=flink-libraries/flink-table/src/test/scala/org/apache/flink/table/runtime/batch/table/CalcITCase.scala}
>   @Test
>   def testUserDefinedFunctionDynamicClassloader() {
> val env = ExecutionEnvironment.getExecutionEnvironment
> val tableEnv = TableEnvironment.getTableEnvironment(env, config)
> val jarFileURI = "file://org/apache/flink/table/udf/HelloWorld.jar"
> val udfClassLoader: ClassLoader = new URLClassLoader(List(new 
> URI(jarFileURI).toURL).toArray)
> val clazz = 
> udfClassLoader.loadClass("org.apache.flink.table.udf.HelloWorld")
> val helloWorldUDF: ScalarFunction = 
> clazz.newInstance().asInstanceOf[ScalarFunction]
> tableEnv.registerFunction("helloWorld", helloWorldUDF)
> val table = env.fromElements("a", "b", "c").toTable(tableEnv, 'text)
> val result = table.select("text.helloWorld()")
> val results = result.toDataSet[Row].collect()
> val expected = "Hello World"
> TestBaseUtils.compareResultAsText(results.asJava, expected)
>   }
> {code}
> where
> {code:title=HelloWorld.java}
> package org.apache.flink.table.udf;
> import org.apache.flink.table.functions.ScalarFunction;
> public class HelloWorld extends ScalarFunction {
>   public String eval() {
> return "Hello World";
>   }
> }
> {code}
> This triggers the following Exception:
> {panel:title=Exception}
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be 
> compiled. This is a bug. Please file an issue.
>   at 
> org.apache.flink.table.codegen.Compiler$class.compile(Compiler.scala:36)
>   at 
> org.apache.flink.table.runtime.FlatMapRunner.compile(FlatMapRunner.scala:31)
>   at 
> org.apache.flink.table.runtime.FlatMapRunner.open(FlatMapRunner.scala:45)
>   at 
> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
>   at 
> org.apache.flink.api.common.operators.base.FlatMapOperatorBase.executeOnCollections(FlatMapOperatorBase.java:62)
> ..
> Caused by: org.codehaus.commons.compiler.CompileException: Line 6, Column 22: 
> Cannot determine simple type name "org"
>   at 
> org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:11672)
>   at 
> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:6416)
>   at 
> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:6177)
> ..
> {panel}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7316) always use off-heap network buffers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118511#comment-16118511
 ] 

ASF GitHub Bot commented on FLINK-7316:
---

Github user NicoK commented on the issue:

https://github.com/apache/flink/pull/4481
  
ok, one test fixed, the other is not so simple but maybe @tillrohrmann can 
help with it:

Inside `ContaineredTaskManagerParameters#create()`, we calculate the amount 
of off-heap space that we need and for yarn, we use exactly this amount for 
setting the `-XX:MaxDirectMemorySize` JVM property without letting room for 
other components and libraries. This worked so far for the network buffers when 
memory as a whole was set to off-/on-heap and the flink-reserved memory was not 
completely used. Now, however, if set to on-heap, the `-XX:MaxDirectMemorySize` 
is too sharp. I'm unsure about the solutions:
1) remove setting `-XX:MaxDirectMemorySize` and let the JVM adjust 
automatically, or
2) add some "sane" default to our off-heap usage?

The same may apply to Mesos if `ResourceProfile(cpuCores, heapMemoryInMB, 
directMemoryInMB, nativeMemoryInMB)` is used. At the moment, only the other 
constructors are used leading to solution 1.


> always use off-heap network buffers
> ---
>
> Key: FLINK-7316
> URL: https://issues.apache.org/jira/browse/FLINK-7316
> Project: Flink
>  Issue Type: Sub-task
>  Components: Core, Network
>Affects Versions: 1.4.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>
> In order to send flink buffers through netty into the network, we need to 
> make the buffers use off-heap memory. Otherwise, there will be a hidden copy 
> happening in the NIO stack.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4481: [FLINK-7316][network] always use off-heap network buffers

[NicoK]

Github user NicoK commented on the issue:

https://github.com/apache/flink/pull/4481
  
ok, one test fixed, the other is not so simple but maybe @tillrohrmann can 
help with it:

Inside `ContaineredTaskManagerParameters#create()`, we calculate the amount 
of off-heap space that we need and for yarn, we use exactly this amount for 
setting the `-XX:MaxDirectMemorySize` JVM property without letting room for 
other components and libraries. This worked so far for the network buffers when 
memory as a whole was set to off-/on-heap and the flink-reserved memory was not 
completely used. Now, however, if set to on-heap, the `-XX:MaxDirectMemorySize` 
is too sharp. I'm unsure about the solutions:
1) remove setting `-XX:MaxDirectMemorySize` and let the JVM adjust 
automatically, or
2) add some "sane" default to our off-heap usage?

The same may apply to Mesos if `ResourceProfile(cpuCores, heapMemoryInMB, 
directMemoryInMB, nativeMemoryInMB)` is used. At the moment, only the other 
constructors are used leading to solution 1.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6997) SavepointITCase fails in master branch sometimes

[Chesnay Schepler (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118510#comment-16118510
 ] 

Chesnay Schepler commented on FLINK-6997:
-

This was probably fixed in d7f9f9e3ef4c3906ac9816fb9d68bdfe3be1f08f.

> SavepointITCase fails in master branch sometimes
> 
>
> Key: FLINK-6997
> URL: https://issues.apache.org/jira/browse/FLINK-6997
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.3.0
>Reporter: Ted Yu
>  Labels: test-stability
>
> I got the following test failure (with commit 
> a0b781461bcf8c2f1d00b93464995f03eda589f1)
> {code}
> testSavepointForJobWithIteration(org.apache.flink.test.checkpointing.SavepointITCase)
>   Time elapsed: 8.129 sec  <<< ERROR!
> java.io.IOException: java.lang.Exception: Failed to complete savepoint
>   at 
> org.apache.flink.runtime.testingUtils.TestingCluster.triggerSavepoint(TestingCluster.scala:342)
>   at 
> org.apache.flink.runtime.testingUtils.TestingCluster.triggerSavepoint(TestingCluster.scala:316)
>   at 
> org.apache.flink.test.checkpointing.SavepointITCase.testSavepointForJobWithIteration(SavepointITCase.java:827)
> Caused by: java.lang.Exception: Failed to complete savepoint
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anon$7.apply(JobManager.scala:821)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anon$7.apply(JobManager.scala:805)
>   at 
> org.apache.flink.runtime.concurrent.impl.FlinkFuture$5.onComplete(FlinkFuture.java:272)
>   at akka.dispatch.OnComplete.internal(Future.scala:247)
>   at akka.dispatch.OnComplete.internal(Future.scala:245)
>   at akka.dispatch.japi$CallbackBridge.apply(Future.scala:175)
>   at akka.dispatch.japi$CallbackBridge.apply(Future.scala:172)
>   at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
>   at 
> akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
>   at 
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
>   at 
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
>   at 
> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
>   at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
>   at 
> akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
>   at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
>   at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>   at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>   at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: java.lang.Exception: Failed to trigger savepoint: Not all required 
> tasks are currently running.
>   at 
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.triggerSavepoint(CheckpointCoordinator.java:382)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:800)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
>   at 
> org.apache.flink.runtime.testingUtils.TestingJobManagerLike$$anonfun$handleTestingMessage$1.applyOrElse(TestingJobManagerLike.scala:95)
>   at scala.PartialFunction$OrElse.apply(PartialFunction.scala:162)
>   at 
> org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:38)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
>   at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
>   at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
>   at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7245) Enhance the operators to support holding back watermarks

[Fabian Hueske (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118503#comment-16118503
 ] 

Fabian Hueske commented on FLINK-7245:
--

Great that we are on the same page!

Let's see if we want to / need to touch the public DataStream API. This is 
always a bit delicate. 
We can also first implement a custom operator that we only use internally for 
the Table API. 

[~aljoscha] what are your thoughts about adding such an operator to the 
DataStream API or extending the {{ProcessFunction}} context and 
{{AbstractStreamOperator}}? The changes for a constant watermark delay should 
be moderate. But I'm not sure if we want this in the public API. What do you 
think?

> Enhance the operators to support holding back watermarks
> 
>
> Key: FLINK-7245
> URL: https://issues.apache.org/jira/browse/FLINK-7245
> Project: Flink
>  Issue Type: New Feature
>  Components: DataStream API
>Reporter: Xingcan Cui
>Assignee: Xingcan Cui
>
> Currently the watermarks are applied and emitted by the 
> {{AbstractStreamOperator}} instantly. 
> {code:java}
> public void processWatermark(Watermark mark) throws Exception {
>   if (timeServiceManager != null) {
>   timeServiceManager.advanceWatermark(mark);
>   }
>   output.emitWatermark(mark);
> }
> {code}
> Some calculation results (with timestamp fields) triggered by these 
> watermarks (e.g., join or aggregate results) may be regarded as delayed by 
> the downstream operators since their timestamps must be less than or equal to 
> the corresponding triggers. 
> This issue aims to add another "working mode", which supports holding back 
> watermarks, to current operators. These watermarks should be blocked and 
> stored by the operators until all the corresponding new generated results are 
> emitted.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7245) Enhance the operators to support holding back watermarks

[Xingcan Cui (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118491#comment-16118491
 ] 

Xingcan Cui commented on FLINK-7245:


Thanks for your responses, [~fhueske]. Actually, we were thinking about the 
same thing. A static delay may work for most of the scenarios. I'll add a 
{{setWatermarkDelay()}} method to the #Context classes of different 
ProcessFunctions and change the {{emit(mark)}} in 
{{AbstractStreamOperator#processWatermark()}} to 
{{emit(holdbackWatermark(mark))}}.

Best, Xingcan

> Enhance the operators to support holding back watermarks
> 
>
> Key: FLINK-7245
> URL: https://issues.apache.org/jira/browse/FLINK-7245
> Project: Flink
>  Issue Type: New Feature
>  Components: DataStream API
>Reporter: Xingcan Cui
>Assignee: Xingcan Cui
>
> Currently the watermarks are applied and emitted by the 
> {{AbstractStreamOperator}} instantly. 
> {code:java}
> public void processWatermark(Watermark mark) throws Exception {
>   if (timeServiceManager != null) {
>   timeServiceManager.advanceWatermark(mark);
>   }
>   output.emitWatermark(mark);
> }
> {code}
> Some calculation results (with timestamp fields) triggered by these 
> watermarks (e.g., join or aggregate results) may be regarded as delayed by 
> the downstream operators since their timestamps must be less than or equal to 
> the corresponding triggers. 
> This issue aims to add another "working mode", which supports holding back 
> watermarks, to current operators. These watermarks should be blocked and 
> stored by the operators until all the corresponding new generated results are 
> emitted.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (FLINK-7368) MetricStore makes cpu spin at 100%

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek updated FLINK-7368:

Fix Version/s: 1.3.3
   1.4.0

> MetricStore makes cpu spin at 100%
> --
>
> Key: FLINK-7368
> URL: https://issues.apache.org/jira/browse/FLINK-7368
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Reporter: Nico Chen
> Fix For: 1.4.0, 1.3.3
>
> Attachments: jm-jstack.log, MyHashMapInfiniteLoopTest.java, 
> MyHashMap.java
>
>
> Flink's `MetricStore` is not thread-safe. multi-treads may acess java' 
> hashmap inside `MetricStore` and can tirgger hashmap's infinte loop. 
> Recently I met the case that flink jobmanager consumed 100% cpu. A part of 
> stacktrace is shown below. The full jstack is in the attachment.
> {code:java}
> "ForkJoinPool-1-worker-19" daemon prio=10 tid=0x7fbdacac9800 nid=0x64c1 
> runnable [0x7fbd7d1c2000]
>java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:494)
> at 
> org.apache.flink.runtime.webmonitor.metrics.MetricStore.addMetric(MetricStore.java:176)
> at 
> org.apache.flink.runtime.webmonitor.metrics.MetricStore.add(MetricStore.java:121)
> at 
> org.apache.flink.runtime.webmonitor.metrics.MetricFetcher.addMetrics(MetricFetcher.java:198)
> at 
> org.apache.flink.runtime.webmonitor.metrics.MetricFetcher.access$500(MetricFetcher.java:58)
> at 
> org.apache.flink.runtime.webmonitor.metrics.MetricFetcher$4.onSuccess(MetricFetcher.java:188)
> at akka.dispatch.OnSuccess.internal(Future.scala:212)
> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:175)
> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:172)
> at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
> at 
> scala.runtime.AbstractPartialFunction.applyOrElse(AbstractPartialFunction.scala:28)
> at 
> scala.concurrent.Future$$anonfun$onSuccess$1.apply(Future.scala:117)
> at 
> scala.concurrent.Future$$anonfun$onSuccess$1.apply(Future.scala:115)
> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
> at 
> java.util.concurrent.ForkJoinTask$AdaptedRunnable.exec(ForkJoinTask.java:1265)
> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:334)
> at 
> java.util.concurrent.ForkJoinWorkerThread.execTask(ForkJoinWorkerThread.java:604)
> at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:784)
> at java.util.concurrent.ForkJoinPool.work(ForkJoinPool.java:646)
> at 
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:398)
> {code}
> There are 24 threads show same stacktrace as above to indicate they are 
> spining at HashMap.put(HashMap.java:494) (I am using Java 1.7.0_6). Many 
> posts indicate multi-threads accessing hashmap cause this problem and I 
> reproduce the case as well. The test code is attached. I only modify the 
> HashMap.transfer() by adding concurrent barriers for different treads in 
> order to simulate the timing of creation of cycles in hashmap's Entry.  My 
> program's stacktrace shows it hangs at same line of 
> HashMap(HashMap.put(HashMap.java:494)) as the stacktrace I post above.
>  Even through `MetricFetcher` has a 10 seconds minimum inteverl between each 
> metrics qurey, it still cannot guarntee query responses do not acess 
> `MtricStore`'s hashmap concurrently.  Thus I think it's a bug to fix.
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (FLINK-7368) MetricStore makes cpu spin at 100%

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek updated FLINK-7368:

Priority: Blocker  (was: Major)

> MetricStore makes cpu spin at 100%
> --
>
> Key: FLINK-7368
> URL: https://issues.apache.org/jira/browse/FLINK-7368
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Reporter: Nico Chen
>Priority: Blocker
> Fix For: 1.4.0, 1.3.3
>
> Attachments: jm-jstack.log, MyHashMapInfiniteLoopTest.java, 
> MyHashMap.java
>
>
> Flink's `MetricStore` is not thread-safe. multi-treads may acess java' 
> hashmap inside `MetricStore` and can tirgger hashmap's infinte loop. 
> Recently I met the case that flink jobmanager consumed 100% cpu. A part of 
> stacktrace is shown below. The full jstack is in the attachment.
> {code:java}
> "ForkJoinPool-1-worker-19" daemon prio=10 tid=0x7fbdacac9800 nid=0x64c1 
> runnable [0x7fbd7d1c2000]
>java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:494)
> at 
> org.apache.flink.runtime.webmonitor.metrics.MetricStore.addMetric(MetricStore.java:176)
> at 
> org.apache.flink.runtime.webmonitor.metrics.MetricStore.add(MetricStore.java:121)
> at 
> org.apache.flink.runtime.webmonitor.metrics.MetricFetcher.addMetrics(MetricFetcher.java:198)
> at 
> org.apache.flink.runtime.webmonitor.metrics.MetricFetcher.access$500(MetricFetcher.java:58)
> at 
> org.apache.flink.runtime.webmonitor.metrics.MetricFetcher$4.onSuccess(MetricFetcher.java:188)
> at akka.dispatch.OnSuccess.internal(Future.scala:212)
> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:175)
> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:172)
> at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
> at 
> scala.runtime.AbstractPartialFunction.applyOrElse(AbstractPartialFunction.scala:28)
> at 
> scala.concurrent.Future$$anonfun$onSuccess$1.apply(Future.scala:117)
> at 
> scala.concurrent.Future$$anonfun$onSuccess$1.apply(Future.scala:115)
> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
> at 
> java.util.concurrent.ForkJoinTask$AdaptedRunnable.exec(ForkJoinTask.java:1265)
> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:334)
> at 
> java.util.concurrent.ForkJoinWorkerThread.execTask(ForkJoinWorkerThread.java:604)
> at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:784)
> at java.util.concurrent.ForkJoinPool.work(ForkJoinPool.java:646)
> at 
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:398)
> {code}
> There are 24 threads show same stacktrace as above to indicate they are 
> spining at HashMap.put(HashMap.java:494) (I am using Java 1.7.0_6). Many 
> posts indicate multi-threads accessing hashmap cause this problem and I 
> reproduce the case as well. The test code is attached. I only modify the 
> HashMap.transfer() by adding concurrent barriers for different treads in 
> order to simulate the timing of creation of cycles in hashmap's Entry.  My 
> program's stacktrace shows it hangs at same line of 
> HashMap(HashMap.put(HashMap.java:494)) as the stacktrace I post above.
>  Even through `MetricFetcher` has a 10 seconds minimum inteverl between each 
> metrics qurey, it still cannot guarntee query responses do not acess 
> `MtricStore`'s hashmap concurrently.  Thus I think it's a bug to fix.
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7352) ExecutionGraphRestartTest timeouts

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118372#comment-16118372
 ] 

Till Rohrmann commented on FLINK-7352:
--

Another instance: https://travis-ci.org/apache/flink/jobs/262140336

> ExecutionGraphRestartTest timeouts
> --
>
> Key: FLINK-7352
> URL: https://issues.apache.org/jira/browse/FLINK-7352
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Coordination, Tests
>Affects Versions: 1.4.0, 1.3.2
>Reporter: Nico Kruber
>Priority: Critical
>  Labels: test-stability
>
> Recently, I received timeouts from some tests in 
> {{ExecutionGraphRestartTest}} like this
> {code}
> Tests in error: 
>   ExecutionGraphRestartTest.testConcurrentLocalFailAndRestart:638 » Timeout
> {code}
> This particular instance is from 1.3.2 RC2 and stuck in 
> {{ExecutionGraphTestUtils#waitUntilDeployedAndSwitchToRunning()}} but I also 
> had instances stuck in {{ExecutionGraphTestUtils#waitUntilJobStatus}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7054) remove LibraryCacheManager#getFile()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118369#comment-16118369
 ] 

ASF GitHub Bot commented on FLINK-7054:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4235


> remove LibraryCacheManager#getFile()
> 
>
> Key: FLINK-7054
> URL: https://issues.apache.org/jira/browse/FLINK-7054
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination, Network
>Affects Versions: 1.4.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
> Fix For: 1.4.0
>
>
> {{LibraryCacheManager#getFile()}} was only used in tests where it is 
> avoidable but if used anywhere else, it may have caused cleanup issues.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (FLINK-7054) remove LibraryCacheManager#getFile()

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann resolved FLINK-7054.
--
   Resolution: Fixed
Fix Version/s: 1.4.0

Fixed via 849990c00c1e7979b366088dbdba3f02a6123445

> remove LibraryCacheManager#getFile()
> 
>
> Key: FLINK-7054
> URL: https://issues.apache.org/jira/browse/FLINK-7054
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination, Network
>Affects Versions: 1.4.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
> Fix For: 1.4.0
>
>
> {{LibraryCacheManager#getFile()}} was only used in tests where it is 
> avoidable but if used anywhere else, it may have caused cleanup issues.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4235: [FLINK-7054] [blob] remove LibraryCacheManager#get...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4235


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7065) separate the flink-streaming-java module from flink-clients

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118354#comment-16118354
 ] 

ASF GitHub Bot commented on FLINK-7065:
---

Github user XuPingyong commented on the issue:

https://github.com/apache/flink/pull/4273
  
Thanks @tillrohrmann , I have updated this PR according to your advice.


> separate the flink-streaming-java module from flink-clients 
> 
>
> Key: FLINK-7065
> URL: https://issues.apache.org/jira/browse/FLINK-7065
> Project: Flink
>  Issue Type: Improvement
>  Components: DataStream API
>Reporter: Xu Pingyong
>Assignee: Xu Pingyong
>
> Motivation:
>  It is not good that "flink-streaming-java" module depends on 
> "flink-clients". Flink-clients should see something in "flink-streaming-java".
> Related Change:
>   1. LocalStreamEnvironment and RemoteStreamEnvironment can also execute 
> a job by the executors(LocalExecutor and RemoteExecutor).  Introduce 
> StreamGraphExecutor which executors a streamGraph as PlanExecutor executors 
> the plan.  StreamGraphExecutor and PlanExecutor all extend Executor.
>   2. Introduce  StreamExecutionEnvironmentFactory which works similarly 
> to ContextEnvironmentFactory in flink-clients.
>   When a object of ContextEnvironmentFactory, 
> OptimizerPlanEnvironmentFactory or PreviewPlanEnvironmentFactory is set into 
> ExecutionEnvironment(by calling initializeContextEnvironment)， the relevant 
> StreamEnvFactory is alsot set into StreamExecutionEnvironment. It is similar 
> when calling unsetContext.
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4273: [FLINK-7065] Separate the flink-streaming-java module fro...

[XuPingyong]

Github user XuPingyong commented on the issue:

https://github.com/apache/flink/pull/4273
  
Thanks @tillrohrmann , I have updated this PR according to your advice.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7065) separate the flink-streaming-java module from flink-clients

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118352#comment-16118352
 ] 

ASF GitHub Bot commented on FLINK-7065:
---

Github user XuPingyong commented on a diff in the pull request:

https://github.com/apache/flink/pull/4273#discussion_r131918248
  
--- Diff: 
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamGraphExecutor.java
 ---
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.environment;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.Executor;
+import org.apache.flink.api.common.ExecutorFactory;
+import org.apache.flink.api.common.JobExecutionResult;
+import org.apache.flink.streaming.api.graph.StreamGraph;
+
+/**
+ * A StreamGraphExecutor executes a StreamGraph.
+ *
+ * The specific implementation (such as the 
org.apache.flink.client.LocalExecutor
+ * and org.apache.flink.client.RemoteExecutor) is created by {@link 
ExecutorFactory}
+ *
+ */
+@Internal
+public interface StreamGraphExecutor extends Executor {
+
+   /**
+* Execute the given program.
+*
+* If the executor has not been started before, then this method 
will start the
+* executor and stop it after the execution has completed. This implies 
that one needs
+* to explicitly start the executor for all programs where multiple 
dataflow parts
+* depend on each other. Otherwise, the previous parts will no longer
+* be available, because the executor immediately shut down after the 
execution.
+*
+* @param streamGraph The streamGraph to execute.
+* @return The execution result, containing for example the net runtime 
of the program, and the accumulators.
+*
+* @throws Exception Thrown, if job submission caused an exception.
+*/
+   JobExecutionResult executeStreamGraph(StreamGraph streamGraph) throws 
Exception;
--- End diff --

Like PlanExecutor， Exception is still be throwed, which may be mixed with 
multi exception types.


> separate the flink-streaming-java module from flink-clients 
> 
>
> Key: FLINK-7065
> URL: https://issues.apache.org/jira/browse/FLINK-7065
> Project: Flink
>  Issue Type: Improvement
>  Components: DataStream API
>Reporter: Xu Pingyong
>Assignee: Xu Pingyong
>
> Motivation:
>  It is not good that "flink-streaming-java" module depends on 
> "flink-clients". Flink-clients should see something in "flink-streaming-java".
> Related Change:
>   1. LocalStreamEnvironment and RemoteStreamEnvironment can also execute 
> a job by the executors(LocalExecutor and RemoteExecutor).  Introduce 
> StreamGraphExecutor which executors a streamGraph as PlanExecutor executors 
> the plan.  StreamGraphExecutor and PlanExecutor all extend Executor.
>   2. Introduce  StreamExecutionEnvironmentFactory which works similarly 
> to ContextEnvironmentFactory in flink-clients.
>   When a object of ContextEnvironmentFactory, 
> OptimizerPlanEnvironmentFactory or PreviewPlanEnvironmentFactory is set into 
> ExecutionEnvironment(by calling initializeContextEnvironment)， the relevant 
> StreamEnvFactory is alsot set into StreamExecutionEnvironment. It is similar 
> when calling unsetContext.
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4273: [FLINK-7065] Separate the flink-streaming-java mod...

[XuPingyong]

Github user XuPingyong commented on a diff in the pull request:

https://github.com/apache/flink/pull/4273#discussion_r131918248
  
--- Diff: 
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamGraphExecutor.java
 ---
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.environment;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.Executor;
+import org.apache.flink.api.common.ExecutorFactory;
+import org.apache.flink.api.common.JobExecutionResult;
+import org.apache.flink.streaming.api.graph.StreamGraph;
+
+/**
+ * A StreamGraphExecutor executes a StreamGraph.
+ *
+ * The specific implementation (such as the 
org.apache.flink.client.LocalExecutor
+ * and org.apache.flink.client.RemoteExecutor) is created by {@link 
ExecutorFactory}
+ *
+ */
+@Internal
+public interface StreamGraphExecutor extends Executor {
+
+   /**
+* Execute the given program.
+*
+* If the executor has not been started before, then this method 
will start the
+* executor and stop it after the execution has completed. This implies 
that one needs
+* to explicitly start the executor for all programs where multiple 
dataflow parts
+* depend on each other. Otherwise, the previous parts will no longer
+* be available, because the executor immediately shut down after the 
execution.
+*
+* @param streamGraph The streamGraph to execute.
+* @return The execution result, containing for example the net runtime 
of the program, and the accumulators.
+*
+* @throws Exception Thrown, if job submission caused an exception.
+*/
+   JobExecutionResult executeStreamGraph(StreamGraph streamGraph) throws 
Exception;
--- End diff --

Like PlanExecutor， Exception is still be throwed, which may be mixed with 
multi exception types.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7065) separate the flink-streaming-java module from flink-clients

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118348#comment-16118348
 ] 

ASF GitHub Bot commented on FLINK-7065:
---

Github user XuPingyong commented on a diff in the pull request:

https://github.com/apache/flink/pull/4273#discussion_r131916494
  
--- Diff: flink-contrib/flink-connector-wikiedits/pom.xml ---
@@ -37,7 +37,7 @@ under the License.


org.apache.flink
-   
flink-streaming-java_${scala.binary.version}
+   
flink-clients_${scala.binary.version}
--- End diff --

It is executed in WikipediaEditsSourceTest. In the latest commit, this 
module still depends on flink-streaming-java, and also flink-clients of test 
scope.


> separate the flink-streaming-java module from flink-clients 
> 
>
> Key: FLINK-7065
> URL: https://issues.apache.org/jira/browse/FLINK-7065
> Project: Flink
>  Issue Type: Improvement
>  Components: DataStream API
>Reporter: Xu Pingyong
>Assignee: Xu Pingyong
>
> Motivation:
>  It is not good that "flink-streaming-java" module depends on 
> "flink-clients". Flink-clients should see something in "flink-streaming-java".
> Related Change:
>   1. LocalStreamEnvironment and RemoteStreamEnvironment can also execute 
> a job by the executors(LocalExecutor and RemoteExecutor).  Introduce 
> StreamGraphExecutor which executors a streamGraph as PlanExecutor executors 
> the plan.  StreamGraphExecutor and PlanExecutor all extend Executor.
>   2. Introduce  StreamExecutionEnvironmentFactory which works similarly 
> to ContextEnvironmentFactory in flink-clients.
>   When a object of ContextEnvironmentFactory, 
> OptimizerPlanEnvironmentFactory or PreviewPlanEnvironmentFactory is set into 
> ExecutionEnvironment(by calling initializeContextEnvironment)， the relevant 
> StreamEnvFactory is alsot set into StreamExecutionEnvironment. It is similar 
> when calling unsetContext.
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4273: [FLINK-7065] Separate the flink-streaming-java mod...

[XuPingyong]

Github user XuPingyong commented on a diff in the pull request:

https://github.com/apache/flink/pull/4273#discussion_r131916494
  
--- Diff: flink-contrib/flink-connector-wikiedits/pom.xml ---
@@ -37,7 +37,7 @@ under the License.


org.apache.flink
-   
flink-streaming-java_${scala.binary.version}
+   
flink-clients_${scala.binary.version}
--- End diff --

It is executed in WikipediaEditsSourceTest. In the latest commit, this 
module still depends on flink-streaming-java, and also flink-clients of test 
scope.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7387) Let RpcEndpoint directly implement RpcGateways

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118345#comment-16118345
 ] 

ASF GitHub Bot commented on FLINK-7387:
---

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/4498

[FLINK-7387] [rpc] Require RpcEndpoints to directly implement RpcGateways

## What is the purpose of the change

This commit changes the relation between `RpcEndpoints` and `RpcGateways`. 
From now on,
the `RpcEndpoints` have to implement the `RpcGateways` they want to support 
instead of
coupling it loosely via a type parameter. In order to obtain self gateway a 
new
method `RpcEndpoint#getSelfGateway(Class)` has been introduced. This method 
can be used
to obtain the `RpcGateway` type at run time to talk to the `RpcEndpoint` 
asynchronously.

All existing `RpcEndpoints` have been adapted to the new model. This 
basically means
that they now return a CompletableFuture instead of X.

## Brief change log

- Introduce `RpcServer` to encapsulate functionality previously contained 
in `RpcEndpoint's` self gateway
- Adapt `RpcService` to start `RpcServer` with correct type bounds
- Let `RpcEndpoints` implement their respective `RpcGateways`
- Remove `RpcCompletenessTest` and `RpcMethod`

## Verifying this change

This change is already covered by existing tests:

- `AsyncCallsTest`, `AkkaRpcServiceTest`, `AkkaRpcActorTest`

This change added tests and can be verified as follows:

- Added `RpcEndpointTest` to ensure that extending `RpcEndpoints` works

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes (but only Flip-6))

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink rpcRework

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4498.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4498


commit 858b27b96bf40377dba73a11bcd31d488b9a68cc
Author: Till Rohrmann 
Date:   2017-08-08T12:43:47Z

[FLINK-7387] [rpc] Require RpcEndpoints to directly implement RpcGateways

This commit changes the relation between RpcEndpoints and RpcGateways. From 
now on,
the RpcEndpoints have to implement the RpcGateways they want to support 
instead of
coupling it loosely via a type parameter. In order to obtain self gateway a 
new
method RpcEndpoint#getSelfGateway(Class) has been introduced. This method 
can be used
to obtain the RpcGateway type at run time to talk to the RpcEndpoint 
asynchronously.

All existing RpcEndpoints have been adapted to the new model. This 
basically means
that they now return a CompletableFuture instead of X.

Add RpcEndpointTest




> Let RpcEndpoint directly implement RpcGateways
> --
>
> Key: FLINK-7387
> URL: https://issues.apache.org/jira/browse/FLINK-7387
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination
>Affects Versions: 1.4.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>  Labels: flip-6
>
> Right now, a {{RpcGateway}} is only implicitly coupled to a {{RpcEndpoint}} 
> by specifying it a s type argument and having the {{RpcCompletenessTest}} 
> watching that the endpoint implements all methods of the gateway.
> This makes it possible that the endpoint and the gateway can have slightly 
> different interfaces. E.g. the gateway defines a method which returns a 
> {{Future}} whereas this method is implemented by the endpoint as a 
> method which returns {{Integer}}. This underlines the fact that the endpoint 
> method call is synchronous whereas the gateway issues an asynchronous call.
> The downside is that it is very cumbersome to establish type hierarchies 
> among {{RpcEndpoints}}. Thus, it is not easily possible to extend a specific 
> {{RcpEndpoint}} to add testing RPCs, for example. Moreover, you rely on the 
> {{RpcCompletenessTest}} to catch all not implemented methods instead of 
> relying 

[GitHub] flink pull request #4498: [FLINK-7387] [rpc] Require RpcEndpoints to directl...

[tillrohrmann]

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/4498

[FLINK-7387] [rpc] Require RpcEndpoints to directly implement RpcGateways

## What is the purpose of the change

This commit changes the relation between `RpcEndpoints` and `RpcGateways`. 
From now on,
the `RpcEndpoints` have to implement the `RpcGateways` they want to support 
instead of
coupling it loosely via a type parameter. In order to obtain self gateway a 
new
method `RpcEndpoint#getSelfGateway(Class)` has been introduced. This method 
can be used
to obtain the `RpcGateway` type at run time to talk to the `RpcEndpoint` 
asynchronously.

All existing `RpcEndpoints` have been adapted to the new model. This 
basically means
that they now return a CompletableFuture instead of X.

## Brief change log

- Introduce `RpcServer` to encapsulate functionality previously contained 
in `RpcEndpoint's` self gateway
- Adapt `RpcService` to start `RpcServer` with correct type bounds
- Let `RpcEndpoints` implement their respective `RpcGateways`
- Remove `RpcCompletenessTest` and `RpcMethod`

## Verifying this change

This change is already covered by existing tests:

- `AsyncCallsTest`, `AkkaRpcServiceTest`, `AkkaRpcActorTest`

This change added tests and can be verified as follows:

- Added `RpcEndpointTest` to ensure that extending `RpcEndpoints` works

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes (but only Flip-6))

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink rpcRework

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4498.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4498


commit 858b27b96bf40377dba73a11bcd31d488b9a68cc
Author: Till Rohrmann 
Date:   2017-08-08T12:43:47Z

[FLINK-7387] [rpc] Require RpcEndpoints to directly implement RpcGateways

This commit changes the relation between RpcEndpoints and RpcGateways. From 
now on,
the RpcEndpoints have to implement the RpcGateways they want to support 
instead of
coupling it loosely via a type parameter. In order to obtain self gateway a 
new
method RpcEndpoint#getSelfGateway(Class) has been introduced. This method 
can be used
to obtain the RpcGateway type at run time to talk to the RpcEndpoint 
asynchronously.

All existing RpcEndpoints have been adapted to the new model. This 
basically means
that they now return a CompletableFuture instead of X.

Add RpcEndpointTest




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (FLINK-7390) Source download links broken for 1.2.0 to 1.3.1

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-7390:

Summary: Source download links broken for 1.2.0 to 1.3.1  (was: Download 
links broken for 1.2.0 to 1.3.1)

> Source download links broken for 1.2.0 to 1.3.1
> ---
>
> Key: FLINK-7390
> URL: https://issues.apache.org/jira/browse/FLINK-7390
> Project: Flink
>  Issue Type: Bug
>  Components: Project Website
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7391) Normalize release entries

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-7391:
---

 Summary: Normalize release entries
 Key: FLINK-7391
 URL: https://issues.apache.org/jira/browse/FLINK-7391
 Project: Flink
  Issue Type: Improvement
  Components: Project Website
Reporter: Chesnay Schepler


The release list at http://flink.apache.org/downloads.html is inconsistent in 
regards to the java/scala docs links. For 1.1.3 and below we only include a 
docs link for the latest version (i.e 1.1.3, but not for 1.1.2), for higher 
versions we have a docs link for every release.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7337) Refactor handling of time indicator attributes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118336#comment-16118336
 ] 

ASF GitHub Bot commented on FLINK-7337:
---

Github user wuchong commented on the issue:

https://github.com/apache/flink/pull/4488
  
@twalthr  I'm working on other issues before Thursday, so I would like to 
have a look at it at Thursday (Beijing). But if you are hurry, I'm fine to 
merge this first.


> Refactor handling of time indicator attributes
> --
>
> Key: FLINK-7337
> URL: https://issues.apache.org/jira/browse/FLINK-7337
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Affects Versions: 1.4.0
>Reporter: Fabian Hueske
>Assignee: Fabian Hueske
>
> After a [discussion on the dev mailing 
> list|https://lists.apache.org/thread.html/735d55f9022df8ff73566a9f1553e14be94f8443986ad46559b35869@%3Cdev.flink.apache.org%3E]
>  I propose the following changes to the current handling of time indicator 
> attributes:
> * Remove the separation of logical and physical row type.
> ** Hold the event-time timestamp as regular Long field in Row
> ** Represent the processing-time indicator type as a null-valued field in Row 
> (1 bit overhead)
> * Remove materialization of event-time timestamps because timestamp is 
> already accessible in Row.
> * Add {{ProcessFunction}} to set timestamp into the timestamp field of a 
> {{StreamRecord}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4488: [FLINK-7337] [table] Refactor internal handling of time i...

[wuchong]

Github user wuchong commented on the issue:

https://github.com/apache/flink/pull/4488
  
@twalthr  I'm working on other issues before Thursday, so I would like to 
have a look at it at Thursday (Beijing). But if you are hurry, I'm fine to 
merge this first.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-7390) Download links for broken for 1.2.0 to 1.3.1

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-7390:
---

 Summary: Download links for broken for 1.2.0 to 1.3.1
 Key: FLINK-7390
 URL: https://issues.apache.org/jira/browse/FLINK-7390
 Project: Flink
  Issue Type: Bug
  Components: Project Website
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (FLINK-7390) Download links broken for 1.2.0 to 1.3.1

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-7390:

Summary: Download links broken for 1.2.0 to 1.3.1  (was: Download links for 
broken for 1.2.0 to 1.3.1)

> Download links broken for 1.2.0 to 1.3.1
> 
>
> Key: FLINK-7390
> URL: https://issues.apache.org/jira/browse/FLINK-7390
> Project: Flink
>  Issue Type: Bug
>  Components: Project Website
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7169) Support AFTER MATCH SKIP function in CEP library API

[Dawid Wysakowicz (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118321#comment-16118321
 ] 

Dawid Wysakowicz commented on FLINK-7169:
-

Hi [~ychen], Sorry for the delay. I will definitely review it this week. 

> Support AFTER MATCH SKIP function in CEP library API
> 
>
> Key: FLINK-7169
> URL: https://issues.apache.org/jira/browse/FLINK-7169
> Project: Flink
>  Issue Type: Sub-task
>  Components: CEP
>Reporter: Yueting Chen
>Assignee: Yueting Chen
> Fix For: 1.4.0
>
>
> In order to support Oracle's MATCH_RECOGNIZE on top of the CEP library, we 
> need to support AFTER MATCH SKIP function in CEP API.
> There're four options in AFTER MATCH SKIP, listed as follows:
> 1. AFTER MATCH SKIP TO NEXT ROW: resume pattern matching at the row after the 
> first row of the current match.
> 2. AFTER MATCH SKIP PAST LAST ROW: resume pattern matching at the next row 
> after the last row of the current match.
> 3. AFTER MATCH SKIP TO FIST *RPV*: resume pattern matching at the first row 
> that is mapped to the row pattern variable RPV.
> 4. AFTER MATCH SKIP TO LAST *RPV*: resume pattern matching at the last row 
> that is mapped to the row pattern variable RPV.
> I think we can introduce a new function to `CEP` class, which takes a new 
> parameter as AfterMatchSKipStrategy.
> The new API may looks like this
> {code}
> public static  PatternStream pattern(DataStream input, Pattern 
> pattern, AfterMatchSkipStrategy afterMatchSkipStrategy) 
> {code}
> We can also make `SKIP TO NEXT ROW` as the default option, because that's 
> what CEP library behaves currently.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4454: [hotfix][docs] Add section in docs about writing u...

[dawidwys]

Github user dawidwys commented on a diff in the pull request:

https://github.com/apache/flink/pull/4454#discussion_r131906203
  
--- Diff: docs/dev/testing.md ---
@@ -0,0 +1,189 @@
+---
+title: "Testing"
+nav-parent_id: dev
+nav-id: testing
+nav-pos: 99
+---
+
+
+This page briefly discusses how to test Flink application in the local 
environment.
+
+* This will be replaced by the TOC
+{:toc}
+
+## Unit testing
+
+It is encouraged to test your classes with unit tests as much as possible. 
For example if one implement following `ReduceFunction`:
+
+~~~java
+public class SumReduce implements ReduceFunction {
+@Override
+public Long reduce(Long value1, Long value2) throws Exception {
+return value1 + value2;
+}
+}
+~~~
+
+it is very easy to unit test it with your favorite framework:
+
+~~~java
+public class SumReduceTest {
+@Test
+public void testSum() throws Exception {
+SumReduce sumReduce = new SumReduce();
+
+assertEquals(42L, sumReduce.reduce(40L, 2L));
+}
+}
+~~~
+
+Or in scala:
+
+~~~scala
+class SumReduce extends ReduceFunction[Long] {
+override def reduce(value1: java.lang.Long,
+value2: java.lang.Long): java.lang.Long = value1 + 
value2
+}
+~~~
+
+~~~scala
+class SumReduceTest extends FlatSpec with Matchers {
+"SumReduce" should "add values" in {
+val sumReduce: SumReduce = new SumReduce()
+sumReduce.reduce(40L, 2L) should be (42L)
+}
+}
+~~~
+
+## Integration testing
+
+You also can write integration tests that are executed against local Flink 
mini cluster.
+In order to do so add a test dependency `flink-test-utils`.
+
+~~~ xml
+
+  org.apache.flink
+  flink-test-utils{{site.scala_version_suffix}}
+  {{site.version}}
+
+~~~
--- End diff --

Just a short explanation. In other places e.g. in `dev/connectors/kafka.md` 
we use other highlighting extension(liquid) and a code like:

{% highlight xml %}

  org.apache.flink
  flink-test-utils{{ site.scala_version_suffix 
}}
  {{site.version }}

{% endhighlight %}

renders into:


  org.apache.flink
  flink-test-utils_2.10
  1.4-SNAPSHOT


`{{site.version }}` and `{{ site.scala_version_suffix }}` are variables 
based on properties of the build. It seems they do not work with the syntax 
used in this site(that uses redcarpet highlighting).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7365) excessive warning logs of attempt to override final parameter: fs.s3.buffer.dir

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118314#comment-16118314
 ] 

Aljoscha Krettek commented on FLINK-7365:
-

This can, for example, be achieved by editing the {{log4j.properties}} file (or 
the respective configuration file of the logging system). For example:
{code}
log4j.logger.org.apache.hadoop.conf.Configuration=ERROR, file
{code}

I don't think it would be good to do this by default, though, since users might 
have failing jobs and they never see the log message than could help them 
figure out why something is failing.

> excessive warning logs of attempt to override final parameter: 
> fs.s3.buffer.dir
> ---
>
> Key: FLINK-7365
> URL: https://issues.apache.org/jira/browse/FLINK-7365
> Project: Flink
>  Issue Type: Bug
>  Components: Configuration
>Affects Versions: 1.3.0
>Reporter: Bowen Li
>
> I'm seeing hundreds of line of the following log in my JobManager log file:
> {code:java}
> 2017-08-03 19:48:45,330 WARN  org.apache.hadoop.conf.Configuration
>   - /usr/lib/hadoop/etc/hadoop/core-site.xml:an attempt to 
> override final parameter: fs.s3.buffer.dir;  Ignoring.
> 2017-08-03 19:48:45,485 WARN  org.apache.hadoop.conf.Configuration
>   - /etc/hadoop/conf/core-site.xml:an attempt to override final 
> parameter: fs.s3.buffer.dir;  Ignoring.
> 2017-08-03 19:48:45,486 WARN  org.apache.hadoop.conf.Configuration
>   - /usr/lib/hadoop/etc/hadoop/core-site.xml:an attempt to 
> override final parameter: fs.s3.buffer.dir;  Ignoring.
> 2017-08-03 19:48:45,626 WARN  org.apache.hadoop.conf.Configuration
>   - /etc/hadoop/conf/core-site.xml:an attempt to override final 
> parameter: fs.s3.buffer.dir;  Ignoring
> ..
> {code}
> Info of my Flink cluster:
> - Running on EMR with emr-5.6.0
> - Using FSStateBackend, writing checkpointing data files to s3
> - Configured s3 with S3AFileSystem according to 
> https://ci.apache.org/projects/flink/flink-docs-release-1.4/setup/aws.html#set-s3-filesystem
> - AWS forbids resetting 'fs.s3.buffer.dir' value (it has a  tag on 
> this property in core-site.xml), so I set 'fs.s3a.buffer.dir' as '/tmp'
> Here's my core-site.xml file:
> {code:java}
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>   fs.s3.buffer.dir
>   /mnt/s3,/mnt1/s3
>   true
> 
> 
>   fs.s3.impl
>   org.apache.hadoop.fs.s3a.S3AFileSystem
> 
> 
>   fs.s3n.impl
>   com.amazon.ws.emr.hadoop.fs.EmrFileSystem
> 
>   
> ipc.client.connect.max.retries.on.timeouts
> 5
>   
>   
> hadoop.security.key.default.bitlength
> 256
>   
>   
> hadoop.proxyuser.hadoop.groups
> *
>   
>   
> hadoop.tmp.dir
> /mnt/var/lib/hadoop/tmp
>   
>   
> hadoop.proxyuser.hadoop.hosts
> *
>   
>   
> io.file.buffer.size
> 65536
>   
>   
> fs.AbstractFileSystem.s3.impl
> org.apache.hadoop.fs.s3.EMRFSDelegate
>   
>   
> fs.s3a.buffer.dir
> /tmp
>   
>   
> fs.s3bfs.impl
> org.apache.hadoop.fs.s3.S3FileSystem
>   
> 
> {code}
> This bug is about excessive logging.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7169) Support AFTER MATCH SKIP function in CEP library API

[Yueting Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118297#comment-16118297
 ] 

Yueting Chen commented on FLINK-7169:
-

[~dawidwys] [~dian.fu]
I hava updated the PR serveral days ago, could you please help to review that? 
Thanks.

> Support AFTER MATCH SKIP function in CEP library API
> 
>
> Key: FLINK-7169
> URL: https://issues.apache.org/jira/browse/FLINK-7169
> Project: Flink
>  Issue Type: Sub-task
>  Components: CEP
>Reporter: Yueting Chen
>Assignee: Yueting Chen
> Fix For: 1.4.0
>
>
> In order to support Oracle's MATCH_RECOGNIZE on top of the CEP library, we 
> need to support AFTER MATCH SKIP function in CEP API.
> There're four options in AFTER MATCH SKIP, listed as follows:
> 1. AFTER MATCH SKIP TO NEXT ROW: resume pattern matching at the row after the 
> first row of the current match.
> 2. AFTER MATCH SKIP PAST LAST ROW: resume pattern matching at the next row 
> after the last row of the current match.
> 3. AFTER MATCH SKIP TO FIST *RPV*: resume pattern matching at the first row 
> that is mapped to the row pattern variable RPV.
> 4. AFTER MATCH SKIP TO LAST *RPV*: resume pattern matching at the last row 
> that is mapped to the row pattern variable RPV.
> I think we can introduce a new function to `CEP` class, which takes a new 
> parameter as AfterMatchSKipStrategy.
> The new API may looks like this
> {code}
> public static  PatternStream pattern(DataStream input, Pattern 
> pattern, AfterMatchSkipStrategy afterMatchSkipStrategy) 
> {code}
> We can also make `SKIP TO NEXT ROW` as the default option, because that's 
> what CEP library behaves currently.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4454: [hotfix][docs] Add section in docs about writing unit/int...

[pnowojski]

Github user pnowojski commented on the issue:

https://github.com/apache/flink/pull/4454
  
Maybe we can merge this as it is and we can create a follow up ticket for 
figuring out what should be mentioned in watermarks testing section? To be 
honest I don't feel good enough qualified to describe that part :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4454: [hotfix][docs] Add section in docs about writing u...

[pnowojski]

Github user pnowojski commented on a diff in the pull request:

https://github.com/apache/flink/pull/4454#discussion_r131900306
  
--- Diff: docs/dev/testing.md ---
@@ -0,0 +1,189 @@
+---
+title: "Testing"
+nav-parent_id: dev
+nav-id: testing
+nav-pos: 99
+---
+
+
+This page briefly discusses how to test Flink application in the local 
environment.
+
+* This will be replaced by the TOC
+{:toc}
+
+## Unit testing
+
+It is encouraged to test your classes with unit tests as much as possible. 
For example if one implement following `ReduceFunction`:
+
+~~~java
+public class SumReduce implements ReduceFunction {
+@Override
+public Long reduce(Long value1, Long value2) throws Exception {
+return value1 + value2;
+}
+}
+~~~
+
+it is very easy to unit test it with your favorite framework:
+
+~~~java
+public class SumReduceTest {
+@Test
+public void testSum() throws Exception {
+SumReduce sumReduce = new SumReduce();
+
+assertEquals(42L, sumReduce.reduce(40L, 2L));
+}
+}
+~~~
+
+Or in scala:
+
+~~~scala
+class SumReduce extends ReduceFunction[Long] {
+override def reduce(value1: java.lang.Long,
+value2: java.lang.Long): java.lang.Long = value1 + 
value2
+}
+~~~
+
+~~~scala
+class SumReduceTest extends FlatSpec with Matchers {
+"SumReduce" should "add values" in {
+val sumReduce: SumReduce = new SumReduce()
+sumReduce.reduce(40L, 2L) should be (42L)
+}
+}
+~~~
+
+## Integration testing
+
+You also can write integration tests that are executed against local Flink 
mini cluster.
+In order to do so add a test dependency `flink-test-utils`.
+
+~~~ xml
+
+  org.apache.flink
+  flink-test-utils{{site.scala_version_suffix}}
+  {{site.version}}
+
+~~~
--- End diff --

Updated but it properties names depends on the user's `pom.xml` file.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (FLINK-7382) Broken links in `Apache Flink Documentation` page

[Timo Walther (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timo Walther resolved FLINK-7382.
-
   Resolution: Fixed
Fix Version/s: 1.3.3
   1.4.0

Fixed in 1.4: 8c7b3d9af5ffe6066835e02fb2ae39fc51f99856
Fixed in 1.3: 5578ecd43006d32aa09762372058b906d44c05a4

> Broken links in `Apache Flink Documentation`  page
> --
>
> Key: FLINK-7382
> URL: https://issues.apache.org/jira/browse/FLINK-7382
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Hai Zhou
>Priority: Minor
> Fix For: 1.4.0, 1.3.3
>
>
> Some links in the * External Resources * section are Broken.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7382) Broken links in `Apache Flink Documentation` page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118266#comment-16118266
 ] 

ASF GitHub Bot commented on FLINK-7382:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4490


> Broken links in `Apache Flink Documentation`  page
> --
>
> Key: FLINK-7382
> URL: https://issues.apache.org/jira/browse/FLINK-7382
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Hai Zhou
>Priority: Minor
>
> Some links in the * External Resources * section are Broken.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4490: [FLINK-7382][docs] Fix some broken links in docs

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4490


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7382) Broken links in `Apache Flink Documentation` page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118258#comment-16118258
 ] 

ASF GitHub Bot commented on FLINK-7382:
---

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/4490
  
Merging...


> Broken links in `Apache Flink Documentation`  page
> --
>
> Key: FLINK-7382
> URL: https://issues.apache.org/jira/browse/FLINK-7382
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Hai Zhou
>Priority: Minor
>
> Some links in the * External Resources * section are Broken.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4490: [FLINK-7382][docs] Fix some broken links in docs

[twalthr]

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/4490
  
Merging...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (FLINK-7301) Rework state documentation

[Timo Walther (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timo Walther resolved FLINK-7301.
-
   Resolution: Fixed
Fix Version/s: 1.4.0

Fixed in 1.4.0: 470706740d5d89de69844a5662166ce94d71d00d

> Rework state documentation
> --
>
> Key: FLINK-7301
> URL: https://issues.apache.org/jira/browse/FLINK-7301
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Timo Walther
>Assignee: Timo Walther
> Fix For: 1.4.0
>
>
> The documentation about state is spread across different pages, but this is 
> not consistent and it is hard to find what you need. I propose:
> "Mention State Backends and link to them in ""Streaming/Working with State"".
> Create category ""State & Fault Tolerance"" under ""Streaming"". Move 
> ""Working with State"", ""Checkpointing"" and ""Queryable State"".
> Move API related parts (90%) of ""Deployment/State & Fault Tolerance/State 
> Backends"" to ""Streaming/State & Fault Tolerance/State Backends"".
> Move all tuning things from ""Debugging/Large State"" to ""Deployment/State & 
> Fault Tolerance/State Backends"".
> Move ""Streaming/Working with State/Custom Serialization for Managed State"" 
> to ""Streaming/State & Fault Tolerance/Custom Serialization"" (Add a link 
> from previous position, also link from ""Data Types & Serialization"")."



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7301) Rework state documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118254#comment-16118254
 ] 

ASF GitHub Bot commented on FLINK-7301:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4441


> Rework state documentation
> --
>
> Key: FLINK-7301
> URL: https://issues.apache.org/jira/browse/FLINK-7301
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> The documentation about state is spread across different pages, but this is 
> not consistent and it is hard to find what you need. I propose:
> "Mention State Backends and link to them in ""Streaming/Working with State"".
> Create category ""State & Fault Tolerance"" under ""Streaming"". Move 
> ""Working with State"", ""Checkpointing"" and ""Queryable State"".
> Move API related parts (90%) of ""Deployment/State & Fault Tolerance/State 
> Backends"" to ""Streaming/State & Fault Tolerance/State Backends"".
> Move all tuning things from ""Debugging/Large State"" to ""Deployment/State & 
> Fault Tolerance/State Backends"".
> Move ""Streaming/Working with State/Custom Serialization for Managed State"" 
> to ""Streaming/State & Fault Tolerance/Custom Serialization"" (Add a link 
> from previous position, also link from ""Data Types & Serialization"")."



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4441: [FLINK-7301] [docs] Rework state documentation

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4441


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7301) Rework state documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118251#comment-16118251
 ] 

ASF GitHub Bot commented on FLINK-7301:
---

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/4441
  
Thanks @alpinegizmo. I will merge this now.


> Rework state documentation
> --
>
> Key: FLINK-7301
> URL: https://issues.apache.org/jira/browse/FLINK-7301
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> The documentation about state is spread across different pages, but this is 
> not consistent and it is hard to find what you need. I propose:
> "Mention State Backends and link to them in ""Streaming/Working with State"".
> Create category ""State & Fault Tolerance"" under ""Streaming"". Move 
> ""Working with State"", ""Checkpointing"" and ""Queryable State"".
> Move API related parts (90%) of ""Deployment/State & Fault Tolerance/State 
> Backends"" to ""Streaming/State & Fault Tolerance/State Backends"".
> Move all tuning things from ""Debugging/Large State"" to ""Deployment/State & 
> Fault Tolerance/State Backends"".
> Move ""Streaming/Working with State/Custom Serialization for Managed State"" 
> to ""Streaming/State & Fault Tolerance/Custom Serialization"" (Add a link 
> from previous position, also link from ""Data Types & Serialization"")."



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4441: [FLINK-7301] [docs] Rework state documentation

[twalthr]

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/4441
  
Thanks @alpinegizmo. I will merge this now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (FLINK-7300) End-to-end tests are instable on Travis

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed FLINK-7300.
---
Resolution: Fixed

Fixed in
65402e034c32e47824fe46427d83eb9c9ea22d30
2ed74ca060ba64fefc7b53a23640d4854329f418

> End-to-end tests are instable on Travis
> ---
>
> Key: FLINK-7300
> URL: https://issues.apache.org/jira/browse/FLINK-7300
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.4.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Aljoscha Krettek
>  Labels: test-stability
> Fix For: 1.4.0
>
>
> It seems like the end-to-end tests are instable, causing the {{misc}} build 
> profile to sporadically fail.
> Incorrect matched output:
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/258569408/log.txt?X-Amz-Expires=30=20170731T060526Z=AWS4-HMAC-SHA256=AKIAJRYRXRSVGNKPKO5A/20170731/us-east-1/s3/aws4_request=host=4ef9ff5e60fe06db53a84be8d73775a46cb595a8caeb806b05dbbf824d3b69e8
> Another failure example of a different cause then the above, also on the 
> end-to-end tests:
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/258841693/log.txt?X-Amz-Expires=30=20170731T060007Z=AWS4-HMAC-SHA256=AKIAJRYRXRSVGNKPKO5A/20170731/us-east-1/s3/aws4_request=host=4a106b3990228b7628c250cc15407bc2c131c8332e1a94ad68d649fe8d32d726



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7337) Refactor handling of time indicator attributes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118222#comment-16118222
 ] 

ASF GitHub Bot commented on FLINK-7337:
---

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/4488
  
@wuchong @sunjincheng121 @shaoxuan-wang do you also want to take a look at 
it? Otherwise I would merge this and work on the followup issue for more 
efficiency.


> Refactor handling of time indicator attributes
> --
>
> Key: FLINK-7337
> URL: https://issues.apache.org/jira/browse/FLINK-7337
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Affects Versions: 1.4.0
>Reporter: Fabian Hueske
>Assignee: Fabian Hueske
>
> After a [discussion on the dev mailing 
> list|https://lists.apache.org/thread.html/735d55f9022df8ff73566a9f1553e14be94f8443986ad46559b35869@%3Cdev.flink.apache.org%3E]
>  I propose the following changes to the current handling of time indicator 
> attributes:
> * Remove the separation of logical and physical row type.
> ** Hold the event-time timestamp as regular Long field in Row
> ** Represent the processing-time indicator type as a null-valued field in Row 
> (1 bit overhead)
> * Remove materialization of event-time timestamps because timestamp is 
> already accessible in Row.
> * Add {{ProcessFunction}} to set timestamp into the timestamp field of a 
> {{StreamRecord}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4488: [FLINK-7337] [table] Refactor internal handling of time i...

[twalthr]

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/4488
  
@wuchong @sunjincheng121 @shaoxuan-wang do you also want to take a look at 
it? Otherwise I would merge this and work on the followup issue for more 
efficiency.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (FLINK-7385) Fix ArrayIndexOutOfBoundsException when object-reuse is enabled

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai resolved FLINK-7385.

Resolution: Fixed

Thanks for the contribution!

Merged to master via 6f5fa7f741538207244368c275bee9958c43a25a.
Merged to release-1.3 via c1f578fba60be7b77e1588367721f57b52b61225.

> Fix ArrayIndexOutOfBoundsException when object-reuse is enabled  
> -
>
> Key: FLINK-7385
> URL: https://issues.apache.org/jira/browse/FLINK-7385
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.1.5, 1.2.1, 1.3.2
>Reporter: Ruidong Li
>Assignee: Ruidong Li
>Priority: Blocker
> Fix For: 1.4.0, 1.3.3
>
>
> In OperatorChain.java, there is a potential ArrayIndexOutOfBoundsException 
> when object-reuse is enabled



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7385) Fix ArrayIndexOutOfBoundsException when object-reuse is enabled

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118207#comment-16118207
 ] 

ASF GitHub Bot commented on FLINK-7385:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4496


> Fix ArrayIndexOutOfBoundsException when object-reuse is enabled  
> -
>
> Key: FLINK-7385
> URL: https://issues.apache.org/jira/browse/FLINK-7385
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.1.5, 1.2.1, 1.3.2
>Reporter: Ruidong Li
>Assignee: Ruidong Li
>Priority: Blocker
> Fix For: 1.4.0, 1.3.3
>
>
> In OperatorChain.java, there is a potential ArrayIndexOutOfBoundsException 
> when object-reuse is enabled



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4496: [FLINK-7385] Fix ArrayIndexOutOfBoundsException wh...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4496


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7337) Refactor handling of time indicator attributes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118198#comment-16118198
 ] 

ASF GitHub Bot commented on FLINK-7337:
---

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/4488
  
I'm fine with a followup issue. +1 to merge this.


> Refactor handling of time indicator attributes
> --
>
> Key: FLINK-7337
> URL: https://issues.apache.org/jira/browse/FLINK-7337
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Affects Versions: 1.4.0
>Reporter: Fabian Hueske
>Assignee: Fabian Hueske
>
> After a [discussion on the dev mailing 
> list|https://lists.apache.org/thread.html/735d55f9022df8ff73566a9f1553e14be94f8443986ad46559b35869@%3Cdev.flink.apache.org%3E]
>  I propose the following changes to the current handling of time indicator 
> attributes:
> * Remove the separation of logical and physical row type.
> ** Hold the event-time timestamp as regular Long field in Row
> ** Represent the processing-time indicator type as a null-valued field in Row 
> (1 bit overhead)
> * Remove materialization of event-time timestamps because timestamp is 
> already accessible in Row.
> * Add {{ProcessFunction}} to set timestamp into the timestamp field of a 
> {{StreamRecord}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4488: [FLINK-7337] [table] Refactor internal handling of time i...

[twalthr]

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/4488
  
I'm fine with a followup issue. +1 to merge this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (FLINK-7389) Remove Calcite PushProjector

[Timo Walther (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timo Walther updated FLINK-7389:

Issue Type: Sub-task  (was: Improvement)
Parent: FLINK-7051

> Remove Calcite PushProjector
> 
>
> Key: FLINK-7389
> URL: https://issues.apache.org/jira/browse/FLINK-7389
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API & SQL
>Reporter: Timo Walther
>
> The PushProjector class is copied from Apache Calcite except that it does not 
> automatically name the field using the name of the operators as the Table API 
> rejects special characters like '-' in the field names.
> We need to find a solution without copying Calcite classes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7389) Remove Calcite PushProjector

[Timo Walther (JIRA)]

Timo Walther created FLINK-7389:
---

 Summary: Remove Calcite PushProjector
 Key: FLINK-7389
 URL: https://issues.apache.org/jira/browse/FLINK-7389
 Project: Flink
  Issue Type: Improvement
  Components: Table API & SQL
Reporter: Timo Walther


The PushProjector class is copied from Apache Calcite except that it does not 
automatically name the field using the name of the operators as the Table API 
rejects special characters like '-' in the field names.

We need to find a solution without copying Calcite classes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-7236) Bump up the Calcite version to 1.14

[Timo Walther (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timo Walther closed FLINK-7236.
---
Resolution: Duplicate

> Bump up the Calcite version to 1.14
> ---
>
> Key: FLINK-7236
> URL: https://issues.apache.org/jira/browse/FLINK-7236
> Project: Flink
>  Issue Type: Bug
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>
> This is the umbrella task to coordinate tasks to upgrade Calcite to 1.14.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (FLINK-7237) Remove DateTimeUtils from Flink once Calcite is upgraded to 1.14

[Timo Walther (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timo Walther updated FLINK-7237:

Issue Type: Sub-task  (was: Bug)
Parent: FLINK-7051

> Remove DateTimeUtils from Flink once Calcite is upgraded to 1.14
> 
>
> Key: FLINK-7237
> URL: https://issues.apache.org/jira/browse/FLINK-7237
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API & SQL
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (FLINK-7237) Remove DateTimeUtils from Flink once Calcite is upgraded to 1.14

[Timo Walther (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timo Walther updated FLINK-7237:

Issue Type: Bug  (was: Sub-task)
Parent: (was: FLINK-7236)

> Remove DateTimeUtils from Flink once Calcite is upgraded to 1.14
> 
>
> Key: FLINK-7237
> URL: https://issues.apache.org/jira/browse/FLINK-7237
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (FLINK-7235) Backport CALCITE-1884 to the Flink repository before Calcite 1.14

[Timo Walther (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timo Walther updated FLINK-7235:

Issue Type: Sub-task  (was: Bug)
Parent: FLINK-7051

> Backport CALCITE-1884 to the Flink repository before Calcite 1.14
> -
>
> Key: FLINK-7235
> URL: https://issues.apache.org/jira/browse/FLINK-7235
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API & SQL
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>
> We need to backport CALCITE-1884 in order to unblock upgrading Calcite to 
> 1.13.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (FLINK-7235) Backport CALCITE-1884 to the Flink repository before Calcite 1.14

[Timo Walther (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timo Walther updated FLINK-7235:

Issue Type: Bug  (was: Sub-task)
Parent: (was: FLINK-6429)

> Backport CALCITE-1884 to the Flink repository before Calcite 1.14
> -
>
> Key: FLINK-7235
> URL: https://issues.apache.org/jira/browse/FLINK-7235
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>
> We need to backport CALCITE-1884 in order to unblock upgrading Calcite to 
> 1.13.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (FLINK-6429) Bump up Calcite version to 1.13

[Timo Walther (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timo Walther resolved FLINK-6429.
-
   Resolution: Fixed
Fix Version/s: 1.4.0

Fixed in 1.4.0: d5770fe8dd1486d457c87c17a7df8dba276e9bcd & 
4dfefd042b151a2ecf8d947da5364fc81470f3d9

> Bump up Calcite version to 1.13
> ---
>
> Key: FLINK-6429
> URL: https://issues.apache.org/jira/browse/FLINK-6429
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Haohui Mai
> Fix For: 1.4.0
>
>
> This is an umbrella issue for all tasks that need to be done once Apache 
> Calcite 1.13 is released.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6429) Bump up Calcite version to 1.13

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118187#comment-16118187
 ] 

ASF GitHub Bot commented on FLINK-6429:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4373


> Bump up Calcite version to 1.13
> ---
>
> Key: FLINK-6429
> URL: https://issues.apache.org/jira/browse/FLINK-6429
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Haohui Mai
>
> This is an umbrella issue for all tasks that need to be done once Apache 
> Calcite 1.13 is released.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4373: [FLINK-6429] [table] Bump up Calcite version to 1....

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4373


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7385) Fix ArrayIndexOutOfBoundsException when object-reuse is enabled

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118124#comment-16118124
 ] 

ASF GitHub Bot commented on FLINK-7385:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/4496
  
Thanks, @tzulitai !  


> Fix ArrayIndexOutOfBoundsException when object-reuse is enabled  
> -
>
> Key: FLINK-7385
> URL: https://issues.apache.org/jira/browse/FLINK-7385
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.1.5, 1.2.1, 1.3.2
>Reporter: Ruidong Li
>Assignee: Ruidong Li
>Priority: Blocker
> Fix For: 1.4.0, 1.3.3
>
>
> In OperatorChain.java, there is a potential ArrayIndexOutOfBoundsException 
> when object-reuse is enabled



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4496: [FLINK-7385] Fix ArrayIndexOutOfBoundsException when obje...

[aljoscha]

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/4496
  
Thanks, @tzulitai ! 😃 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7388) ProcessFunction.onTimer() sets processing time as timestamp

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118122#comment-16118122
 ] 

Aljoscha Krettek commented on FLINK-7388:
-

This sounds legit, as long as we don't have explicit control over the timestamp 
of data that is emitted as the result of a timer firing. See also relevant Beam 
issue: BEAM-2535.

> ProcessFunction.onTimer() sets processing time as timestamp
> ---
>
> Key: FLINK-7388
> URL: https://issues.apache.org/jira/browse/FLINK-7388
> Project: Flink
>  Issue Type: Improvement
>  Components: DataStream API
>Affects Versions: 1.4.0, 1.3.2
>Reporter: Fabian Hueske
>
> The {{ProcessFunction.onTimer()}} method sets the current processing time as 
> event-time timestamp when it is called from a processing time timer.
> I don't think this behavior is useful. Processing time timestamps won't be 
> aligned with watermarks and are not deterministic. The only reason would be 
> to have _some_ value in the timestamp field. However, the behavior is very 
> subtle and might not be noticed by users.
> IMO, it would be better to erase the timestamp. This will cause downstream 
> operator that rely on timestamps to fail and notify the users that the logic 
> they implemented was probably not what they intended to do.
> What do you think [~aljoscha]?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7388) ProcessFunction.onTimer() sets processing time as timestamp

[Fabian Hueske (JIRA)]

Fabian Hueske created FLINK-7388:


 Summary: ProcessFunction.onTimer() sets processing time as 
timestamp
 Key: FLINK-7388
 URL: https://issues.apache.org/jira/browse/FLINK-7388
 Project: Flink
  Issue Type: Improvement
  Components: DataStream API
Affects Versions: 1.3.2, 1.4.0
Reporter: Fabian Hueske


The {{ProcessFunction.onTimer()}} method sets the current processing time as 
event-time timestamp when it is called from a processing time timer.

I don't think this behavior is useful. Processing time timestamps won't be 
aligned with watermarks and are not deterministic. The only reason would be to 
have _some_ value in the timestamp field. However, the behavior is very subtle 
and might not be noticed by users.

IMO, it would be better to erase the timestamp. This will cause downstream 
operator that rely on timestamps to fail and notify the users that the logic 
they implemented was probably not what they intended to do.

What do you think [~aljoscha]?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7337) Refactor handling of time indicator attributes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118114#comment-16118114
 ] 

ASF GitHub Bot commented on FLINK-7337:
---

Github user fhueske commented on the issue:

https://github.com/apache/flink/pull/4488
  
Thanks for the review @twalthr. I addressed your comments and updated the 
PR.

I think it would be very good to handle the timestamps internally as longs. 
The change seems to be a bit more involved because we need to touch the 
serialization logic and various type conversion and code generation issues. I'd 
rather do this as a follow up to this PR. What do you think?

Fabian


> Refactor handling of time indicator attributes
> --
>
> Key: FLINK-7337
> URL: https://issues.apache.org/jira/browse/FLINK-7337
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Affects Versions: 1.4.0
>Reporter: Fabian Hueske
>Assignee: Fabian Hueske
>
> After a [discussion on the dev mailing 
> list|https://lists.apache.org/thread.html/735d55f9022df8ff73566a9f1553e14be94f8443986ad46559b35869@%3Cdev.flink.apache.org%3E]
>  I propose the following changes to the current handling of time indicator 
> attributes:
> * Remove the separation of logical and physical row type.
> ** Hold the event-time timestamp as regular Long field in Row
> ** Represent the processing-time indicator type as a null-valued field in Row 
> (1 bit overhead)
> * Remove materialization of event-time timestamps because timestamp is 
> already accessible in Row.
> * Add {{ProcessFunction}} to set timestamp into the timestamp field of a 
> {{StreamRecord}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4488: [FLINK-7337] [table] Refactor internal handling of time i...

[fhueske]

Github user fhueske commented on the issue:

https://github.com/apache/flink/pull/4488
  
Thanks for the review @twalthr. I addressed your comments and updated the 
PR.

I think it would be very good to handle the timestamps internally as longs. 
The change seems to be a bit more involved because we need to touch the 
serialization logic and various type conversion and code generation issues. I'd 
rather do this as a follow up to this PR. What do you think?

Fabian


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (FLINK-7343) Kafka010ProducerITCase instability

[Piotr Nowojski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Piotr Nowojski closed FLINK-7343.
-
   Resolution: Fixed
Fix Version/s: 1.4.0

Hopefully with two recent changes this issue will not show up again.

> Kafka010ProducerITCase instability
> --
>
> Key: FLINK-7343
> URL: https://issues.apache.org/jira/browse/FLINK-7343
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Reporter: Piotr Nowojski
>Assignee: Piotr Nowojski
>  Labels: test-stability
> Fix For: 1.4.0
>
>
> As reported by [~till.rohrmann] in 
> https://issues.apache.org/jira/browse/FLINK-6996 there seems to be a test 
> instability with 
> `Kafka010ProducerITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink`
> https://travis-ci.org/tillrohrmann/flink/jobs/258538641
> It is probably related to log.flush intervals in Kafka, which delay flushing 
> the data to files and potentially causing data loses on killing Kafka brokers 
> in the tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7343) Kafka010ProducerITCase instability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118105#comment-16118105
 ] 

ASF GitHub Bot commented on FLINK-7343:
---

Github user pnowojski closed the pull request at:

https://github.com/apache/flink/pull/4470


> Kafka010ProducerITCase instability
> --
>
> Key: FLINK-7343
> URL: https://issues.apache.org/jira/browse/FLINK-7343
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Reporter: Piotr Nowojski
>Assignee: Piotr Nowojski
>  Labels: test-stability
>
> As reported by [~till.rohrmann] in 
> https://issues.apache.org/jira/browse/FLINK-6996 there seems to be a test 
> instability with 
> `Kafka010ProducerITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink`
> https://travis-ci.org/tillrohrmann/flink/jobs/258538641
> It is probably related to log.flush intervals in Kafka, which delay flushing 
> the data to files and potentially causing data loses on killing Kafka brokers 
> in the tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7343) Kafka010ProducerITCase instability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118104#comment-16118104
 ] 

ASF GitHub Bot commented on FLINK-7343:
---

Github user pnowojski commented on the issue:

https://github.com/apache/flink/pull/4470
  
Thanks :)


> Kafka010ProducerITCase instability
> --
>
> Key: FLINK-7343
> URL: https://issues.apache.org/jira/browse/FLINK-7343
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Reporter: Piotr Nowojski
>Assignee: Piotr Nowojski
>  Labels: test-stability
>
> As reported by [~till.rohrmann] in 
> https://issues.apache.org/jira/browse/FLINK-6996 there seems to be a test 
> instability with 
> `Kafka010ProducerITCase>KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink`
> https://travis-ci.org/tillrohrmann/flink/jobs/258538641
> It is probably related to log.flush intervals in Kafka, which delay flushing 
> the data to files and potentially causing data loses on killing Kafka brokers 
> in the tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4470: [FLINK-7343] Simulate network failures in kafka at...

[pnowojski]

Github user pnowojski closed the pull request at:

https://github.com/apache/flink/pull/4470


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #4470: [FLINK-7343] Simulate network failures in kafka at-least-...

[pnowojski]

Github user pnowojski commented on the issue:

https://github.com/apache/flink/pull/4470
  
Thanks :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3580: [FLINK-5791] Support an optimal matching based slot manag...

[shuai-xu]

Github user shuai-xu commented on the issue:

https://github.com/apache/flink/pull/3580
  
@tillrohrmann Could you please help to review this pr? And there is a 
question, the pending requests in slot managers are not in order. So requests 
arrived later may be fulfilled earlier, does this matter?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5791) Support an optimal matching based slot manager for flip6 yarn mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118098#comment-16118098
 ] 

ASF GitHub Bot commented on FLINK-5791:
---

Github user shuai-xu commented on the issue:

https://github.com/apache/flink/pull/3580
  
@tillrohrmann Could you please help to review this pr? And there is a 
question, the pending requests in slot managers are not in order. So requests 
arrived later may be fulfilled earlier, does this matter?


> Support an optimal matching based slot manager for flip6 yarn mode
> --
>
> Key: FLINK-5791
> URL: https://issues.apache.org/jira/browse/FLINK-5791
> Project: Flink
>  Issue Type: Improvement
>  Components: YARN
>Reporter: shuai.xu
>Assignee: shuai.xu
>  Labels: flip-6
>
> In flip6, for yarn mode, it request containers from yarn according to the 
> resource profile in slot requests. And when slots come back after container 
> starting up, we should supporting an optimal matching based slot manager 
> which can assign the slot to the request whose resource profile closest to 
> it. This could reduce resource waste and avoid OOM.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4473: [FLINK-7367][kinesis connector] Parameterize more configs...

[tzulitai]

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/4473
  
+1 to option 3. Looks like we all agree on that :) @bowenli86 please feel 
free to continue with that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7367) Parameterize more configs for FlinkKinesisProducer (RecordMaxBufferedTime, MaxConnections, RequestTimeout, etc)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118084#comment-16118084
 ] 

ASF GitHub Bot commented on FLINK-7367:
---

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/4473
  
+1 to option 3. Looks like we all agree on that :) @bowenli86 please feel 
free to continue with that.


> Parameterize more configs for FlinkKinesisProducer (RecordMaxBufferedTime, 
> MaxConnections, RequestTimeout, etc)
> ---
>
> Key: FLINK-7367
> URL: https://issues.apache.org/jira/browse/FLINK-7367
> Project: Flink
>  Issue Type: Improvement
>  Components: Kinesis Connector
>Affects Versions: 1.3.0
>Reporter: Bowen Li
>Assignee: Bowen Li
> Fix For: 1.3.3
>
>
> Right now, FlinkKinesisProducer only expose two configs for the underlying 
> KinesisProducer:
> - AGGREGATION_MAX_COUNT
> - COLLECTION_MAX_COUNT
> Well, according to [AWS 
> doc|http://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-config.html] 
> and [their sample on 
> github|https://github.com/awslabs/amazon-kinesis-producer/blob/master/java/amazon-kinesis-producer-sample/default_config.properties],
>  developers can set more to make the max use of KinesisProducer, and make it 
> fault-tolerant (e.g. by increasing timeout).
> I select a few more configs that we need when using Flink with Kinesis:
> - MAX_CONNECTIONS
> - RATE_LIMIT
> - RECORD_MAX_BUFFERED_TIME
> - RECORD_TIME_TO_LIVE
> - REQUEST_TIMEOUT
> We need to parameterize FlinkKinesisProducer to pass in the above params, in 
> order to cater to our need



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4496: [FLINK-7385] Fix ArrayIndexOutOfBoundsException when obje...

[tzulitai]

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/4496
  
Thanks! Merging this to `master` and `release-1.3` ..


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7385) Fix ArrayIndexOutOfBoundsException when object-reuse is enabled

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118079#comment-16118079
 ] 

ASF GitHub Bot commented on FLINK-7385:
---

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/4496
  
Thanks! Merging this to `master` and `release-1.3` ..


> Fix ArrayIndexOutOfBoundsException when object-reuse is enabled  
> -
>
> Key: FLINK-7385
> URL: https://issues.apache.org/jira/browse/FLINK-7385
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.1.5, 1.2.1, 1.3.2
>Reporter: Ruidong Li
>Assignee: Ruidong Li
>Priority: Blocker
> Fix For: 1.4.0, 1.3.3
>
>
> In OperatorChain.java, there is a potential ArrayIndexOutOfBoundsException 
> when object-reuse is enabled



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-4499) Introduce findbugs maven plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118073#comment-16118073
 ] 

ASF GitHub Bot commented on FLINK-4499:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/2422
  
@smarthi, could you please close this PR, now that we have added spotbugs 
support?


> Introduce findbugs maven plugin
> ---
>
> Key: FLINK-4499
> URL: https://issues.apache.org/jira/browse/FLINK-4499
> Project: Flink
>  Issue Type: Improvement
>  Components: Build System
>Reporter: Ted Yu
>Assignee: Suneel Marthi
> Fix For: 1.4.0
>
>
> As suggested by Stephan in FLINK-4482, this issue is to add 
> findbugs-maven-plugin into the build process so that we can detect lack of 
> proper locking and other defects automatically.
> We can begin with small set of rules.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #2422: FLINK-4499: [WIP] Introduce findbugs maven plugin

[aljoscha]

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/2422
  
@smarthi, could you please close this PR, now that we have added spotbugs 
support?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7373) Using URLClassLoader to load UDF triggers HepPlanner unexpected ClassNotFoundException

[Fabian Hueske (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118069#comment-16118069
 ] 

Fabian Hueske commented on FLINK-7373:
--

Hi [~walterddr], I looked into this issue and could reproduce it.
The problem is that the {{HelloWorld}} class is not available when the 
generated code is compiled. In order to execute the code, the class must be 
present in the classpath.

I see two solutions:
1) add the jar file with the UDFs classes to the classpath (include it in the 
user jar or put it into the flink lib folder)
2) extend the function registration with a special remote jar case that 
dynamically loads jars before a function is compiled.

Option 2 requires quite a few changes because we need to provide the paths of 
all used jars to the code compiling (and running) functions.

> Using URLClassLoader to load UDF triggers HepPlanner unexpected 
> ClassNotFoundException
> --
>
> Key: FLINK-7373
> URL: https://issues.apache.org/jira/browse/FLINK-7373
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Affects Versions: 1.3.1
>Reporter: Rong Rong
>
> Using URLClassLoader to load, say from Artifactory, and instantiate UDF 
> instances will cause some Rule failed during runHepPlanner or 
> runVolcanoPlanner.
> One example could add an ITCase in:
> {code:title=flink-libraries/flink-table/src/test/scala/org/apache/flink/table/runtime/batch/table/CalcITCase.scala}
>   @Test
>   def testUserDefinedFunctionDynamicClassloader() {
> val env = ExecutionEnvironment.getExecutionEnvironment
> val tableEnv = TableEnvironment.getTableEnvironment(env, config)
> val jarFileURI = "file://org/apache/flink/table/udf/HelloWorld.jar"
> val udfClassLoader: ClassLoader = new URLClassLoader(List(new 
> URI(jarFileURI).toURL).toArray)
> val clazz = 
> udfClassLoader.loadClass("org.apache.flink.table.udf.HelloWorld")
> val helloWorldUDF: ScalarFunction = 
> clazz.newInstance().asInstanceOf[ScalarFunction]
> tableEnv.registerFunction("helloWorld", helloWorldUDF)
> val table = env.fromElements("a", "b", "c").toTable(tableEnv, 'text)
> val result = table.select("text.helloWorld()")
> val results = result.toDataSet[Row].collect()
> val expected = "Hello World"
> TestBaseUtils.compareResultAsText(results.asJava, expected)
>   }
> {code}
> where
> {code:title=HelloWorld.java}
> package org.apache.flink.table.udf;
> import org.apache.flink.table.functions.ScalarFunction;
> public class HelloWorld extends ScalarFunction {
>   public String eval() {
> return "Hello World";
>   }
> }
> {code}
> This triggers the following Exception:
> {panel:title=Exception}
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be 
> compiled. This is a bug. Please file an issue.
>   at 
> org.apache.flink.table.codegen.Compiler$class.compile(Compiler.scala:36)
>   at 
> org.apache.flink.table.runtime.FlatMapRunner.compile(FlatMapRunner.scala:31)
>   at 
> org.apache.flink.table.runtime.FlatMapRunner.open(FlatMapRunner.scala:45)
>   at 
> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
>   at 
> org.apache.flink.api.common.operators.base.FlatMapOperatorBase.executeOnCollections(FlatMapOperatorBase.java:62)
> ..
> Caused by: org.codehaus.commons.compiler.CompileException: Line 6, Column 22: 
> Cannot determine simple type name "org"
>   at 
> org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:11672)
>   at 
> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:6416)
>   at 
> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:6177)
> ..
> {panel}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4495: [FLINK-6982] [guava] Introduce flink-shaded-guava-...

[zentol]

Github user zentol closed the pull request at:

https://github.com/apache/flink/pull/4495


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---