[jira] [Commented] (FLINK-6990) Poor performance with Sliding Time Windows
[
https://issues.apache.org/jira/browse/FLINK-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060477#comment-16060477
]
Cody commented on FLINK-6990:
-
I ran the code on my Mac and didn't see the pause.
> Poor performance with Sliding Time Windows
> --
>
> Key: FLINK-6990
> URL: https://issues.apache.org/jira/browse/FLINK-6990
> Project: Flink
> Issue Type: Improvement
>Affects Versions: 1.3.0
> Environment: OSX 10.11.4
> 2.8 GHz Intel Core i7
> 16 GB 1600 MHz DDR3
>Reporter: Brice Bingman
>
> I'm experiencing poor performance when using sliding time windows. Here is a
> simple example that performs poorly for me:
> {code:java}
> public class FlinkPerfTest {
> public static void main(String[] args) throws Exception {
> StreamExecutionEnvironment see =
> StreamExecutionEnvironment.getExecutionEnvironment();
> //Streaming 10,000 events per second
> see.addSource(new SourceFunction() {
> transient ScheduledExecutorService executor;
> @Override
> public synchronized void run(final SourceContext ctx)
> throws Exception {
> executor = Executors.newSingleThreadScheduledExecutor();
> executor.scheduleAtFixedRate(new Runnable() {
> @Override
> public void run() {
> for (int k = 0; k < 10; k++) {
> for (int i = 0; i < 1000; i++) {
> TestObject obj = new TestObject();
> obj.setKey(k);
> ctx.collect(obj);
> }
> }
> }
> }, 0, 1, TimeUnit.SECONDS);
> this.wait();
> }
> @Override
> public synchronized void cancel() {
> executor.shutdown();
> this.notify();
> }
> }).keyBy("key")
> .window(SlidingProcessingTimeWindows.of(Time.minutes(10),
> Time.seconds(1))).apply(new WindowFunction TimeWindow>() {
> @Override
> public void apply(Tuple key, TimeWindow window,
> Iterable input, Collector out) throws Exception {
> int count = 0;
> for (Object obj : input) {
> count++;
> }
> out.collect(key.getField(0) + ": " + count);
> }
> })
> .print();
> see.execute();
> }
> public static class TestObject {
> private Integer key;
> public Integer getKey() {
> return key;
> }
> public void setKey(Integer key) {
> this.key = key;
> }
> }
> }
> {code}
> When running this, flink periodically pauses for long periods of time. I
> would expect a steady stream of output at 1 second intervals. For
> comparison, you can switch to a count window of similar size which peforms
> just fine:
> {code:java}
>.countWindow(60, 1000).apply(new
> WindowFunction() {
> @Override
> public void apply(Tuple key, GlobalWindow window,
> Iterable input, Collector out) throws Exception {
> int count = 0;
> for (Object obj : input) {
> count++;
> }
> out.collect(key.getField(0) + ": " + count);
> }
> })
> {code}
> I would expect the sliding time window to perform similarly to a count
> window. The sliding time window also uses significantly more cpu and memory
> than the count window. I would also expect resource consumption to be
> similar.
> A possible cause could be that the SystemProcessingTimeService.TriggerTask is
> locking with the checkpointLock which acts like a global lock. There should
> be a lock per key or preferably a lock-less solution.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6960) Add E() supported in SQL
[ https://issues.apache.org/jira/browse/FLINK-6960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060458#comment-16060458 ] ASF GitHub Bot commented on FLINK-6960: --- Github user wuchong commented on the issue: https://github.com/apache/flink/pull/4152 Thanks for the updating. I will merge this. > Add E() supported in SQL > > > Key: FLINK-6960 > URL: https://issues.apache.org/jira/browse/FLINK-6960 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL >Reporter: sunjincheng >Assignee: sunjincheng > Labels: starter > > E=Math.E -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink issue #4152: [FLINK-6960][table] Add E supported in SQL.
Github user wuchong commented on the issue: https://github.com/apache/flink/pull/4152 Thanks for the updating. I will merge this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6966) Add maxParallelism and UIDs to all operators generated by the Table API / SQL
[ https://issues.apache.org/jira/browse/FLINK-6966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060425#comment-16060425 ] Jark Wu commented on FLINK-6966: Yes, I agree with you [~sunjincheng121]. > Add maxParallelism and UIDs to all operators generated by the Table API / SQL > - > > Key: FLINK-6966 > URL: https://issues.apache.org/jira/browse/FLINK-6966 > Project: Flink > Issue Type: Improvement > Components: Table API & SQL >Affects Versions: 1.4.0 >Reporter: Fabian Hueske > > At the moment, the Table API does not assign UIDs and the max parallelism to > operators (except for operators with parallelism 1). > We should do that to avoid problems when rescaling or restarting jobs from > savepoints. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-6649) Improve Non-window group aggregate with configurable `earlyFire`.
[ https://issues.apache.org/jira/browse/FLINK-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060404#comment-16060404 ] ASF GitHub Bot commented on FLINK-6649: --- Github user sunjincheng121 commented on the issue: https://github.com/apache/flink/pull/4157 I have update the PR. The change is Move setAggregationResults from processElement method to onTimer method. Benefits are: * Reduce the delay. * Reduce row copy. Welcome anybody to review. Best, SunJincheng > Improve Non-window group aggregate with configurable `earlyFire`. > - > > Key: FLINK-6649 > URL: https://issues.apache.org/jira/browse/FLINK-6649 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL >Affects Versions: 1.4.0 >Reporter: sunjincheng >Assignee: sunjincheng > > Currently, Non-windowed group aggregate is earlyFiring at count(1), that is > every row will emit a aggregate result. But some times user want config count > number (`early firing with count[N]`) , to reduce the downstream pressure. > This JIRA. will enable the config of e`earlyFiring` for Non-windowed group > aggregate. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink issue #4157: [FLINK-6649][table]Improve Non-window group aggregate wit...
Github user sunjincheng121 commented on the issue: https://github.com/apache/flink/pull/4157 I have update the PR. The change is Move setAggregationResults from processElement method to onTimer method. Benefits are: * Reduce the delay. * Reduce row copy. Welcome anybody to review. Best, SunJincheng --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-6992) add support for IN, NOT IN in tableAPI
Ruidong Li created FLINK-6992:
-
Summary: add support for IN, NOT IN in tableAPI
Key: FLINK-6992
URL: https://issues.apache.org/jira/browse/FLINK-6992
Project: Flink
Issue Type: Improvement
Components: Table API & SQL
Reporter: Ruidong Li
Assignee: Ruidong Li
Priority: Minor
to support table.where('a in ("x", "y", "z"))
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLINK-6991) Inaccessible link under Gelly document
[
https://issues.apache.org/jira/browse/FLINK-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mingleizhang updated FLINK-6991:
Component/s: Documentation
> Inaccessible link under Gelly document
> --
>
> Key: FLINK-6991
> URL: https://issues.apache.org/jira/browse/FLINK-6991
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Reporter: mingleizhang
>
> When I visited the web
> http://flink.apache.org/news/2015/08/24/introducing-flink-gelly.html#top,
> then at the bottom of the page there is a Gelly Documentation link. It
> belongs to a non-existent resources. It gives me the following style stuff.
> {noformat}
> No Such Resource
> File not found.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (FLINK-6991) Inaccessible link under Gelly document
[
https://issues.apache.org/jira/browse/FLINK-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mingleizhang reassigned FLINK-6991:
---
Assignee: (was: mingleizhang)
> Inaccessible link under Gelly document
> --
>
> Key: FLINK-6991
> URL: https://issues.apache.org/jira/browse/FLINK-6991
> Project: Flink
> Issue Type: Bug
>Reporter: mingleizhang
>
> When I visited the web
> http://flink.apache.org/news/2015/08/24/introducing-flink-gelly.html#top,
> then at the bottom of the page there is a Gelly Documentation link. It
> belongs to a non-existent resources. It gives me the following style stuff.
> {noformat}
> No Such Resource
> File not found.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLINK-6991) Inaccessible link under Gelly document
[
https://issues.apache.org/jira/browse/FLINK-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mingleizhang updated FLINK-6991:
Description:
When I visited the web
http://flink.apache.org/news/2015/08/24/introducing-flink-gelly.html#top, then
at the bottom of the page there is a Gelly Documentation link. It belongs to a
non-existent resources. It gives me the following style stuff.
{noformat}
No Such Resource
File not found.
{noformat}
was:When I visited the web
http://flink.apache.org/news/2015/08/24/introducing-flink-gelly.html#top, then
at the bottom of the page there is a Gelly Documentation link. It belongs to a
non-existent resources.
> Inaccessible link under Gelly document
> --
>
> Key: FLINK-6991
> URL: https://issues.apache.org/jira/browse/FLINK-6991
> Project: Flink
> Issue Type: Bug
>Reporter: mingleizhang
>Assignee: mingleizhang
>
> When I visited the web
> http://flink.apache.org/news/2015/08/24/introducing-flink-gelly.html#top,
> then at the bottom of the page there is a Gelly Documentation link. It
> belongs to a non-existent resources. It gives me the following style stuff.
> {noformat}
> No Such Resource
> File not found.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-6991) Inaccessible link under Gelly document
mingleizhang created FLINK-6991: --- Summary: Inaccessible link under Gelly document Key: FLINK-6991 URL: https://issues.apache.org/jira/browse/FLINK-6991 Project: Flink Issue Type: Bug Reporter: mingleizhang Assignee: mingleizhang When I visited the web http://flink.apache.org/news/2015/08/24/introducing-flink-gelly.html#top, then at the bottom of the page there is a Gelly Documentation link. It belongs to a non-existent resources. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-6937) Fix link markdown in Production Readiness Checklist doc
[ https://issues.apache.org/jira/browse/FLINK-6937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060320#comment-16060320 ] ASF GitHub Bot commented on FLINK-6937: --- Github user juanpaulo commented on the issue: https://github.com/apache/flink/pull/4134 > I will fix it this time for you :) cool. thanks! > Fix link markdown in Production Readiness Checklist doc > --- > > Key: FLINK-6937 > URL: https://issues.apache.org/jira/browse/FLINK-6937 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Juan Paulo Gutierrez >Priority: Minor > Fix For: 1.4.0, 1.3.2 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink issue #4134: [FLINK-6937] Fix link markdown in Production Readiness Ch...
Github user juanpaulo commented on the issue: https://github.com/apache/flink/pull/4134 > I will fix it this time for you :) cool. thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6649) Improve Non-window group aggregate with configurable `earlyFire`.
[
https://issues.apache.org/jira/browse/FLINK-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060289#comment-16060289
]
ASF GitHub Bot commented on FLINK-6649:
---
Github user sunjincheng121 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4157#discussion_r123656803
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/runtime/aggregate/GroupAggProcessFunctionWithUpdateInterval.scala
---
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.table.runtime.aggregate
+
+import java.lang.{Long => JLong}
+
+import org.apache.flink.api.common.ExecutionConfig
+import org.apache.flink.api.common.state.{ValueState, ValueStateDescriptor}
+import org.apache.flink.api.common.typeutils.TypeSerializer
+import org.apache.flink.api.java.typeutils.RowTypeInfo
+import org.apache.flink.configuration.Configuration
+import org.apache.flink.streaming.api.functions.ProcessFunction
+import org.apache.flink.table.api.{StreamQueryConfig, Types}
+import org.apache.flink.table.codegen.{Compiler,
GeneratedAggregationsFunction}
+import org.apache.flink.table.runtime.types.CRow
+import org.apache.flink.types.Row
+import org.apache.flink.util.Collector
+import org.slf4j.{Logger, LoggerFactory}
+
+/**
+ * Aggregate Function used for the groupby (without window) aggregate
+ * with update interval config.
+ *
+ * @param genAggregations Generated aggregate helper function
+ * @param aggregationStateType The row type info of aggregation
+ * @param outputRowType The row type info of output.
+ */
+class GroupAggProcessFunctionWithUpdateInterval(
+private val genAggregations: GeneratedAggregationsFunction,
+private val aggregationStateType: RowTypeInfo,
+private val outputRowType: RowTypeInfo,
+private val generateRetraction: Boolean,
+private val queryConfig: StreamQueryConfig)
+ extends ProcessFunctionWithCleanupState[CRow, CRow](queryConfig)
+with Compiler[GeneratedAggregations] {
+
+ protected val LOG: Logger = LoggerFactory.getLogger(this.getClass)
+ protected var function: GeneratedAggregations = _
+
+ private var emitRow: Row = _
+ protected var newRow: CRow = _
+ protected var prevRow: CRow = _
+
+ private var typeSerializer: TypeSerializer[Row] = _
+
+ // stores the accumulators
+ protected var state: ValueState[Row] = _
+
+ // counts the number of added and retracted input records
+ protected var cntState: ValueState[JLong] = _
+
+ // stores the last emit row
+ private var preEmitState: ValueState[Row] = _
+
+ // stores the current emit row
+ private var emitState: ValueState[Row] = _
+
+ // stores the emit time
+ private var emitTimerState: ValueState[JLong] = _
+
+
+ override def open(config: Configuration) {
+LOG.debug(s"Compiling AggregateHelper: ${genAggregations.name}\n\n" +
+s"Code:\n${genAggregations.code}")
+val clazz = compile(
+ getRuntimeContext.getUserCodeClassLoader,
+ genAggregations.name,
+ genAggregations.code)
+LOG.debug("Instantiating AggregateHelper.")
+function = clazz.newInstance()
+
+emitRow = function.createOutputRow
+newRow = new CRow(function.createOutputRow, true)
+prevRow = new CRow(function.createOutputRow, false)
+typeSerializer = outputRowType.createSerializer(new ExecutionConfig())
+
+state = getRuntimeContext.getState(
+ new ValueStateDescriptor[Row]("GroupAggregateState",
aggregationStateType))
+cntState = getRuntimeContext.getState(
+ new ValueStateDescriptor[JLong]("GroupAggregateInputCounter",
Types.LONG))
+preEmitState = getRuntimeContext.getState(
+ new ValueStateDescriptor[Row]("GroupAggregatePreEmitState",
outputRowTyp
[GitHub] flink pull request #4157: [FLINK-6649][table]Improve Non-window group aggreg...
Github user sunjincheng121 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4157#discussion_r123656803
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/runtime/aggregate/GroupAggProcessFunctionWithUpdateInterval.scala
---
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.table.runtime.aggregate
+
+import java.lang.{Long => JLong}
+
+import org.apache.flink.api.common.ExecutionConfig
+import org.apache.flink.api.common.state.{ValueState, ValueStateDescriptor}
+import org.apache.flink.api.common.typeutils.TypeSerializer
+import org.apache.flink.api.java.typeutils.RowTypeInfo
+import org.apache.flink.configuration.Configuration
+import org.apache.flink.streaming.api.functions.ProcessFunction
+import org.apache.flink.table.api.{StreamQueryConfig, Types}
+import org.apache.flink.table.codegen.{Compiler,
GeneratedAggregationsFunction}
+import org.apache.flink.table.runtime.types.CRow
+import org.apache.flink.types.Row
+import org.apache.flink.util.Collector
+import org.slf4j.{Logger, LoggerFactory}
+
+/**
+ * Aggregate Function used for the groupby (without window) aggregate
+ * with update interval config.
+ *
+ * @param genAggregations Generated aggregate helper function
+ * @param aggregationStateType The row type info of aggregation
+ * @param outputRowType The row type info of output.
+ */
+class GroupAggProcessFunctionWithUpdateInterval(
+private val genAggregations: GeneratedAggregationsFunction,
+private val aggregationStateType: RowTypeInfo,
+private val outputRowType: RowTypeInfo,
+private val generateRetraction: Boolean,
+private val queryConfig: StreamQueryConfig)
+ extends ProcessFunctionWithCleanupState[CRow, CRow](queryConfig)
+with Compiler[GeneratedAggregations] {
+
+ protected val LOG: Logger = LoggerFactory.getLogger(this.getClass)
+ protected var function: GeneratedAggregations = _
+
+ private var emitRow: Row = _
+ protected var newRow: CRow = _
+ protected var prevRow: CRow = _
+
+ private var typeSerializer: TypeSerializer[Row] = _
+
+ // stores the accumulators
+ protected var state: ValueState[Row] = _
+
+ // counts the number of added and retracted input records
+ protected var cntState: ValueState[JLong] = _
+
+ // stores the last emit row
+ private var preEmitState: ValueState[Row] = _
+
+ // stores the current emit row
+ private var emitState: ValueState[Row] = _
+
+ // stores the emit time
+ private var emitTimerState: ValueState[JLong] = _
+
+
+ override def open(config: Configuration) {
+LOG.debug(s"Compiling AggregateHelper: ${genAggregations.name}\n\n" +
+s"Code:\n${genAggregations.code}")
+val clazz = compile(
+ getRuntimeContext.getUserCodeClassLoader,
+ genAggregations.name,
+ genAggregations.code)
+LOG.debug("Instantiating AggregateHelper.")
+function = clazz.newInstance()
+
+emitRow = function.createOutputRow
+newRow = new CRow(function.createOutputRow, true)
+prevRow = new CRow(function.createOutputRow, false)
+typeSerializer = outputRowType.createSerializer(new ExecutionConfig())
+
+state = getRuntimeContext.getState(
+ new ValueStateDescriptor[Row]("GroupAggregateState",
aggregationStateType))
+cntState = getRuntimeContext.getState(
+ new ValueStateDescriptor[JLong]("GroupAggregateInputCounter",
Types.LONG))
+preEmitState = getRuntimeContext.getState(
+ new ValueStateDescriptor[Row]("GroupAggregatePreEmitState",
outputRowType))
+emitState = getRuntimeContext.getState(
+ new ValueStateDescriptor[Row]("GroupAggregateEmitState",
outputRowType))
+emitTimerState = getRuntimeContext.getState(
+ new ValueStateDescriptor[JLong]("emitTimeState", Type
[jira] [Commented] (FLINK-6358) Write job details for Gelly examples
[ https://issues.apache.org/jira/browse/FLINK-6358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060274#comment-16060274 ] ASF GitHub Bot commented on FLINK-6358: --- GitHub user greghogan opened a pull request: https://github.com/apache/flink/pull/4170 [FLINK-6358] [gelly] Write job details for Gelly examples Add an option to write job details to a file in JSON format. Job details include: job ID, runtime, parameters with values, and accumulators with values. You can merge this pull request into a Git repository by running: $ git pull https://github.com/greghogan/flink 6358_write_job_details_for_gelly_examples Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/4170.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4170 commit f7a89bc9e5ac5e9b34dd05db1cdcdde4e3a9882d Author: Greg Hogan Date: 2017-06-21T14:25:57Z [FLINK-6358] [gelly] Write job details for Gelly examples Add an option to write job details to a file in JSON format. Job details include: job ID, runtime, parameters with values, and accumulators with values. > Write job details for Gelly examples > > > Key: FLINK-6358 > URL: https://issues.apache.org/jira/browse/FLINK-6358 > Project: Flink > Issue Type: Improvement > Components: Gelly >Affects Versions: 1.3.0 >Reporter: Greg Hogan >Assignee: Greg Hogan >Priority: Minor > > Add an option to write job details to a file in JSON format. Job details > include: job ID, runtime, parameters with values, and accumulators with > values. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink pull request #4170: [FLINK-6358] [gelly] Write job details for Gelly e...
GitHub user greghogan opened a pull request: https://github.com/apache/flink/pull/4170 [FLINK-6358] [gelly] Write job details for Gelly examples Add an option to write job details to a file in JSON format. Job details include: job ID, runtime, parameters with values, and accumulators with values. You can merge this pull request into a Git repository by running: $ git pull https://github.com/greghogan/flink 6358_write_job_details_for_gelly_examples Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/4170.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4170 commit f7a89bc9e5ac5e9b34dd05db1cdcdde4e3a9882d Author: Greg Hogan Date: 2017-06-21T14:25:57Z [FLINK-6358] [gelly] Write job details for Gelly examples Add an option to write job details to a file in JSON format. Job details include: job ID, runtime, parameters with values, and accumulators with values. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6964) Fix recovery for incremental checkpoints in StandaloneCompletedCheckpointStore
[
https://issues.apache.org/jira/browse/FLINK-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060027#comment-16060027
]
Cliff Resnick commented on FLINK-6964:
--
[~srichter] I tried your fix. After resuming from a checkpoint, the first
subsequent checkpoint got to 100% and hung, then expired several minutes later.
The second one repeated this. If logging will help you perhaps if you can add
some TRACE level logs and let me know the scopes, and I'll create a gist.
> Fix recovery for incremental checkpoints in StandaloneCompletedCheckpointStore
> --
>
> Key: FLINK-6964
> URL: https://issues.apache.org/jira/browse/FLINK-6964
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Reporter: Stefan Richter
>Assignee: Stefan Richter
>
> {{StandaloneCompletedCheckpointStore}} does not register shared states ion
> resume. However, for externalized checkpoints, it register the checkpoint
> from which it resumed. This checkpoint gets added to the completed checkpoint
> store as part of resume.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-6990) Poor performance with Sliding Time Windows
Brice Bingman created FLINK-6990:
Summary: Poor performance with Sliding Time Windows
Key: FLINK-6990
URL: https://issues.apache.org/jira/browse/FLINK-6990
Project: Flink
Issue Type: Improvement
Affects Versions: 1.3.0
Environment: OSX 10.11.4
2.8 GHz Intel Core i7
16 GB 1600 MHz DDR3
Reporter: Brice Bingman
I'm experiencing poor performance when using sliding time windows. Here is a
simple example that performs poorly for me:
{code:java}
public class FlinkPerfTest {
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment see =
StreamExecutionEnvironment.getExecutionEnvironment();
//Streaming 10,000 events per second
see.addSource(new SourceFunction() {
transient ScheduledExecutorService executor;
@Override
public synchronized void run(final SourceContext ctx)
throws Exception {
executor = Executors.newSingleThreadScheduledExecutor();
executor.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
for (int k = 0; k < 10; k++) {
for (int i = 0; i < 1000; i++) {
TestObject obj = new TestObject();
obj.setKey(k);
ctx.collect(obj);
}
}
}
}, 0, 1, TimeUnit.SECONDS);
this.wait();
}
@Override
public synchronized void cancel() {
executor.shutdown();
this.notify();
}
}).keyBy("key")
.window(SlidingProcessingTimeWindows.of(Time.minutes(10),
Time.seconds(1))).apply(new WindowFunction() {
@Override
public void apply(Tuple key, TimeWindow window,
Iterable input, Collector out) throws Exception {
int count = 0;
for (Object obj : input) {
count++;
}
out.collect(key.getField(0) + ": " + count);
}
})
.print();
see.execute();
}
public static class TestObject {
private Integer key;
public Integer getKey() {
return key;
}
public void setKey(Integer key) {
this.key = key;
}
}
}
{code}
When running this, flink periodically pauses for long periods of time. I would
expect a steady stream of output at 1 second intervals. For comparison, you
can switch to a count window of similar size which peforms just fine:
{code:java}
.countWindow(60, 1000).apply(new
WindowFunction() {
@Override
public void apply(Tuple key, GlobalWindow window,
Iterable input, Collector out) throws Exception {
int count = 0;
for (Object obj : input) {
count++;
}
out.collect(key.getField(0) + ": " + count);
}
})
{code}
I would expect the sliding time window to perform similarly to a count window.
The sliding time window also uses significantly more cpu and memory than the
count window. I would also expect resource consumption to be similar.
A possible cause could be that the SystemProcessingTimeService.TriggerTask is
locking with the checkpointLock which acts like a global lock. There should be
a lock per key or preferably a lock-less solution.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6923) Kafka connector needs to expose information about in-flight record in AbstractFetcher base class
[ https://issues.apache.org/jira/browse/FLINK-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059867#comment-16059867 ] ASF GitHub Bot commented on FLINK-6923: --- Github user zhenzhongxu commented on the issue: https://github.com/apache/flink/pull/4149 Hi @tzulitai, in this particular case, we actually disabled Flink checkpointing (because we do not want to rely on fixed interval barrier to trigger sink flush/offset commit). As a workaround, we manually trigger snapshot and commit upon sink flushing by hooking into source operator's consumer object. Thus this change. We have also previously discussed this approach with @StephanEwen > Kafka connector needs to expose information about in-flight record in > AbstractFetcher base class > > > Key: FLINK-6923 > URL: https://issues.apache.org/jira/browse/FLINK-6923 > Project: Flink > Issue Type: Improvement > Components: Kafka Connector >Reporter: Zhenzhong Xu >Assignee: Zhenzhong Xu >Priority: Minor > > We have a use case where we have our custom Fetcher implementation that > extends AbstractFetcher base class. We need to periodically get current in > flight (in processing) records' partition and offset information. > This can be easily exposed in AbstractFetcher class. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink issue #4149: [FLINK-6923] [Kafka Connector] Expose in-processing/in-fl...
Github user zhenzhongxu commented on the issue: https://github.com/apache/flink/pull/4149 Hi @tzulitai, in this particular case, we actually disabled Flink checkpointing (because we do not want to rely on fixed interval barrier to trigger sink flush/offset commit). As a workaround, we manually trigger snapshot and commit upon sink flushing by hooking into source operator's consumer object. Thus this change. We have also previously discussed this approach with @StephanEwen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4167: Test PR, please ignore
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4167 @greghogan Sure, I'll try that in the second run! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6389) Upgrade hbase dependency to 1.3.1
[ https://issues.apache.org/jira/browse/FLINK-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059813#comment-16059813 ] ASF GitHub Bot commented on FLINK-6389: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/4151 merging. > Upgrade hbase dependency to 1.3.1 > - > > Key: FLINK-6389 > URL: https://issues.apache.org/jira/browse/FLINK-6389 > Project: Flink > Issue Type: Improvement > Components: Batch Connectors and Input/Output Formats >Reporter: Ted Yu >Assignee: mingleizhang >Priority: Minor > > hbase 1.3.1 has been released. > It fixes compatibility issue in 1.3.0 release, among other bug fixes. > We should upgrade to hbase 1.3.1 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink issue #4151: [FLINK-6389] [connector] Upgrade hbase dependency to 1.3....
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4151 merging. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4034: [FLINK-6786] [metrics] Deduplicate QueryScopeIntoTest
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4034 merging. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6786) Remove duplicate QueryScopeInfoTest
[
https://issues.apache.org/jira/browse/FLINK-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059811#comment-16059811
]
ASF GitHub Bot commented on FLINK-6786:
---
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/4034
merging.
> Remove duplicate QueryScopeInfoTest
> ---
>
> Key: FLINK-6786
> URL: https://issues.apache.org/jira/browse/FLINK-6786
> Project: Flink
> Issue Type: Improvement
> Components: Metrics, Tests
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.4.0
>
>
> The {{QueryScopeInfoTest}} exists twice in {{runtime/metrics}}, under
> {{groups/}} and {{dump/}}.
> These should be merged together.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6956) Table examples broken
[
https://issues.apache.org/jira/browse/FLINK-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059806#comment-16059806
]
ASF GitHub Bot commented on FLINK-6956:
---
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/4148
Will fix the typo while merging.
> Table examples broken
> -
>
> Key: FLINK-6956
> URL: https://issues.apache.org/jira/browse/FLINK-6956
> Project: Flink
> Issue Type: Bug
> Components: Examples, Table API & SQL
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.4.0
>
>
> When running the examples you get this exception:
> {code}
> Caused by: org.apache.flink.table.api.TableException: Class 'class
> org.apache.flink.table.examples.java.WordCountSQL$WC' described in type
> information
> 'GenericType' must be
> static and globally accessible.
> at
> org.apache.flink.table.api.TableException$.apply(exceptions.scala:53)
> at
> org.apache.flink.table.api.TableEnvironment$.validateType(TableEnvironment.scala:936)
> at
> org.apache.flink.table.api.TableEnvironment.getFieldInfo(TableEnvironment.scala:616)
> at
> org.apache.flink.table.api.BatchTableEnvironment.registerDataSetInternal(BatchTableEnvironment.scala:248)
> at
> org.apache.flink.table.api.java.BatchTableEnvironment.registerDataSet(BatchTableEnvironment.scala:129)
> at
> org.apache.flink.table.examples.java.WordCountSQL.main(WordCountSQL.java:53)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:525)
> ... 13 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #4148: [FLINK-6956] Make table example POJO classes public
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4148 Will fix the typo while merging. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6967) Fully separate batch and storm examples
[ https://issues.apache.org/jira/browse/FLINK-6967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059803#comment-16059803 ] ASF GitHub Bot commented on FLINK-6967: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/4159 merging. > Fully separate batch and storm examples > --- > > Key: FLINK-6967 > URL: https://issues.apache.org/jira/browse/FLINK-6967 > Project: Flink > Issue Type: Improvement > Components: Examples, Storm Compatibility >Affects Versions: 1.4.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Minor > Fix For: 1.4.0 > > > Like the streaming examples (see FLINK-6863) the storm examples have a > dependency on the batch examples, exclusively for the WordCount example data. > I propose to duplicate the test data again for the storm examples. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink issue #4159: [FLINK-6967] Remove batch-examples dependency from storm ...
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4159 merging. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6988) Add Apache Kafka 0.11 connector
[ https://issues.apache.org/jira/browse/FLINK-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059774#comment-16059774 ] Tzu-Li (Gordon) Tai commented on FLINK-6988: We did have an offline discussion with [~rmetzger] for dropping 08 support. Having the need to maintain common features across all Kafka versions is certainly starting to feel a bit awkward for Kafka 08 compared to 09+, due to missing features in the old consumer / producer APIs. I think we perhaps need to do a poll on the mailing list to get a feel of how many of our users are still using Kafka 08. > Add Apache Kafka 0.11 connector > --- > > Key: FLINK-6988 > URL: https://issues.apache.org/jira/browse/FLINK-6988 > Project: Flink > Issue Type: Improvement > Components: Kafka Connector >Affects Versions: 1.3.1 >Reporter: Piotr Nowojski >Assignee: Piotr Nowojski > > Kafka 0.11 (it will be released very soon) add supports for transactions. > Thanks to that, Flink might be able to implement Kafka sink supporting > "exactly-once" semantic. API changes and whole transactions support is > described in > [KIP-98|https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging]. > The goal is to mimic implementation of existing BucketingSink. New > FlinkKafkaProducer011 would > * upon creation begin transaction, store transaction identifiers into the > state and would write all incoming data to an output Kafka topic using that > transaction > * on `snapshotState` call, it would flush the data and write in state > information that current transaction is pending to be committed > * on `notifyCheckpointComplete` we would commit this pending transaction > * in case of crash between `snapshotState` and `notifyCheckpointComplete` we > either abort this pending transaction (if not every participant successfully > saved the snapshot) or restore and commit it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6989) Refactor examples with Output interface
Greg Hogan created FLINK-6989: - Summary: Refactor examples with Output interface Key: FLINK-6989 URL: https://issues.apache.org/jira/browse/FLINK-6989 Project: Flink Issue Type: Sub-task Components: Gelly Affects Versions: 1.4.0 Reporter: Greg Hogan Assignee: Greg Hogan Fix For: 1.4.0 The current organization of the Gelly examples retains full flexibility by handling the Graph input to the algorithm Driver and having the Driver overload interfaces for the various output types. The outputs must be made independent in order to support Transforms which are applied between the Driver and Output (and also between the Input and Driver). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-6988) Add Apache Kafka 0.11 connector
[ https://issues.apache.org/jira/browse/FLINK-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059735#comment-16059735 ] Chesnay Schepler commented on FLINK-6988: - This would increase our number of supported kafka version to 4. What's the plan for removing kafka 0.8? > Add Apache Kafka 0.11 connector > --- > > Key: FLINK-6988 > URL: https://issues.apache.org/jira/browse/FLINK-6988 > Project: Flink > Issue Type: Improvement > Components: Kafka Connector >Affects Versions: 1.3.1 >Reporter: Piotr Nowojski >Assignee: Piotr Nowojski > > Kafka 0.11 (it will be released very soon) add supports for transactions. > Thanks to that, Flink might be able to implement Kafka sink supporting > "exactly-once" semantic. API changes and whole transactions support is > described in > [KIP-98|https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging]. > The goal is to mimic implementation of existing BucketingSink. New > FlinkKafkaProducer011 would > * upon creation begin transaction, store transaction identifiers into the > state and would write all incoming data to an output Kafka topic using that > transaction > * on `snapshotState` call, it would flush the data and write in state > information that current transaction is pending to be committed > * on `notifyCheckpointComplete` we would commit this pending transaction > * in case of crash between `snapshotState` and `notifyCheckpointComplete` we > either abort this pending transaction (if not every participant successfully > saved the snapshot) or restore and commit it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (FLINK-6988) Add Apache Kafka 0.11 connector
[ https://issues.apache.org/jira/browse/FLINK-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059699#comment-16059699 ] Piotr Nowojski edited comment on FLINK-6988 at 6/22/17 5:15 PM: Unfortunately KafkaProducer's API is very limited. Especially it doesn't allow to implement two phase commit protocol like it is done in BucketingSink, because it doesn't allow for neither resuming nor committing transactions from different workers after crash (last bullet point above). This is because every time user calls `Producer::initTransactions()`, all pending (not committed) transactions are being automatically aborted by Kafka Server. Calling `Producer::initTransactions()` is neccessary to obtain `producerId` and `epoch` values from the Kafka server, which are crucial for manipulating transactions. Fortunately there is a walk around for this issue. It seems like Kafka's REST API is more flexible and we should be possible to resume transactions. Every time we begin transaction we can store `producerId` and `epoch` on the state. In case we need to commit pending transaction on another worker (after crash), instead of calling `KafkaProducer::initTransactions()` we can restore `producerId` and `epoch` from the state and commit this pending transaction using those restored values. "Hacky" part is that `producerId` and `epoch` values are hidden behind private fields in package private classes. That means we can not overload `KafkaProducer` to obtain or set them. That leaves as with two options. We either reimplement KafkaProducer using Kafka's REST API (we could copy/paste most of their code) or we use JVM reflection to manually manipulate official KafkaProducer class. was (Author: pnowojski): Unfortunately KafkaProducer's API is very limited. Especially it doesn't allow to implement two phase commit protocol like it is done in BucketingSink, because it doesn't allow for neither resuming nor committing transactions from different workers after crash (last bullet point above). This is because every time user calls `Producer::initTransactions()`, all pending (not committed) transactions are being automatically aborted by Kafka Server. Calling `Producer::initTransactions()` is neccessary to obtain `producerId` and `epoch` values from the Kafka server, which are crucial for manipulating transactions. Fortunately there is a walk around this issue. It seems like Kafka's REST API is more flexible and we should be possible to resume transactions. Every time we begin transaction we can store `producerId` and `epoch` on the state. In case we need to commit pending transaction on another worker (after crash), instead of calling `KafkaProducer::initTransactions()` we can restore `producerId` and `epoch` from the state and commit this pending transaction using those restored values. "Hacky" part is that `producerId` and `epoch` values are hidden behind private fields in package private classes. That means we can not overload `KafkaProducer` to obtain or set them. That leaves as with two options. We either reimplement KafkaProducer using Kafka's REST API (we could copy/paste most of their code) or we use JVM reflection to manually manipulate official KafkaProducer class. > Add Apache Kafka 0.11 connector > --- > > Key: FLINK-6988 > URL: https://issues.apache.org/jira/browse/FLINK-6988 > Project: Flink > Issue Type: Improvement > Components: Kafka Connector >Affects Versions: 1.3.1 >Reporter: Piotr Nowojski >Assignee: Piotr Nowojski > > Kafka 0.11 (it will be released very soon) add supports for transactions. > Thanks to that, Flink might be able to implement Kafka sink supporting > "exactly-once" semantic. API changes and whole transactions support is > described in > [KIP-98|https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging]. > The goal is to mimic implementation of existing BucketingSink. New > FlinkKafkaProducer011 would > * upon creation begin transaction, store transaction identifiers into the > state and would write all incoming data to an output Kafka topic using that > transaction > * on `snapshotState` call, it would flush the data and write in state > information that current transaction is pending to be committed > * on `notifyCheckpointComplete` we would commit this pending transaction > * in case of crash between `snapshotState` and `notifyCheckpointComplete` we > either abort this pending transaction (if not every participant successfully > saved the snapshot) or restore and commit it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLINK-6988) Add Apache Kafka 0.11 connector
[ https://issues.apache.org/jira/browse/FLINK-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski updated FLINK-6988: -- Description: Kafka 0.11 (it will be released very soon) add supports for transactions. Thanks to that, Flink might be able to implement Kafka sink supporting "exactly-once" semantic. API changes and whole transactions support is described in [KIP-98|https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging]. The goal is to mimic implementation of existing BucketingSink. New FlinkKafkaProducer011 would * upon creation begin transaction, store transaction identifiers into the state and would write all incoming data to an output Kafka topic using that transaction * on `snapshotState` call, it would flush the data and write in state information that current transaction is pending to be committed * on `notifyCheckpointComplete` we would commit this pending transaction * in case of crash between `snapshotState` and `notifyCheckpointComplete` we either abort this pending transaction (if not every participant successfully saved the snapshot) or restore and commit it. was: Kafka 0.11 (it will be released very soon) add supports for transactions. Thanks to that, Flink might be able to implement Kafka sink supporting "exactly-once" semantic. API changes and whole transactions support is described in [KIP-98|https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging]. The goal is to mimic implementation of existing BucketingSink. New FlinkKafkaProducer011 would * upon creation begin transaction, store transaction identifiers into the state and would write all incoming data to output topic using that transaction * on `snapshotState` call, it would flush the data and write in state information that current transaction is pending to be committed * on `notifyCheckpointComplete` we would commit this pending transaction * in case of crash between `snapshotState` and `notifyCheckpointComplete` we either abort this pending transaction (if not every participant successfully saved the snapshot) or restore and commit it. > Add Apache Kafka 0.11 connector > --- > > Key: FLINK-6988 > URL: https://issues.apache.org/jira/browse/FLINK-6988 > Project: Flink > Issue Type: Improvement > Components: Kafka Connector >Affects Versions: 1.3.1 >Reporter: Piotr Nowojski >Assignee: Piotr Nowojski > > Kafka 0.11 (it will be released very soon) add supports for transactions. > Thanks to that, Flink might be able to implement Kafka sink supporting > "exactly-once" semantic. API changes and whole transactions support is > described in > [KIP-98|https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging]. > The goal is to mimic implementation of existing BucketingSink. New > FlinkKafkaProducer011 would > * upon creation begin transaction, store transaction identifiers into the > state and would write all incoming data to an output Kafka topic using that > transaction > * on `snapshotState` call, it would flush the data and write in state > information that current transaction is pending to be committed > * on `notifyCheckpointComplete` we would commit this pending transaction > * in case of crash between `snapshotState` and `notifyCheckpointComplete` we > either abort this pending transaction (if not every participant successfully > saved the snapshot) or restore and commit it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-6988) Add Apache Kafka 0.11 connector
[ https://issues.apache.org/jira/browse/FLINK-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059699#comment-16059699 ] Piotr Nowojski commented on FLINK-6988: --- Unfortunately KafkaProducer's API is very limited. Especially it doesn't allow to implement two phase commit protocol like it is done in BucketingSink, because it doesn't allow for neither resuming nor committing transactions from different workers after crash (last bullet point above). This is because every time user calls `Producer::initTransactions()`, all pending (not committed) transactions are being automatically aborted by Kafka Server. Calling `Producer::initTransactions()` is neccessary to obtain `producerId` and `epoch` values from the Kafka server, which are crucial for manipulating transactions. Fortunately there is a walk around this issue. It seems like Kafka's REST API is more flexible and we should be possible to resume transactions. Every time we begin transaction we can store `producerId` and `epoch` on the state. In case we need to commit pending transaction on another worker (after crash), instead of calling `KafkaProducer::initTransactions()` we can restore `producerId` and `epoch` from the state and commit this pending transaction using those restored values. "Hacky" part is that `producerId` and `epoch` values are hidden behind private fields in package private classes. That means we can not overload `KafkaProducer` to obtain or set them. That leaves as with two options. We either reimplement KafkaProducer using Kafka's REST API (we could copy/paste most of their code) or we use JVM reflection to manually manipulate official KafkaProducer class. > Add Apache Kafka 0.11 connector > --- > > Key: FLINK-6988 > URL: https://issues.apache.org/jira/browse/FLINK-6988 > Project: Flink > Issue Type: Improvement > Components: Kafka Connector >Affects Versions: 1.3.1 >Reporter: Piotr Nowojski >Assignee: Piotr Nowojski > > Kafka 0.11 (it will be released very soon) add supports for transactions. > Thanks to that, Flink might be able to implement Kafka sink supporting > "exactly-once" semantic. API changes and whole transactions support is > described in > [KIP-98|https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging]. > The goal is to mimic implementation of existing BucketingSink. New > FlinkKafkaProducer011 would > * upon creation begin transaction, store transaction identifiers into the > state and would write all incoming data to output topic using that transaction > * on `snapshotState` call, it would flush the data and write in state > information that current transaction is pending to be committed > * on `notifyCheckpointComplete` we would commit this pending transaction > * in case of crash between `snapshotState` and `notifyCheckpointComplete` we > either abort this pending transaction (if not every participant successfully > saved the snapshot) or restore and commit it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLINK-6988) Add Apache Kafka 0.11 connector
[ https://issues.apache.org/jira/browse/FLINK-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski updated FLINK-6988: -- Description: Kafka 0.11 (it will be released very soon) add supports for transactions. Thanks to that, Flink might be able to implement Kafka sink supporting "exactly-once" semantic. API changes and whole transactions support is described in [KIP-98|https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging]. The goal is to mimic implementation of existing BucketingSink. New FlinkKafkaProducer011 would * upon creation begin transaction, store transaction identifiers into the state and would write all incoming data to output topic using that transaction * on `snapshotState` call, it would flush the data and write in state information that current transaction is pending to be committed * on `notifyCheckpointComplete` we would commit this pending transaction * in case of crash between `snapshotState` and `notifyCheckpointComplete` we either abort this pending transaction (if not every participant successfully saved the snapshot) or restore and commit it. was: Kafka 0.11 (it will be released very soon) add supports for transactions. Thanks to that, Flink might be able to implement Kafka sink supporting "exactly-once" semantic. API changes and whole transactions support is described in [KIP-98|https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging]. The goal is to mimic implementation of existing BucketingSink. New KafkaProducer011 would * upon creation begin transaction, store transaction identifiers into the state and would write all incoming data to output topic using that transaction * on `snapshotState` call, it would flush the data and write in state information that current transaction is pending to be committed * on `notifyCheckpointComplete` we would commit this pending transaction * in case of crash between `snapshotState` and `notifyCheckpointComplete` we either abort this pending transaction (if not every participant successfully saved the snapshot) or restore and commit it. > Add Apache Kafka 0.11 connector > --- > > Key: FLINK-6988 > URL: https://issues.apache.org/jira/browse/FLINK-6988 > Project: Flink > Issue Type: Improvement > Components: Kafka Connector >Affects Versions: 1.3.1 >Reporter: Piotr Nowojski >Assignee: Piotr Nowojski > > Kafka 0.11 (it will be released very soon) add supports for transactions. > Thanks to that, Flink might be able to implement Kafka sink supporting > "exactly-once" semantic. API changes and whole transactions support is > described in > [KIP-98|https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging]. > The goal is to mimic implementation of existing BucketingSink. New > FlinkKafkaProducer011 would > * upon creation begin transaction, store transaction identifiers into the > state and would write all incoming data to output topic using that transaction > * on `snapshotState` call, it would flush the data and write in state > information that current transaction is pending to be committed > * on `notifyCheckpointComplete` we would commit this pending transaction > * in case of crash between `snapshotState` and `notifyCheckpointComplete` we > either abort this pending transaction (if not every participant successfully > saved the snapshot) or restore and commit it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-6988) Add Apache Kafka 0.11 connector
Piotr Nowojski created FLINK-6988: - Summary: Add Apache Kafka 0.11 connector Key: FLINK-6988 URL: https://issues.apache.org/jira/browse/FLINK-6988 Project: Flink Issue Type: Improvement Components: Kafka Connector Affects Versions: 1.3.1 Reporter: Piotr Nowojski Assignee: Piotr Nowojski Kafka 0.11 (it will be released very soon) add supports for transactions. Thanks to that, Flink might be able to implement Kafka sink supporting "exactly-once" semantic. API changes and whole transactions support is described in [KIP-98|https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging]. The goal is to mimic implementation of existing BucketingSink. New KafkaProducer011 would * upon creation begin transaction, store transaction identifiers into the state and would write all incoming data to output topic using that transaction * on `snapshotState` call, it would flush the data and write in state information that current transaction is pending to be committed * on `notifyCheckpointComplete` we would commit this pending transaction * in case of crash between `snapshotState` and `notifyCheckpointComplete` we either abort this pending transaction (if not every participant successfully saved the snapshot) or restore and commit it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLINK-6493) Ineffective null check in RegisteredOperatorBackendStateMetaInfo#equals()
[
https://issues.apache.org/jira/browse/FLINK-6493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated FLINK-6493:
--
Description:
{code}
&& ((partitionStateSerializer == null && ((Snapshot)
obj).getPartitionStateSerializer() == null)
|| partitionStateSerializer.equals(((Snapshot)
obj).getPartitionStateSerializer()))
&& ((partitionStateSerializerConfigSnapshot == null && ((Snapshot)
obj).getPartitionStateSerializerConfigSnapshot() == null)
|| partitionStateSerializerConfigSnapshot.equals(((Snapshot)
obj).getPartitionStateSerializerConfigSnapshot()));
{code}
The null check for partitionStateSerializer /
partitionStateSerializerConfigSnapshot is in combination with another clause.
This may lead to NPE in the partitionStateSerializer.equals() call.
was:
{code}
&& ((partitionStateSerializer == null && ((Snapshot)
obj).getPartitionStateSerializer() == null)
|| partitionStateSerializer.equals(((Snapshot)
obj).getPartitionStateSerializer()))
&& ((partitionStateSerializerConfigSnapshot == null && ((Snapshot)
obj).getPartitionStateSerializerConfigSnapshot() == null)
|| partitionStateSerializerConfigSnapshot.equals(((Snapshot)
obj).getPartitionStateSerializerConfigSnapshot()));
{code}
The null check for partitionStateSerializer /
partitionStateSerializerConfigSnapshot is in combination with another clause.
This may lead to NPE in the partitionStateSerializer.equals() call.
> Ineffective null check in RegisteredOperatorBackendStateMetaInfo#equals()
> -
>
> Key: FLINK-6493
> URL: https://issues.apache.org/jira/browse/FLINK-6493
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Reporter: Ted Yu
>Priority: Minor
>
> {code}
> && ((partitionStateSerializer == null && ((Snapshot)
> obj).getPartitionStateSerializer() == null)
> || partitionStateSerializer.equals(((Snapshot)
> obj).getPartitionStateSerializer()))
> && ((partitionStateSerializerConfigSnapshot == null && ((Snapshot)
> obj).getPartitionStateSerializerConfigSnapshot() == null)
> || partitionStateSerializerConfigSnapshot.equals(((Snapshot)
> obj).getPartitionStateSerializerConfigSnapshot()));
> {code}
> The null check for partitionStateSerializer /
> partitionStateSerializerConfigSnapshot is in combination with another clause.
> This may lead to NPE in the partitionStateSerializer.equals() call.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLINK-6838) RescalingITCase fails in master branch
[
https://issues.apache.org/jira/browse/FLINK-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated FLINK-6838:
--
Description:
{code}
Tests in error:
RescalingITCase.testSavepointRescalingInKeyedState[1] » JobExecution Job
execu...
RescalingITCase.testSavepointRescalingWithKeyedAndNonPartitionedState[1] »
JobExecution
{code}
Both failed with similar cause:
{code}
testSavepointRescalingInKeyedState[1](org.apache.flink.test.checkpointing.RescalingITCase)
Time elapsed: 4.813 sec <<< ERROR!
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:933)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:876)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:876)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.apache.flink.streaming.runtime.tasks.AsynchronousException:
java.lang.Exception: Could not materialize checkpoint 4 for operator Flat Map
-> Sink: Unnamed (1/2).
at
org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:967)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Exception: Could not materialize checkpoint 4 for operator
Flat Map -> Sink: Unnamed (1/2).
at
org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:967)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Cannot
register Closeable, registry is already closed. Closing argument.
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:43)
at
org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:894)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Cannot register Closeable, registry is already
closed. Closing argument.
at
org.apache.flink.util.AbstractCloseableRegistry.registerClosable(AbstractCloseableRegistry.java:66)
at
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBFullSnapshotOperation.openCheckpointStream(RocksDBKeyedStateBackend.java:495)
at
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$3.openIOHandle(RocksDBKeyedStateBackend.java:394)
at
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$3.openIOHandle(RocksDBKeyedStateBackend.java:390)
at
org.apache.flink.runtime.io.async.AbstractAsyncIOCallable.call(AbstractAsyncIOCallable.java:67)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:40)
at
org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:894)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
{code}
was:
{code}
Tests in error:
RescalingITCase.testSavepointRes
[jira] [Commented] (FLINK-6357) ParameterTool get unrequested parameters
[
https://issues.apache.org/jira/browse/FLINK-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059683#comment-16059683
]
ASF GitHub Bot commented on FLINK-6357:
---
GitHub user greghogan opened a pull request:
https://github.com/apache/flink/pull/4169
[FLINK-6357] [java] ParameterTool get unrequested parameters
Adds ParameterTool#getUnrequestedParameters returning a Set of
parameter arguments names not yet requested by ParameterTool#has or any of the
ParameterTool#get methods.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/greghogan/flink
6357_parametertool_get_unrequested_parameters
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4169.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4169
commit c981ad4ffe828396d5c038465ace17637b585f8f
Author: Greg Hogan
Date: 2017-06-21T12:18:55Z
[FLINK-6357] [java] ParameterTool get unrequested parameters
Adds ParameterTool#getUnrequestedParameters returning a Set of
parameter arguments names not yet requested by ParameterTool#has or any
of the ParameterTool#get methods.
> ParameterTool get unrequested parameters
>
>
> Key: FLINK-6357
> URL: https://issues.apache.org/jira/browse/FLINK-6357
> Project: Flink
> Issue Type: Improvement
> Components: Java API
>Affects Versions: 1.3.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Minor
>
> The Gelly examples use {{ParameterTool}} to parse required and optional
> parameters. In the latter case we should detect if a user mistypes a
> parameter name. I would like to add a {{Set
> getUnrequestedParameters()}} method returning parameter names not requested
> by {{has}} or any of the {{get}} methods.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink pull request #4169: [FLINK-6357] [java] ParameterTool get unrequested ...
GitHub user greghogan opened a pull request: https://github.com/apache/flink/pull/4169 [FLINK-6357] [java] ParameterTool get unrequested parameters Adds ParameterTool#getUnrequestedParameters returning a Set of parameter arguments names not yet requested by ParameterTool#has or any of the ParameterTool#get methods. You can merge this pull request into a Git repository by running: $ git pull https://github.com/greghogan/flink 6357_parametertool_get_unrequested_parameters Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/4169.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4169 commit c981ad4ffe828396d5c038465ace17637b585f8f Author: Greg Hogan Date: 2017-06-21T12:18:55Z [FLINK-6357] [java] ParameterTool get unrequested parameters Adds ParameterTool#getUnrequestedParameters returning a Set of parameter arguments names not yet requested by ParameterTool#has or any of the ParameterTool#get methods. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6987) TextInputFormatTest fails when run in path containing spaces
[
https://issues.apache.org/jira/browse/FLINK-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059669#comment-16059669
]
ASF GitHub Bot commented on FLINK-6987:
---
Github user zhangminglei commented on the issue:
https://github.com/apache/flink/pull/4168
Oops. I might be wrong. I will check again.
> TextInputFormatTest fails when run in path containing spaces
>
>
> Key: FLINK-6987
> URL: https://issues.apache.org/jira/browse/FLINK-6987
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.3.1
>Reporter: Timo Walther
>Assignee: mingleizhang
>
> The test {{TextInputFormatTest.testNestedFileRead}} fails if the path
> contains spaces.
> Reason: "Test erroneous"
> I was building Flink on MacOS 10.12.5 and the folder was called "flink-1.3.1
> 2".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #4168: [FLINK-6987] Fix erroneous when path containing spaces
Github user zhangminglei commented on the issue: https://github.com/apache/flink/pull/4168 Oops. I might be wrong. I will check again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6987) TextInputFormatTest fails when run in path containing spaces
[
https://issues.apache.org/jira/browse/FLINK-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059657#comment-16059657
]
ASF GitHub Bot commented on FLINK-6987:
---
Github user zhangminglei commented on the issue:
https://github.com/apache/flink/pull/4168
I dont think it has an underlying issue as below line of code throw an
error, and it is not relavant to the ```TextInputFormat``` class . ```tmpDir```
has spaces and then trigger that problem. If my understanding is wrong, please
help me out.
```File tempFile = File.createTempFile("TextInputFormatTest", ".tmp",
tmpDir);```
> TextInputFormatTest fails when run in path containing spaces
>
>
> Key: FLINK-6987
> URL: https://issues.apache.org/jira/browse/FLINK-6987
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.3.1
>Reporter: Timo Walther
>Assignee: mingleizhang
>
> The test {{TextInputFormatTest.testNestedFileRead}} fails if the path
> contains spaces.
> Reason: "Test erroneous"
> I was building Flink on MacOS 10.12.5 and the folder was called "flink-1.3.1
> 2".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #4168: [FLINK-6987] Fix erroneous when path containing spaces
Github user zhangminglei commented on the issue:
https://github.com/apache/flink/pull/4168
I dont think it has an underlying issue as below line of code throw an
error, and it is not relavant to the ```TextInputFormat``` class . ```tmpDir```
has spaces and then trigger that problem. If my understanding is wrong, please
help me out.
```File tempFile = File.createTempFile("TextInputFormatTest", ".tmp",
tmpDir);```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Assigned] (FLINK-6969) Add support for deferred computation for group window aggregates
[
https://issues.apache.org/jira/browse/FLINK-6969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sunjincheng reassigned FLINK-6969:
--
Assignee: sunjincheng
> Add support for deferred computation for group window aggregates
>
>
> Key: FLINK-6969
> URL: https://issues.apache.org/jira/browse/FLINK-6969
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Reporter: Fabian Hueske
>Assignee: sunjincheng
>
> Deferred computation is a strategy to deal with late arriving data and avoid
> updates of previous results. Instead of computing a result as soon as it is
> possible (i.e., when a corresponding watermark was received), deferred
> computation adds a configurable amount of slack time in which late data is
> accepted before the result is compute. For example, instead of computing a
> tumbling window of 1 hour at each full hour, we can add a deferred
> computation interval of 15 minute to compute the result quarter past each
> full hour.
> This approach adds latency but can reduce the number of update esp. in use
> cases where the user cannot influence the generation of watermarks. It is
> also useful if the data is emitted to a system that cannot update result
> (files or Kafka). The deferred computation interval should be configured via
> the {{QueryConfig}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #4167: Test PR, please ignore
Github user greghogan commented on the issue: https://github.com/apache/flink/pull/4167 May also be able to revert `-Dflink.forkCountTestPackage=1` to 2 in `tools/travis_mvn_watchdog.sh` with memory increased from 4 GB to 7.5 GB (https://docs.travis-ci.com/user/ci-environment/). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4168: [FLINK-6987] Fix erroneous when path containing spaces
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4168 I doubt this fixes the underlying issue, it just masks it by storing the input files in paths containing spaces. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6987) TextInputFormatTest fails when run in path containing spaces
[
https://issues.apache.org/jira/browse/FLINK-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059621#comment-16059621
]
ASF GitHub Bot commented on FLINK-6987:
---
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/4168
I doubt this fixes the underlying issue, it just masks it by storing the
input files in paths containing spaces.
> TextInputFormatTest fails when run in path containing spaces
>
>
> Key: FLINK-6987
> URL: https://issues.apache.org/jira/browse/FLINK-6987
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.3.1
>Reporter: Timo Walther
>Assignee: mingleizhang
>
> The test {{TextInputFormatTest.testNestedFileRead}} fails if the path
> contains spaces.
> Reason: "Test erroneous"
> I was building Flink on MacOS 10.12.5 and the folder was called "flink-1.3.1
> 2".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6987) TextInputFormatTest fails when run in path containing spaces
[
https://issues.apache.org/jira/browse/FLINK-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059606#comment-16059606
]
ASF GitHub Bot commented on FLINK-6987:
---
Github user zhangminglei commented on the issue:
https://github.com/apache/flink/pull/4168
Hi @twalthr , you can test it again. I think it works fine now.
> TextInputFormatTest fails when run in path containing spaces
>
>
> Key: FLINK-6987
> URL: https://issues.apache.org/jira/browse/FLINK-6987
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.3.1
>Reporter: Timo Walther
>Assignee: mingleizhang
>
> The test {{TextInputFormatTest.testNestedFileRead}} fails if the path
> contains spaces.
> Reason: "Test erroneous"
> I was building Flink on MacOS 10.12.5 and the folder was called "flink-1.3.1
> 2".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #4168: [FLINK-6987] Fix erroneous when path containing spaces
Github user zhangminglei commented on the issue: https://github.com/apache/flink/pull/4168 Hi @twalthr , you can test it again. I think it works fine now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6987) TextInputFormatTest fails when run in path containing spaces
[
https://issues.apache.org/jira/browse/FLINK-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059598#comment-16059598
]
ASF GitHub Bot commented on FLINK-6987:
---
GitHub user zhangminglei opened a pull request:
https://github.com/apache/flink/pull/4168
[FLINK-6987] Fix erroneous when path containing spaces
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhangminglei/flink flink-6987
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4168.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4168
commit b146f8e085f45b801cb8411962c6e3cc95f206c9
Author: zhangminglei
Date: 2017-06-22T16:05:44Z
[FLINK-6987] Fix erroneous when path containing spaces
> TextInputFormatTest fails when run in path containing spaces
>
>
> Key: FLINK-6987
> URL: https://issues.apache.org/jira/browse/FLINK-6987
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.3.1
>Reporter: Timo Walther
>Assignee: mingleizhang
>
> The test {{TextInputFormatTest.testNestedFileRead}} fails if the path
> contains spaces.
> Reason: "Test erroneous"
> I was building Flink on MacOS 10.12.5 and the folder was called "flink-1.3.1
> 2".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink pull request #4168: [FLINK-6987] Fix erroneous when path containing sp...
GitHub user zhangminglei opened a pull request:
https://github.com/apache/flink/pull/4168
[FLINK-6987] Fix erroneous when path containing spaces
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhangminglei/flink flink-6987
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4168.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4168
commit b146f8e085f45b801cb8411962c6e3cc95f206c9
Author: zhangminglei
Date: 2017-06-22T16:05:44Z
[FLINK-6987] Fix erroneous when path containing spaces
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Assigned] (FLINK-6987) TextInputFormatTest fails when run in path containing spaces
[
https://issues.apache.org/jira/browse/FLINK-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mingleizhang reassigned FLINK-6987:
---
Assignee: mingleizhang
> TextInputFormatTest fails when run in path containing spaces
>
>
> Key: FLINK-6987
> URL: https://issues.apache.org/jira/browse/FLINK-6987
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.3.1
>Reporter: Timo Walther
>Assignee: mingleizhang
>
> The test {{TextInputFormatTest.testNestedFileRead}} fails if the path
> contains spaces.
> Reason: "Test erroneous"
> I was building Flink on MacOS 10.12.5 and the folder was called "flink-1.3.1
> 2".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6987) TextInputFormatTest fails when run in path containing spaces
[
https://issues.apache.org/jira/browse/FLINK-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059584#comment-16059584
]
mingleizhang commented on FLINK-6987:
-
I just test on my windows 7. Also happened. I will give a PR to fix this issue
soon enough.
> TextInputFormatTest fails when run in path containing spaces
>
>
> Key: FLINK-6987
> URL: https://issues.apache.org/jira/browse/FLINK-6987
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.3.1
>Reporter: Timo Walther
>
> The test {{TextInputFormatTest.testNestedFileRead}} fails if the path
> contains spaces.
> Reason: "Test erroneous"
> I was building Flink on MacOS 10.12.5 and the folder was called "flink-1.3.1
> 2".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLINK-6750) Table API / SQL Docs: Table Sources & Sinks Page
[
https://issues.apache.org/jira/browse/FLINK-6750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-6750:
--
Fix Version/s: (was: 1.3.1)
1.3.2
> Table API / SQL Docs: Table Sources & Sinks Page
>
>
> Key: FLINK-6750
> URL: https://issues.apache.org/jira/browse/FLINK-6750
> Project: Flink
> Issue Type: Task
> Components: Documentation, Table API & SQL
>Affects Versions: 1.3.0
>Reporter: Fabian Hueske
>Assignee: Timo Walther
> Fix For: 1.4.0, 1.3.2
>
>
> Update and refine the documentation about TableSources and TableSinks.
> There are a few TODOs left in {{./docs/dev/table/sourceSinks.md}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLINK-6954) Flink 1.3 checkpointing failing with KeyedCEPPatternOperator
[
https://issues.apache.org/jira/browse/FLINK-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-6954:
--
Fix Version/s: (was: 1.3.1)
1.3.2
> Flink 1.3 checkpointing failing with KeyedCEPPatternOperator
>
>
> Key: FLINK-6954
> URL: https://issues.apache.org/jira/browse/FLINK-6954
> Project: Flink
> Issue Type: Bug
> Components: CEP, DataStream API, State Backends, Checkpointing
>Affects Versions: 1.3.0
> Environment: yarn, flink 1.3, HDFS
>Reporter: Shashank Agarwal
> Fix For: 1.3.2
>
>
> After upgrading to Flink 1.3 Checkpointing is not working, it's failing again
> and again. Check operator state. I have checked with both Rocks DB state
> backend and FS state backend. Check stack trace.
> {code}
> java.lang.Exception: Could not perform checkpoint 1 for operator
> KeyedCEPPatternOperator -> Map (6/6).
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:550)
> at
> org.apache.flink.streaming.runtime.io.BarrierBuffer.notifyCheckpoint(BarrierBuffer.java:378)
> at
> org.apache.flink.streaming.runtime.io.BarrierBuffer.processBarrier(BarrierBuffer.java:281)
> at
> org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:183)
> at
> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:213)
> at
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:69)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:262)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.Exception: Could not complete snapshot 1 for operator
> KeyedCEPPatternOperator -> Map (6/6).
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:406)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.checkpointStreamOperator(StreamTask.java:1157)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.executeCheckpointing(StreamTask.java:1089)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.checkpointState(StreamTask.java:653)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:589)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:542)
> ... 8 more
> Caused by: java.lang.UnsupportedOperationException
> at
> org.apache.flink.api.scala.typeutils.TraversableSerializer.snapshotConfiguration(TraversableSerializer.scala:155)
> at
> org.apache.flink.api.common.typeutils.CompositeTypeSerializerConfigSnapshot.(CompositeTypeSerializerConfigSnapshot.java:53)
> at
> org.apache.flink.api.scala.typeutils.OptionSerializer$OptionSerializerConfigSnapshot.(OptionSerializer.scala:139)
> at
> org.apache.flink.api.scala.typeutils.OptionSerializer.snapshotConfiguration(OptionSerializer.scala:104)
> at
> org.apache.flink.api.scala.typeutils.OptionSerializer.snapshotConfiguration(OptionSerializer.scala:28)
> at
> org.apache.flink.api.common.typeutils.CompositeTypeSerializerConfigSnapshot.(CompositeTypeSerializerConfigSnapshot.java:53)
> at
> org.apache.flink.api.java.typeutils.runtime.TupleSerializerConfigSnapshot.(TupleSerializerConfigSnapshot.java:45)
> at
> org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase.snapshotConfiguration(TupleSerializerBase.java:132)
> at
> org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase.snapshotConfiguration(TupleSerializerBase.java:39)
> at
> org.apache.flink.api.common.typeutils.CompositeTypeSerializerConfigSnapshot.(CompositeTypeSerializerConfigSnapshot.java:53)
> at
> org.apache.flink.api.common.typeutils.base.CollectionSerializerConfigSnapshot.(CollectionSerializerConfigSnapshot.java:39)
> at
> org.apache.flink.api.common.typeutils.base.ListSerializer.snapshotConfiguration(ListSerializer.java:183)
> at
> org.apache.flink.api.common.typeutils.base.ListSerializer.snapshotConfiguration(ListSerializer.java:47)
> at
> org.apache.flink.api.common.typeutils.CompositeTypeSerializerConfigSnapshot.(CompositeTypeSerializerConfigSnapshot.java:53)
> at
> org.apache.flink.api.common.typeutils.base.MapSerializerConfigSnapshot.(MapSerializerConfigSnapshot.java:38)
> at
> org.apache.flink.runtime.state.HashMapSerializer.snapshotConfiguration(HashMapSerializer.java:210)
> at
> org.apache.flink.runtime.state.RegisteredKeyedBackendStateMetaInfo.snap
[jira] [Updated] (FLINK-6898) Limit size of operator component in metric name
[ https://issues.apache.org/jira/browse/FLINK-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-6898: -- Fix Version/s: (was: 1.3.1) 1.3.2 > Limit size of operator component in metric name > --- > > Key: FLINK-6898 > URL: https://issues.apache.org/jira/browse/FLINK-6898 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0, 1.4.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Critical > Fix For: 1.4.0, 1.3.2 > > > The operator name for some operators (specifically windows) can be very, very > long (250+) characters. > I propose to limit the total space that the operator component can take up in > a metric name to 60 characters. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLINK-6900) Limit size of indiivual components in DropwizardReporter
[ https://issues.apache.org/jira/browse/FLINK-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-6900: -- Fix Version/s: (was: 1.3.1) 1.3.2 > Limit size of indiivual components in DropwizardReporter > > > Key: FLINK-6900 > URL: https://issues.apache.org/jira/browse/FLINK-6900 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0, 1.4.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler > Fix For: 1.4.0, 1.3.2 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink pull request #4167: Test PR, please ignore
GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/4167 Test PR, please ignore I'm using this PR to test changes to the travis setup. You can safely ignore this, and I will close it once I'm done. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/flink mttravis2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/4167.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4167 commit 00a3824d8c2b102b3d4dd3c15b4f2454a926429b Author: zentol Date: 2017-06-22T13:16:35Z switch to trusty image --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6960) Add E() supported in SQL
[ https://issues.apache.org/jira/browse/FLINK-6960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059526#comment-16059526 ] ASF GitHub Bot commented on FLINK-6960: --- Github user sunjincheng121 commented on the issue: https://github.com/apache/flink/pull/4152 Hi @fhueske thanks for your review and I like your suggestion. :) Hi @wuchong I have updated the PR. please take look at the changes. :) > Add E() supported in SQL > > > Key: FLINK-6960 > URL: https://issues.apache.org/jira/browse/FLINK-6960 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL >Reporter: sunjincheng >Assignee: sunjincheng > Labels: starter > > E=Math.E -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink issue #4152: [FLINK-6960][table] Add E supported in SQL.
Github user sunjincheng121 commented on the issue: https://github.com/apache/flink/pull/4152 Hi @fhueske thanks for your review and I like your suggestion. :) Hi @wuchong I have updated the PR. please take look at the changes. :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6925) Add CONCAT/CONCAT_WS supported in SQL
[
https://issues.apache.org/jira/browse/FLINK-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059523#comment-16059523
]
ASF GitHub Bot commented on FLINK-6925:
---
Github user sunjincheng121 commented on the issue:
https://github.com/apache/flink/pull/4138
@shaoxuan-wang @wuchong
> Add CONCAT/CONCAT_WS supported in SQL
> -
>
> Key: FLINK-6925
> URL: https://issues.apache.org/jira/browse/FLINK-6925
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Affects Versions: 1.4.0
>Reporter: sunjincheng
>Assignee: sunjincheng
>
> CONCAT(str1,str2,...)Returns the string that results from concatenating the
> arguments. May have one or more arguments. If all arguments are nonbinary
> strings, the result is a nonbinary string. If the arguments include any
> binary strings, the result is a binary string. A numeric argument is
> converted to its equivalent nonbinary string form.
> CONCAT() returns NULL if any argument is NULL.
> * Syntax:
> CONCAT(str1,str2,...)
> * Arguments
> ** str1,str2,... -
> * Return Types
> string
> * Example:
> CONCAT('F', 'lin', 'k') -> 'Flink'
> CONCAT('M', NULL, 'L') -> NULL
> CONCAT(14.3) -> '14.3'
> * See more:
> ** [MySQL|
> https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_concat]
> CONCAT_WS() stands for Concatenate With Separator and is a special form of
> CONCAT(). The first argument is the separator for the rest of the arguments.
> The separator is added between the strings to be concatenated. The separator
> can be a string, as can the rest of the arguments. If the separator is NULL,
> the result is NULL.
> * Syntax:
> CONCAT_WS(separator,str1,str2,...)
> * Arguments
> ** separator -
> ** str1,str2,... -
> * Return Types
> string
> * Example:
> CONCAT_WS(',','First name','Second name','Last Name') -> 'First name,Second
> name,Last Name'
> * See more:
> ** [MySQL|
> https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_concat-ws]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #4138: [FLINK-6925][table]Add CONCAT/CONCAT_WS supported in SQL
Github user sunjincheng121 commented on the issue: https://github.com/apache/flink/pull/4138 @shaoxuan-wang @wuchong --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4587) Yet another java.lang.NoSuchFieldError: INSTANCE
[
https://issues.apache.org/jira/browse/FLINK-4587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059508#comment-16059508
]
Razvan commented on FLINK-4587:
---
@[~RenkaiGe] I have the same issue. Using Flink 1.2 and created an application
to call some REST endpoints using HttpClient. Does this mean I need to build
Flink?
> Yet another java.lang.NoSuchFieldError: INSTANCE
>
>
> Key: FLINK-4587
> URL: https://issues.apache.org/jira/browse/FLINK-4587
> Project: Flink
> Issue Type: Bug
> Components: Build System, Local Runtime
>Affects Versions: 1.2.0
> Environment: Latest SNAPSHOT
>Reporter: Renkai Ge
> Attachments: diff in mvn clean package.png, flink-explore-src.zip
>
>
> For some reason I need to use org.apache.httpcomponents:httpasyncclient:4.1.2
> in flink.
> The source file is:
> {code}
> import org.apache.flink.streaming.api.scala._
> import org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFactory
> /**
> * Created by renkai on 16/9/7.
> */
> object Main {
> def main(args: Array[String]): Unit = {
> val instance = ManagedNHttpClientConnectionFactory.INSTANCE
> println("instance = " + instance)
> val env = StreamExecutionEnvironment.getExecutionEnvironment
> val stream = env.fromCollection(1 to 100)
> val result = stream.map { x =>
> x * 2
> }
> result.print()
> env.execute("xixi")
> }
> }
> {code}
> and
> {code}
> name := "flink-explore"
> version := "1.0"
> scalaVersion := "2.11.8"
> crossPaths := false
> libraryDependencies ++= Seq(
> "org.apache.flink" %% "flink-scala" % "1.2-SNAPSHOT"
> exclude("com.google.code.findbugs", "jsr305"),
> "org.apache.flink" %% "flink-connector-kafka-0.8" % "1.2-SNAPSHOT"
> exclude("com.google.code.findbugs", "jsr305"),
> "org.apache.flink" %% "flink-streaming-scala" % "1.2-SNAPSHOT"
> exclude("com.google.code.findbugs", "jsr305"),
> "org.apache.flink" %% "flink-clients" % "1.2-SNAPSHOT"
> exclude("com.google.code.findbugs", "jsr305"),
> "org.apache.httpcomponents" % "httpasyncclient" % "4.1.2"
> )
> {code}
> I use `sbt assembly` to get a fat jar.
> If I run the command
> {code}
> java -cp flink-explore-assembly-1.0.jar Main
> {code}
> I get the result
> {code}
> instance =
> org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFactory@4909b8da
> log4j:WARN No appenders could be found for logger
> (org.apache.flink.api.scala.ClosureCleaner$).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> Connected to JobManager at Actor[akka://flink/user/jobmanager_1#-1177584915]
> 09/07/2016 12:05:26 Job execution switched to status RUNNING.
> 09/07/2016 12:05:26 Source: Collection Source(1/1) switched to SCHEDULED
> 09/07/2016 12:05:26 Source: Collection Source(1/1) switched to DEPLOYING
> ...
> 09/07/2016 12:05:26 Map -> Sink: Unnamed(20/24) switched to RUNNING
> 09/07/2016 12:05:26 Map -> Sink: Unnamed(19/24) switched to RUNNING
> 15> 30
> 20> 184
> ...
> 19> 182
> 1> 194
> 8> 160
> 09/07/2016 12:05:26 Source: Collection Source(1/1) switched to FINISHED
> ...
> 09/07/2016 12:05:26 Map -> Sink: Unnamed(1/24) switched to FINISHED
> 09/07/2016 12:05:26 Job execution switched to status FINISHED.
> {code}
> Nothing special.
> But if I run the jar by
> {code}
> ./bin/flink run shop-monitor-flink-assembly-1.0.jar
> {code}
> I will get an error
> {code}
> $ ./bin/flink run flink-explore-assembly-1.0.jar
> Cluster configuration: Standalone cluster with JobManager at /127.0.0.1:6123
> Using address 127.0.0.1:6123 to connect to JobManager.
> JobManager web interface address http://127.0.0.1:8081
> Starting execution of program
>
> The program finished with the following exception:
> java.lang.NoSuchFieldError: INSTANCE
> at
> org.apache.http.impl.nio.codecs.DefaultHttpRequestWriterFactory.(DefaultHttpRequestWriterFactory.java:53)
> at
> org.apache.http.impl.nio.codecs.DefaultHttpRequestWriterFactory.(DefaultHttpRequestWriterFactory.java:57)
> at
> org.apache.http.impl.nio.codecs.DefaultHttpRequestWriterFactory.(DefaultHttpRequestWriterFactory.java:47)
> at
> org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFactory.(ManagedNHttpClientConnectionFactory.java:75)
> at
> org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFactory.(ManagedNHttpClientConnectionFactory.java:83)
> at
> org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFactory.(ManagedNHttpClientConnectionFactory.java:64)
> at Main$.main(Main.scala:9)
> at Main.main(Main.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflec
[jira] [Updated] (FLINK-6810) Add Some built-in Scalar Function supported
[
https://issues.apache.org/jira/browse/FLINK-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sunjincheng updated FLINK-6810:
---
Labels: starter (was: )
> Add Some built-in Scalar Function supported
> ---
>
> Key: FLINK-6810
> URL: https://issues.apache.org/jira/browse/FLINK-6810
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Affects Versions: 1.4.0
>Reporter: sunjincheng
> Labels: starter
>
> In this JIRA, will create some sub-task for add specific scalar function,
> such as mathematical-function {{LOG}}, date-functions
> {{DATEADD}},string-functions {{LPAD}}, etc.
> I think is good way to let SQL work, and then add TableAPI to supported. So I
> suggest one scalar function create two sub-task, one is for SQL. another for
> TableAPI.
> *Note:*
> Every scalar function should add TableAPI doc in
> {{./docs/dev/table/tableApi.md#built-in-functions}}.
> Add SQL doc in {{./docs/dev/table/sql.md#built-in-functions}}.
> Welcome anybody to add the sub-task about standard database scalar function.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (FLINK-6966) Add maxParallelism and UIDs to all operators generated by the Table API / SQL
[
https://issues.apache.org/jira/browse/FLINK-6966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059493#comment-16059493
]
sunjincheng edited comment on FLINK-6966 at 6/22/17 2:53 PM:
-
Hi [~fhueske] That's true. I got it. In our side we already meet that problem.
we can not increase the parallelism of an operator if we do not set
{{maxParallelism}}.
* About {{maxParallelism}}, it is not easy to set every operator's
{{maxParallelism}} by SQL/TableAPI. I think we can set global
{{maxParallelism}} by {{StreamExecutionEnvironment}}. Of course we can define a
default value for user or give a hint message if the user is not set.
* About UID, for the first version I think we can just deal with the operators
which are generated by {{DataStreamRel#translateToPlan}}.
Please let me know what you think?
Hi [~jark] I think {{QueryConfig}} is difficult to set set every operator's
{{maxParallelism}}, If we want a global value, the
{{StreamExecutionEnvironment}} is enough. What do you think?
Feel free to correct me If there are any incorrect describe. [~fhueske] [~jark]
was (Author: sunjincheng121):
Hi [~fhueske] That's true. I got it. In our side we already meet that problem.
we can not increase the parallelism of an operator if we do not set
{{maxParallelism}}.
* About {{maxParallelism}}, it is not easy to set every operator's
{{maxParallelism}} by SQL/TableAPI. I think we can set global
{{maxParallelism}} by {{StreamExecutionEnvironment}}. Of course we can define a
default value for user or give a hint message if the user is not set.
* About UID, for the first version I think we can just deal with the operators
which are generated by {{DataStreamRel#translateToPlan}}.
Please let me know what you think?
Hi [~jark] I think {{QueryConfig }} is difficult to set set every operator's
{{maxParallelism}}, If we want a global value, the
{{StreamExecutionEnvironment}} is enough. What do you think?
Feel free to correct me If there are any incorrect describe. [~fhueske] [~jark]
> Add maxParallelism and UIDs to all operators generated by the Table API / SQL
> -
>
> Key: FLINK-6966
> URL: https://issues.apache.org/jira/browse/FLINK-6966
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.4.0
>Reporter: Fabian Hueske
>
> At the moment, the Table API does not assign UIDs and the max parallelism to
> operators (except for operators with parallelism 1).
> We should do that to avoid problems when rescaling or restarting jobs from
> savepoints.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6966) Add maxParallelism and UIDs to all operators generated by the Table API / SQL
[
https://issues.apache.org/jira/browse/FLINK-6966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059493#comment-16059493
]
sunjincheng commented on FLINK-6966:
Hi [~fhueske] That's true. I got it. In our side we already meet that problem.
we can not increase the parallelism of an operator if we do not set
{{maxParallelism}}.
* About {{maxParallelism}}, it is not easy to set every operator's
{{maxParallelism}} by SQL/TableAPI. I think we can set global
{{maxParallelism}} by {{StreamExecutionEnvironment}}. Of course we can define a
default value for user or give a hint message if the user is not set.
* About UID, for the first version I think we can just deal with the operators
which are generated by {{DataStreamRel#translateToPlan}}.
Please let me know what you think?
Hi [~jark] I think {{QueryConfig }} is difficult to set set every operator's
{{maxParallelism}}, If we want a global value, the
{{StreamExecutionEnvironment}} is enough. What do you think?
Feel free to correct me If there are any incorrect describe. [~fhueske] [~jark]
> Add maxParallelism and UIDs to all operators generated by the Table API / SQL
> -
>
> Key: FLINK-6966
> URL: https://issues.apache.org/jira/browse/FLINK-6966
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.4.0
>Reporter: Fabian Hueske
>
> At the moment, the Table API does not assign UIDs and the max parallelism to
> operators (except for operators with parallelism 1).
> We should do that to avoid problems when rescaling or restarting jobs from
> savepoints.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLINK-6987) TextInputFormatTest fails when run in path containing spaces
[
https://issues.apache.org/jira/browse/FLINK-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-6987:
Summary: TextInputFormatTest fails when run in path containing spaces
(was: Flink does not build if path contains spaces)
> TextInputFormatTest fails when run in path containing spaces
>
>
> Key: FLINK-6987
> URL: https://issues.apache.org/jira/browse/FLINK-6987
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.3.1
>Reporter: Timo Walther
>
> The test {{TextInputFormatTest.testNestedFileRead}} fails if the path
> contains spaces.
> Reason: "Test erroneous"
> I was building Flink on MacOS 10.12.5 and the folder was called "flink-1.3.1
> 2".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLINK-6987) Flink does not build if path contains spaces
[
https://issues.apache.org/jira/browse/FLINK-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walther updated FLINK-6987:
Affects Version/s: 1.3.1
> Flink does not build if path contains spaces
>
>
> Key: FLINK-6987
> URL: https://issues.apache.org/jira/browse/FLINK-6987
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.3.1
>Reporter: Timo Walther
>
> The test {{TextInputFormatTest.testNestedFileRead}} fails if the path
> contains spaces.
> Reason: "Test erroneous"
> I was building Flink on MacOS 10.12.5 and the folder was called "flink-1.3.1
> 2".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-6987) Flink does not build if path contains spaces
Timo Walther created FLINK-6987:
---
Summary: Flink does not build if path contains spaces
Key: FLINK-6987
URL: https://issues.apache.org/jira/browse/FLINK-6987
Project: Flink
Issue Type: Bug
Components: Build System
Reporter: Timo Walther
The test {{TextInputFormatTest.testNestedFileRead}} fails if the path contains
spaces.
Reason: "Test erroneous"
I was building Flink on MacOS 10.12.5 and the folder was called "flink-1.3.1 2".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6925) Add CONCAT/CONCAT_WS supported in SQL
[
https://issues.apache.org/jira/browse/FLINK-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059454#comment-16059454
]
ASF GitHub Bot commented on FLINK-6925:
---
Github user sunjincheng121 commented on the issue:
https://github.com/apache/flink/pull/4138
I have updated the PR. Welcome anyone to review.
> Add CONCAT/CONCAT_WS supported in SQL
> -
>
> Key: FLINK-6925
> URL: https://issues.apache.org/jira/browse/FLINK-6925
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Affects Versions: 1.4.0
>Reporter: sunjincheng
>Assignee: sunjincheng
>
> CONCAT(str1,str2,...)Returns the string that results from concatenating the
> arguments. May have one or more arguments. If all arguments are nonbinary
> strings, the result is a nonbinary string. If the arguments include any
> binary strings, the result is a binary string. A numeric argument is
> converted to its equivalent nonbinary string form.
> CONCAT() returns NULL if any argument is NULL.
> * Syntax:
> CONCAT(str1,str2,...)
> * Arguments
> ** str1,str2,... -
> * Return Types
> string
> * Example:
> CONCAT('F', 'lin', 'k') -> 'Flink'
> CONCAT('M', NULL, 'L') -> NULL
> CONCAT(14.3) -> '14.3'
> * See more:
> ** [MySQL|
> https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_concat]
> CONCAT_WS() stands for Concatenate With Separator and is a special form of
> CONCAT(). The first argument is the separator for the rest of the arguments.
> The separator is added between the strings to be concatenated. The separator
> can be a string, as can the rest of the arguments. If the separator is NULL,
> the result is NULL.
> * Syntax:
> CONCAT_WS(separator,str1,str2,...)
> * Arguments
> ** separator -
> ** str1,str2,... -
> * Return Types
> string
> * Example:
> CONCAT_WS(',','First name','Second name','Last Name') -> 'First name,Second
> name,Last Name'
> * See more:
> ** [MySQL|
> https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_concat-ws]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #4138: [FLINK-6925][table]Add CONCAT/CONCAT_WS supported in SQL
Github user sunjincheng121 commented on the issue: https://github.com/apache/flink/pull/4138 I have updated the PR. Welcome anyone to review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-6986) Broken links to Photoshop images
Greg Hogan created FLINK-6986: - Summary: Broken links to Photoshop images Key: FLINK-6986 URL: https://issues.apache.org/jira/browse/FLINK-6986 Project: Flink Issue Type: Bug Components: Project Website Reporter: Greg Hogan Assignee: Greg Hogan Priority: Minor The "Black outline logo with text" links on the [community|https://flink.apache.org/community.html] page are broken. I'd like to see if we can find a comprehensive solution for broken links. I only noticed this due to random clicking. I think Google can report broken links or we could run our own scan. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-6785) Ineffective checks in MetricRegistryTest
[
https://issues.apache.org/jira/browse/FLINK-6785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059396#comment-16059396
]
ASF GitHub Bot commented on FLINK-6785:
---
Github user NicoK commented on the issue:
https://github.com/apache/flink/pull/4035
change looks good but fails the new checkstyle rules:
```
[INFO] There are 4 errors reported by Checkstyle 6.19 with
/tools/maven/strict-checkstyle.xml ruleset.
[ERROR]
src/test/java/org/apache/flink/runtime/metrics/MetricRegistryTest.java:[256]
(whitespace) EmptyLineSeparator: There is more than 1 empty line one after
another.
[ERROR]
src/test/java/org/apache/flink/runtime/metrics/MetricRegistryTest.java:[259]
(regexp) RegexpSingleline: Trailing whitespace
[ERROR]
src/test/java/org/apache/flink/runtime/metrics/MetricRegistryTest.java:[265]
(whitespace) EmptyLineSeparator: There is more than 1 empty line one after
another.
[ERROR]
src/test/java/org/apache/flink/runtime/metrics/MetricRegistryTest.java:[278]
(regexp) RegexpSingleline: Trailing whitespace
```
> Ineffective checks in MetricRegistryTest
>
>
> Key: FLINK-6785
> URL: https://issues.apache.org/jira/browse/FLINK-6785
> Project: Flink
> Issue Type: Bug
> Components: Metrics, Tests
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>
> Several tests in {{MetricRegistryTest}} have reporters doing assertions. By
> design exceptions from reporters are however catched and logged, and thus
> can't fail the test.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #4035: [FLINK-6785] [metrics] Fix ineffective asserts in MetricR...
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4035 change looks good but fails the new checkstyle rules: ``` [INFO] There are 4 errors reported by Checkstyle 6.19 with /tools/maven/strict-checkstyle.xml ruleset. [ERROR] src/test/java/org/apache/flink/runtime/metrics/MetricRegistryTest.java:[256] (whitespace) EmptyLineSeparator: There is more than 1 empty line one after another. [ERROR] src/test/java/org/apache/flink/runtime/metrics/MetricRegistryTest.java:[259] (regexp) RegexpSingleline: Trailing whitespace [ERROR] src/test/java/org/apache/flink/runtime/metrics/MetricRegistryTest.java:[265] (whitespace) EmptyLineSeparator: There is more than 1 empty line one after another. [ERROR] src/test/java/org/apache/flink/runtime/metrics/MetricRegistryTest.java:[278] (regexp) RegexpSingleline: Trailing whitespace ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6857) Add global default Kryo serializer configuration to StreamExecutionEnvironment
[
https://issues.apache.org/jira/browse/FLINK-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059211#comment-16059211
]
mingleizhang commented on FLINK-6857:
-
Hi, [~tzulitai] I have gave a PR to this issue. Could you take a review ? :D
> Add global default Kryo serializer configuration to StreamExecutionEnvironment
> --
>
> Key: FLINK-6857
> URL: https://issues.apache.org/jira/browse/FLINK-6857
> Project: Flink
> Issue Type: Improvement
> Components: Configuration, Type Serialization System
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: mingleizhang
>
> See ML for original discussion:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/KryoException-Encountered-unregistered-class-ID-td13476.html.
> We should have an additional {{setDefaultKryoSerializer}} method that allows
> overriding the global default serializer that is not tied to specific classes
> (out-of-the-box Kryo uses the {{FieldSerializer}} if no matches for default
> serializer settings can be found for a class). Internally in Flink's
> {{KryoSerializer}}, this would only be a matter of proxying that configured
> global default serializer for Kryo by calling
> {{Kryo.setDefaultSerializer(...)}} on the created Kryo instance.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6857) Add global default Kryo serializer configuration to StreamExecutionEnvironment
[
https://issues.apache.org/jira/browse/FLINK-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059208#comment-16059208
]
ASF GitHub Bot commented on FLINK-6857:
---
GitHub user zhangminglei opened a pull request:
https://github.com/apache/flink/pull/4166
[FLINK-6857] [types] Add global default Kryo serializer configuration…
… to StreamExecutionEnvironment
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhangminglei/flink flink-6857
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4166.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4166
commit f8fbab7640c0713487de6eefa01e03597aea47cc
Author: zhangminglei
Date: 2017-06-22T11:28:39Z
[FLINK-6857] [types] Add global default Kryo serializer configuration to
StreamExecutionEnvironment
> Add global default Kryo serializer configuration to StreamExecutionEnvironment
> --
>
> Key: FLINK-6857
> URL: https://issues.apache.org/jira/browse/FLINK-6857
> Project: Flink
> Issue Type: Improvement
> Components: Configuration, Type Serialization System
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: mingleizhang
>
> See ML for original discussion:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/KryoException-Encountered-unregistered-class-ID-td13476.html.
> We should have an additional {{setDefaultKryoSerializer}} method that allows
> overriding the global default serializer that is not tied to specific classes
> (out-of-the-box Kryo uses the {{FieldSerializer}} if no matches for default
> serializer settings can be found for a class). Internally in Flink's
> {{KryoSerializer}}, this would only be a matter of proxying that configured
> global default serializer for Kryo by calling
> {{Kryo.setDefaultSerializer(...)}} on the created Kryo instance.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink pull request #4166: [FLINK-6857] [types] Add global default Kryo seria...
GitHub user zhangminglei opened a pull request:
https://github.com/apache/flink/pull/4166
[FLINK-6857] [types] Add global default Kryo serializer configurationâ¦
⦠to StreamExecutionEnvironment
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhangminglei/flink flink-6857
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4166.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4166
commit f8fbab7640c0713487de6eefa01e03597aea47cc
Author: zhangminglei
Date: 2017-06-22T11:28:39Z
[FLINK-6857] [types] Add global default Kryo serializer configuration to
StreamExecutionEnvironment
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-6897) Re-add support for Java 8 lambdas in CEP library
[
https://issues.apache.org/jira/browse/FLINK-6897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059137#comment-16059137
]
ASF GitHub Bot commented on FLINK-6897:
---
GitHub user dawidwys opened a pull request:
https://github.com/apache/flink/pull/4165
[FLINK-6897][cep] Re-add support for Java 8 lambdas in CEP library
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dawidwys/flink cep-conditions-lambda
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4165.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4165
commit ade1464ae4460fe7a23903685ec21cbf768d4c66
Author: Dawid Wysakowicz
Date: 2017-06-22T10:35:03Z
[FLINK-6897] Re-add support for Java 8 lambdas in CEP library
> Re-add support for Java 8 lambdas in CEP library
>
>
> Key: FLINK-6897
> URL: https://issues.apache.org/jira/browse/FLINK-6897
> Project: Flink
> Issue Type: Bug
> Components: CEP
>Affects Versions: 1.3.0
>Reporter: Kostas Kloudas
>Assignee: Dawid Wysakowicz
>
> Relevant [mailing list
> thread|http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Java-8-lambdas-for-CEP-patterns-won-t-compile-td13626.html]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink pull request #4165: [FLINK-6897][cep] Re-add support for Java 8 lambda...
GitHub user dawidwys opened a pull request:
https://github.com/apache/flink/pull/4165
[FLINK-6897][cep] Re-add support for Java 8 lambdas in CEP library
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dawidwys/flink cep-conditions-lambda
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4165.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4165
commit ade1464ae4460fe7a23903685ec21cbf768d4c66
Author: Dawid Wysakowicz
Date: 2017-06-22T10:35:03Z
[FLINK-6897] Re-add support for Java 8 lambdas in CEP library
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-6762) Cannot rescale externalized incremental checkpoints
[ https://issues.apache.org/jira/browse/FLINK-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059133#comment-16059133 ] Gyula Fora commented on FLINK-6762: --- In any case the job should fail at recovery instead of the first checkpoint. > Cannot rescale externalized incremental checkpoints > --- > > Key: FLINK-6762 > URL: https://issues.apache.org/jira/browse/FLINK-6762 > Project: Flink > Issue Type: Bug > Components: State Backends, Checkpointing >Affects Versions: 1.3.0 >Reporter: Gyula Fora >Priority: Critical > > When a job is rescaled from an externalized incremental checkpoint, the > subsequent checkpoints fail with the following error: > org.apache.flink.runtime.checkpoint.CheckpointException: Could not finalize > the pending checkpoint 3205. > at > org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:861) > at > org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveAcknowledgeMessage(CheckpointCoordinator.java:776) > at > org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$handleCheckpointMessage$1.apply$mcV$sp(JobManager.scala:1462) > at > org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$handleCheckpointMessage$1.apply(JobManager.scala:1461) > at > org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$handleCheckpointMessage$1.apply(JobManager.scala:1461) > at > scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) > at > scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) > at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) > at > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) > at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > Caused by: java.io.IOException: Unknown implementation of StreamStateHandle: > class org.apache.flink.runtime.state.PlaceholderStreamStateHandle > at > org.apache.flink.runtime.checkpoint.savepoint.SavepointV2Serializer.serializeStreamStateHandle(SavepointV2Serializer.java:484) > at > org.apache.flink.runtime.checkpoint.savepoint.SavepointV2Serializer.serializeStreamStateHandleMap(SavepointV2Serializer.java:342) > at > org.apache.flink.runtime.checkpoint.savepoint.SavepointV2Serializer.serializeKeyedStateHandle(SavepointV2Serializer.java:329) > at > org.apache.flink.runtime.checkpoint.savepoint.SavepointV2Serializer.serializeSubtaskState(SavepointV2Serializer.java:270) > at > org.apache.flink.runtime.checkpoint.savepoint.SavepointV2Serializer.serialize(SavepointV2Serializer.java:122) > at > org.apache.flink.runtime.checkpoint.savepoint.SavepointV2Serializer.serialize(SavepointV2Serializer.java:66) > at > org.apache.flink.runtime.checkpoint.savepoint.SavepointStore.storeSavepointToHandle(SavepointStore.java:199) > at > org.apache.flink.runtime.checkpoint.savepoint.SavepointStore.storeExternalizedCheckpointToHandle(SavepointStore.java:164) > at > org.apache.flink.runtime.checkpoint.PendingCheckpoint.finalizeCheckpointExternalized(PendingCheckpoint.java:286) > at > org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:851) > Full log: > https://gist.github.com/gyfora/693b9a720aace843ff4570e504c4a242 > Rescaling with savepoints work. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-6943) Improve exceptions within TypeExtractionUtils#getSingleAbstractMethod
[ https://issues.apache.org/jira/browse/FLINK-6943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059127#comment-16059127 ] ASF GitHub Bot commented on FLINK-6943: --- Github user dawidwys commented on the issue: https://github.com/apache/flink/pull/4140 Great, thanks! > Improve exceptions within TypeExtractionUtils#getSingleAbstractMethod > - > > Key: FLINK-6943 > URL: https://issues.apache.org/jira/browse/FLINK-6943 > Project: Flink > Issue Type: Bug > Components: Core >Reporter: Dawid Wysakowicz >Assignee: Dawid Wysakowicz > > Exception message seems to be inexact. > Also if there is no SAM, sam would be null upon returning from the method. > The suggestion from a review was to change the message and add a check (for > null sam) prior to returning. > Another suggestion is to check if the given method is an interface, as only > for interface it is possible to pass lambda. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink issue #4140: [FLINK-6943] Improve exceptions within TypeExtractionUtil...
Github user dawidwys commented on the issue: https://github.com/apache/flink/pull/4140 Great, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6966) Add maxParallelism and UIDs to all operators generated by the Table API / SQL
[
https://issues.apache.org/jira/browse/FLINK-6966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059121#comment-16059121
]
Jark Wu commented on FLINK-6966:
We can add an entry to the {{QueryConfig}} to set the max parallelism, and only
set {{maxParallelism}} for the state relative operators.
> Add maxParallelism and UIDs to all operators generated by the Table API / SQL
> -
>
> Key: FLINK-6966
> URL: https://issues.apache.org/jira/browse/FLINK-6966
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Affects Versions: 1.4.0
>Reporter: Fabian Hueske
>
> At the moment, the Table API does not assign UIDs and the max parallelism to
> operators (except for operators with parallelism 1).
> We should do that to avoid problems when rescaling or restarting jobs from
> savepoints.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6925) Add CONCAT/CONCAT_WS supported in SQL
[
https://issues.apache.org/jira/browse/FLINK-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059118#comment-16059118
]
ASF GitHub Bot commented on FLINK-6925:
---
Github user sunjincheng121 commented on the issue:
https://github.com/apache/flink/pull/4138
Need add SQL doc, And a `concat_ws` bug need be fix. Updated the PR. later.
> Add CONCAT/CONCAT_WS supported in SQL
> -
>
> Key: FLINK-6925
> URL: https://issues.apache.org/jira/browse/FLINK-6925
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Affects Versions: 1.4.0
>Reporter: sunjincheng
>Assignee: sunjincheng
>
> CONCAT(str1,str2,...)Returns the string that results from concatenating the
> arguments. May have one or more arguments. If all arguments are nonbinary
> strings, the result is a nonbinary string. If the arguments include any
> binary strings, the result is a binary string. A numeric argument is
> converted to its equivalent nonbinary string form.
> CONCAT() returns NULL if any argument is NULL.
> * Syntax:
> CONCAT(str1,str2,...)
> * Arguments
> ** str1,str2,... -
> * Return Types
> string
> * Example:
> CONCAT('F', 'lin', 'k') -> 'Flink'
> CONCAT('M', NULL, 'L') -> NULL
> CONCAT(14.3) -> '14.3'
> * See more:
> ** [MySQL|
> https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_concat]
> CONCAT_WS() stands for Concatenate With Separator and is a special form of
> CONCAT(). The first argument is the separator for the rest of the arguments.
> The separator is added between the strings to be concatenated. The separator
> can be a string, as can the rest of the arguments. If the separator is NULL,
> the result is NULL.
> * Syntax:
> CONCAT_WS(separator,str1,str2,...)
> * Arguments
> ** separator -
> ** str1,str2,... -
> * Return Types
> string
> * Example:
> CONCAT_WS(',','First name','Second name','Last Name') -> 'First name,Second
> name,Last Name'
> * See more:
> ** [MySQL|
> https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_concat-ws]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #4138: [FLINK-6925][table]Add CONCAT/CONCAT_WS supported in SQL
Github user sunjincheng121 commented on the issue: https://github.com/apache/flink/pull/4138 Need add SQL doc, And a `concat_ws` bug need be fix. Updated the PR. later. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Closed] (FLINK-6952) Add link to Javadocs
[ https://issues.apache.org/jira/browse/FLINK-6952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ufuk Celebi closed FLINK-6952. -- Resolution: Fixed Fix Version/s: 1.4.0 1.3.1 1.2.2 Fixed in bb7f08b (release-1.2), 5fa60c7 (release-1.3), e7c887c (master). > Add link to Javadocs > > > Key: FLINK-6952 > URL: https://issues.apache.org/jira/browse/FLINK-6952 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Ufuk Celebi >Assignee: Ufuk Celebi >Priority: Minor > Fix For: 1.2.2, 1.3.1, 1.4.0 > > > The project webpage and the docs are missing links to the Javadocs. > I think we should add them as part of the external links at the bottom of the > doc navigation (above "Project Page"). > In the same manner we could add a link to the Scaladocs, but if I remember > correctly there was a problem with the build of the Scaladocs. Correct, > [~aljoscha]? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (FLINK-6985) Remove bugfix version from docs title
[ https://issues.apache.org/jira/browse/FLINK-6985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ufuk Celebi closed FLINK-6985. -- Resolution: Fixed Fix Version/s: 1.2.0 1.3.0 1.4.0 Fixed in bd6277b (release-1.2), 9139791 (release-1.3), 0287758 (master). > Remove bugfix version from docs title > - > > Key: FLINK-6985 > URL: https://issues.apache.org/jira/browse/FLINK-6985 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Ufuk Celebi >Assignee: Ufuk Celebi >Priority: Minor > Fix For: 1.4.0, 1.3.0, 1.2.0 > > > The docs HTML title contains the minor version of the corresponding release. > This can be confusing as we build the docs nightly from the respective > release branch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink pull request #4163: [FLINK-6952] [FLINK-6985] [docs] Add Javadocs link...
Github user uce closed the pull request at: https://github.com/apache/flink/pull/4163 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #4164: [FLINK-6952] [FLINK-6985] [docs] Add Javadocs link...
Github user uce closed the pull request at: https://github.com/apache/flink/pull/4164 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6952) Add link to Javadocs
[ https://issues.apache.org/jira/browse/FLINK-6952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059107#comment-16059107 ] ASF GitHub Bot commented on FLINK-6952: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/4162 > Add link to Javadocs > > > Key: FLINK-6952 > URL: https://issues.apache.org/jira/browse/FLINK-6952 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Ufuk Celebi >Assignee: Ufuk Celebi >Priority: Minor > > The project webpage and the docs are missing links to the Javadocs. > I think we should add them as part of the external links at the bottom of the > doc navigation (above "Project Page"). > In the same manner we could add a link to the Scaladocs, but if I remember > correctly there was a problem with the build of the Scaladocs. Correct, > [~aljoscha]? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-6952) Add link to Javadocs
[ https://issues.apache.org/jira/browse/FLINK-6952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059109#comment-16059109 ] ASF GitHub Bot commented on FLINK-6952: --- Github user uce closed the pull request at: https://github.com/apache/flink/pull/4164 > Add link to Javadocs > > > Key: FLINK-6952 > URL: https://issues.apache.org/jira/browse/FLINK-6952 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Ufuk Celebi >Assignee: Ufuk Celebi >Priority: Minor > > The project webpage and the docs are missing links to the Javadocs. > I think we should add them as part of the external links at the bottom of the > doc navigation (above "Project Page"). > In the same manner we could add a link to the Scaladocs, but if I remember > correctly there was a problem with the build of the Scaladocs. Correct, > [~aljoscha]? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-6952) Add link to Javadocs
[ https://issues.apache.org/jira/browse/FLINK-6952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059108#comment-16059108 ] ASF GitHub Bot commented on FLINK-6952: --- Github user uce closed the pull request at: https://github.com/apache/flink/pull/4163 > Add link to Javadocs > > > Key: FLINK-6952 > URL: https://issues.apache.org/jira/browse/FLINK-6952 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Ufuk Celebi >Assignee: Ufuk Celebi >Priority: Minor > > The project webpage and the docs are missing links to the Javadocs. > I think we should add them as part of the external links at the bottom of the > doc navigation (above "Project Page"). > In the same manner we could add a link to the Scaladocs, but if I remember > correctly there was a problem with the build of the Scaladocs. Correct, > [~aljoscha]? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink pull request #4162: [FLINK-6952] [FLINK-6985] [docs] Add Javadocs link...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/4162 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6958) Async I/O timeout not work
[
https://issues.apache.org/jira/browse/FLINK-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059106#comment-16059106
]
Jark Wu commented on FLINK-6958:
I have dug into this issue. When the source is a bounded source, such as
{{env.fromCollection}} , and the {{AsyncFunction}} do nothing. The async i/o
operator will hang on the {{close()}} method to wait the pending element queue
finished.
The reason is that we register the timeout timer using
{{ProcessingTimeService}}. The source complete very soon, and then the
AsyncOperator exit the {{run()}} block and call {{quiesceAndAwaitPending}} on
the TimeService. But we remove all pending timers in it , because we set the
policy:
{{timeService.setExecuteExistingDelayedTasksAfterShutdownPolicy(false);}}.
That's why all the pending element will never complete and result in the dead
lock.
[~till.rohrmann] Why do we remove all the pending timers ? Can we set the
policy to true?
> Async I/O timeout not work
> --
>
> Key: FLINK-6958
> URL: https://issues.apache.org/jira/browse/FLINK-6958
> Project: Flink
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 1.2.1
>Reporter: feng xiaojie
>
> when use Async I/O with UnorderedStreamElementQueue, the queue will always
> full if you don't call the AsyncCollector.collect to ack them.
> Timeout shall collect these entries when the timeout trigger,but it isn't work
> I debug find,
> when time out, it will call resultFuture.completeExceptionally(error);
> but not call UnorderedStreamElementQueue.onCompleteHandler
> it will cause that async i/o hang always
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (FLINK-6958) Async I/O timeout not work
[ https://issues.apache.org/jira/browse/FLINK-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu reassigned FLINK-6958: -- Assignee: Jark Wu > Async I/O timeout not work > -- > > Key: FLINK-6958 > URL: https://issues.apache.org/jira/browse/FLINK-6958 > Project: Flink > Issue Type: Bug > Components: Streaming >Affects Versions: 1.2.1 >Reporter: feng xiaojie >Assignee: Jark Wu > > when use Async I/O with UnorderedStreamElementQueue, the queue will always > full if you don't call the AsyncCollector.collect to ack them. > Timeout shall collect these entries when the timeout trigger,but it isn't work > I debug find, > when time out, it will call resultFuture.completeExceptionally(error); > but not call UnorderedStreamElementQueue.onCompleteHandler > it will cause that async i/o hang always -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-6932) Update the inaccessible Dataflow Model paper link
[ https://issues.apache.org/jira/browse/FLINK-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059098#comment-16059098 ] mingleizhang commented on FLINK-6932: - You are very welcome [~tzulitai]. > Update the inaccessible Dataflow Model paper link > - > > Key: FLINK-6932 > URL: https://issues.apache.org/jira/browse/FLINK-6932 > Project: Flink > Issue Type: Bug > Components: Documentation >Reporter: mingleizhang >Assignee: mingleizhang > Labels: None > Fix For: 1.3.1 > > > I tried to access the Dataflow Model paper link which under > [https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/event_time.html], > then it gives me an error [ 404 ] instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (FLINK-6937) Fix link markdown in Production Readiness Checklist doc
[ https://issues.apache.org/jira/browse/FLINK-6937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tzu-Li (Gordon) Tai closed FLINK-6937. -- > Fix link markdown in Production Readiness Checklist doc > --- > > Key: FLINK-6937 > URL: https://issues.apache.org/jira/browse/FLINK-6937 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Juan Paulo Gutierrez >Priority: Minor > Fix For: 1.4.0, 1.3.2 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
