[jira] [Commented] (FLINK-24351) translate "JSON Function" pages into Chinese
[ https://issues.apache.org/jira/browse/FLINK-24351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17423106#comment-17423106 ] ZhuoYu Chen commented on FLINK-24351: - Hi [~liliwei] , I am very interested in this，and I want do some job for flink，can I help to do that？ Thank you > translate "JSON Function" pages into Chinese > - > > Key: FLINK-24351 > URL: https://issues.apache.org/jira/browse/FLINK-24351 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Reporter: liwei li >Priority: Major > > translate "JSON Function" pages into Chinese, > docs/data/sql_functions_zh.yml > > https://github.com/apache/flink/pull/17275#issuecomment-924536467 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-24370) [FLIP-171] Documentation for Generic AsyncSinkBase
[ https://issues.apache.org/jira/browse/FLINK-24370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17423105#comment-17423105 ] ZhuoYu Chen commented on FLINK-24370: - Hi [~CrynetLogistics] , I am very interested in this，and I want do some job for flink，can I help to do that？ Thank you > [FLIP-171] Documentation for Generic AsyncSinkBase > -- > > Key: FLINK-24370 > URL: https://issues.apache.org/jira/browse/FLINK-24370 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Reporter: Zichen Liu >Assignee: Zichen Liu >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > h2. Motivation > To write documentation for FLIP-171 Async Sink Base. This will help sink > implementers get acquainted with the necessary information to write their > concrete sinks. > h2. References > More details to be found > [https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22644) Translate "Native Kubernetes" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-22644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17423104#comment-17423104 ] ZhuoYu Chen commented on FLINK-22644: - Hi [~rudeigerc] , I am very interested in this，and I want do some job for flink，can I help to do that？ Thank you > Translate "Native Kubernetes" page into Chinese > --- > > Key: FLINK-22644 > URL: https://issues.apache.org/jira/browse/FLINK-22644 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Yuchen Cheng >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned, > pull-request-available > > The page url is > https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22797) Improve incremental checkpoints documentation
[
https://issues.apache.org/jira/browse/FLINK-22797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17423102#comment-17423102
]
ZhuoYu Chen commented on FLINK-22797:
-
Hi [~Nicolaus Weidner] , I am very interested in this，and I want do some job
for flink，can I help to do that？
Thank you
> Improve incremental checkpoints documentation
> -
>
> Key: FLINK-22797
> URL: https://issues.apache.org/jira/browse/FLINK-22797
> Project: Flink
> Issue Type: Improvement
> Components: Documentation, Runtime / State Backends
>Reporter: Nicolaus Weidner
>Priority: Minor
>
> Information on incremental checkpoints is somewhat scattered and hard to find
> in the documentation. In particular, the main page on checkpoints [1] should
> contain at least a short description of incremental checkpoints. They are
> currently mentioned in only one sentence, without any further explanation.
> A very short summary with a link to the RocksDB section [2] would suffice,
> since they are specific to that state backend.
> They are also mentioned on the config pages [3 and 4], which likewise contain
> incomplete information and no link ("may not be supported by every state
> backend").
> One basically has to follow the above clues and deduce that there may be more
> information in the state backends section.
> [1]
> [https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/state/checkpoints/]
> [2]
> [https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/state/state_backends/#incremental-checkpoints]
> [3]
> [https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/datastream/fault-tolerance/checkpointing/]
> [4]
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23427) Translate the page of "Blocking Shuffle" into Chinese
[ https://issues.apache.org/jira/browse/FLINK-23427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17423101#comment-17423101 ] ZhuoYu Chen commented on FLINK-23427: - Hi [~Aiden Gong] , I am very interested in this，and I want do some job for flink，can I help to do that？ Thank you > Translate the page of "Blocking Shuffle" into Chinese > - > > Key: FLINK-23427 > URL: https://issues.apache.org/jira/browse/FLINK-23427 > Project: Flink > Issue Type: Improvement > Components: chinese-translation, Documentation >Affects Versions: 1.14.0 >Reporter: Aiden Gong >Assignee: Aiden Gong >Priority: Minor > Labels: pull-request-available > Fix For: 1.15.0 > > > Translate the internal page > "[https://ci.apache.org/projects/flink/flink-docs-master/zh/docs/ops/batch/blocking_shuffle/|https://ci.apache.org/projects/flink/flink-docs-master/zh/docs/ops/batch/blocking_shuffle/]; > into Chinese. > The doc located in "flink/docs/content.zh/docs/ops/batch/blocking_shuffle.md" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-24345) Translate "File Systems" page of "Internals" into Chinese
[ https://issues.apache.org/jira/browse/FLINK-24345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17423100#comment-17423100 ] ZhuoYu Chen commented on FLINK-24345: - Hi [~Liebing] , I am very interested in this，and I want do some job for flink，can I help to do that？ Thank you > Translate "File Systems" page of "Internals" into Chinese > - > > Key: FLINK-24345 > URL: https://issues.apache.org/jira/browse/FLINK-24345 > Project: Flink > Issue Type: Improvement > Components: chinese-translation, Documentation >Reporter: Liebing Yu >Priority: Major > > The page url is > https://nightlies.apache.org/flink/flink-docs-master/docs/internals/filesystems/ > The markdown file is located in > flink/docs/content/docs/internals/filesystems.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API
flinkbot edited a comment on pull request #17360: URL: https://github.com/apache/flink/pull/17360#issuecomment-927544424 ## CI report: * 3bbe36d1af9b8830e4a8605dced3b1442c53d64e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24688) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17400: [FLINK-24432][rocksdb] RocksIteratorWrapper.seekToLast() logic typo
flinkbot edited a comment on pull request #17400: URL: https://github.com/apache/flink/pull/17400#issuecomment-931731823 ## CI report: * be875f4f2424814aa5a2953e21fbd3fa151bb879 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24686) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] tisonkun commented on pull request #5: [FLINK-1][iteration] Adds the iteration module and API
tisonkun commented on pull request #5: URL: https://github.com/apache/flink-ml/pull/5#issuecomment-931855039 @gaoyunhaii how this PR can be "FLINK-1"? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] becketqin commented on a change in pull request #5: [FLINK-1][iteration] Adds the iteration module and API
becketqin commented on a change in pull request #5:
URL: https://github.com/apache/flink-ml/pull/5#discussion_r719903195
##
File path:
flink-ml-iteration/src/main/java/org/apache/flink/ml/iteration/Iterations.java
##
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.iteration;
+
+import org.apache.flink.annotation.PublicEvolving;
+
+/** A helper class to create iterations. */
+@PublicEvolving
+public class Iterations {
+
+/**
+ * This method uses an iteration body to process records in unbounded data
streams.
+ *
+ * This method invokes the iteration body with the following
parameters: 1) The 1st parameter
+ * is a list of input variable streams, which are created as the union of
the initial variable
+ * streams and the corresponding feedback variable streams (returned by
the iteration body). 2)
+ * The 2nd parameter is the data streams given to this method.
+ *
+ * The epoch values are determined as described below. See
IterationListener for how the
+ * epoch values are used. 1) All records in the initial variable streams
and initial data
+ * streams has epoch=0. 2) For any record emitted by this operator into a
non-feedback stream,
+ * the epoch of this emitted record = the epoch of the input record that
triggers this emission.
+ * If this record is emitted by onEpochWatermarkIncremented(), then the
epoch of this record =
+ * epochWatermark. 3) For any record emitted by this operator into a
feedback variable stream,
+ * the epoch of the emitted record = the epoch of the input record that
triggers this emission +
+ * 1. If this record is emitted by onEpochWatermarkIncremented(), then the
epoch of this record
+ * = epochWatermark.
+ *
+ * The iteration would not terminate if at least one of its inputs is
unbounded. Otherwise it
+ * will terminated after all the inputs are terminated and no more records
are iterating.
+ *
+ * Required: 1) The parallelism of any stream in the initial variable
streams must equal to
+ * the parallelism of the stream at the same index of the feedback
variable streams returned by
+ * the IterationBody.
+ *
+ * @param initVariableStreams The initial variable streams. These streams
will be merged with
+ * the feedback variable streams before being used as the 1st
parameter to invoke the
+ * iteration body.
+ * @param dataStreams The data streams. These streams will be used as the
2nd parameter to
+ * invoke the iteration body.
+ * @param body The computation logic which takes variable/data streams and
returns
+ * variable/output streams.
+ * @return The list of output streams returned by the iteration boy.
+ */
+public static DataStreamList iterateUnboundedStreams(
+DataStreamList initVariableStreams, DataStreamList dataStreams,
IterationBody body) {
+return null;
+}
+
+/**
+ * This method uses an iteration body to process records in some bounded
data streams
+ * iteratively until a termination criteria is reached (e.g. the given
number of rounds is
+ * completed or no further variable update is needed). Because this method
does not replay
+ * records in the data streams, the iteration body needs to cache those
records in order to
+ * visit those records repeatedly.
+ *
+ * This method invokes the iteration body with the following
parameters: 1) The 1st parameter
+ * is a list of input variable streams, which are created as the union of
the initial variable
+ * streams and the corresponding feedback variable streams (returned by
the iteration body). 2)
+ * The 2nd parameter is the data streams given to this method.
+ *
+ * The epoch values are determined as described below. See
IterationListener for how the
+ * epoch values are used. 1) All records in the initial variable streams
has epoch=0. 2) All
+ * records in the data streams has epoch=0. 3) For any record emitted by
this operator into a
+ * non-feedback stream, the epoch of this emitted record = the epoch of
the input record
[GitHub] [flink-ml] becketqin commented on a change in pull request #5: [FLINK-1][iteration] Adds the iteration module and API
becketqin commented on a change in pull request #5:
URL: https://github.com/apache/flink-ml/pull/5#discussion_r719859647
##
File path:
flink-ml-iteration/src/main/java/org/apache/flink/ml/iteration/Iterations.java
##
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.iteration;
+
+import org.apache.flink.annotation.PublicEvolving;
+
+/** A helper class to create iterations. */
+@PublicEvolving
+public class Iterations {
+
+/**
+ * This method uses an iteration body to process records in unbounded data
streams.
+ *
+ * This method invokes the iteration body with the following
parameters: 1) The 1st parameter
+ * is a list of input variable streams, which are created as the union of
the initial variable
+ * streams and the corresponding feedback variable streams (returned by
the iteration body). 2)
+ * The 2nd parameter is the data streams given to this method.
+ *
+ * The epoch values are determined as described below. See
IterationListener for how the
+ * epoch values are used. 1) All records in the initial variable streams
and initial data
+ * streams has epoch=0. 2) For any record emitted by this operator into a
non-feedback stream,
+ * the epoch of this emitted record = the epoch of the input record that
triggers this emission.
+ * If this record is emitted by onEpochWatermarkIncremented(), then the
epoch of this record =
+ * epochWatermark. 3) For any record emitted by this operator into a
feedback variable stream,
+ * the epoch of the emitted record = the epoch of the input record that
triggers this emission +
+ * 1. If this record is emitted by onEpochWatermarkIncremented(), then the
epoch of this record
+ * = epochWatermark.
Review comment:
Before we jump in and explain how epoch is determined, can we first give
some context on what the epoch stands for and used for? For example:
```
Each of the records involved in the iteration has an epoch, including the
records in the variable
stream, feedback stream, output stream and the side output emitted by
IterationListener. The
iterations use the record epoch to mark the progress of the iteration.
Different types of iterations
assign the epoch slightly different.
```
Also I think an example would actually help a lot. We may refer the java doc
to the FLIP where we have diagrams that are easier to understand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-ml] becketqin commented on a change in pull request #6: [FLINK-2][iteration] Add broadcast output to broadcast events to all the downstream tasks
becketqin commented on a change in pull request #6:
URL: https://github.com/apache/flink-ml/pull/6#discussion_r719874290
##
File path:
flink-ml-iteration/src/main/java/org/apache/flink/ml/iteration/broadcast/OutputReflectionContext.java
##
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.iteration.broadcast;
+
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.ml.iteration.utils.ReflectionUtils;
+import org.apache.flink.runtime.io.network.api.writer.RecordWriter;
+import org.apache.flink.runtime.plugable.SerializationDelegate;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.runtime.io.RecordWriterOutput;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.OutputTag;
+
+import java.lang.reflect.Field;
+
+/** The reflection utilities to parse the output and create the broadcast
output */
+public class OutputReflectionContext {
Review comment:
Should this class be a util class with everything static? Or at least a
singleton class?
##
File path:
flink-ml-iteration/src/main/java/org/apache/flink/ml/iteration/broadcast/BroadcastOutputFactory.java
##
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.iteration.broadcast;
+
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.runtime.io.network.api.writer.RecordWriter;
+import org.apache.flink.runtime.plugable.SerializationDelegate;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.runtime.streamrecord.StreamElement;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.OutputTag;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/** Factor that creates the corresponding {@link BroadcastOutput} from the
{@link Output}. */
+public class BroadcastOutputFactory {
+
+/**
+ * Creates the wrapper broadcast ooutput from {@code output}.
+ *
+ * @param output the original output.
+ * @param numRecordsOut the counter for the number of output record.
+ * @return the wrapped broadcast output.
+ */
+public static BroadcastOutput createBroadcastOutput(
+Output> output, Counter numRecordsOut) {
+
+OutputReflectionContext outputReflectionContext = new
OutputReflectionContext();
+
+List> internalOutputs = new ArrayList<>();
+if (outputReflectionContext.isBroadcastingOutput(output)) {
Review comment:
Is it possible to have nested `BroadcastingOutput`?
##
File path:
flink-ml-iteration/src/main/java/org/apache/flink/ml/iteration/broadcast/ChainingBroadcastOutput.java
##
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *
[jira] [Resolved] (FLINK-24200) CheckpointBarrierTrackerTest.testTwoLastBarriersOneByOne fails on azure
[
https://issues.apache.org/jira/browse/FLINK-24200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuan Mei resolved FLINK-24200.
--
Resolution: Fixed
merged commit
[{{1a9d782}}|https://github.com/apache/flink/commit/1a9d78259668f3d9c1a52dd278e8502d1449f780]
into apache:release-1.12
> CheckpointBarrierTrackerTest.testTwoLastBarriersOneByOne fails on azure
> ---
>
> Key: FLINK-24200
> URL: https://issues.apache.org/jira/browse/FLINK-24200
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.12.5
>Reporter: Xintong Song
>Assignee: Anton Kalashnikov
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.12.6
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=23719=logs=f0ac5c25-1168-55a5-07ff-0e88223afed9=0dbaca5d-7c38-52e6-f4fe-2fb69ccb3ada=8983
> {code}
> [ERROR] Tests run: 14, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 0.893 s <<< FAILURE! - in
> org.apache.flink.streaming.runtime.io.CheckpointBarrierTrackerTest
> [ERROR]
> testTwoLastBarriersOneByOne(org.apache.flink.streaming.runtime.io.CheckpointBarrierTrackerTest)
> Time elapsed: 0.093 s <<< FAILURE!
> java.lang.AssertionError:
> Expected: a value less than or equal to <30L>
> but: <33L> was greater than <30L>
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> at org.junit.Assert.assertThat(Assert.java:956)
> at org.junit.Assert.assertThat(Assert.java:923)
> at
> org.apache.flink.streaming.runtime.io.CheckpointBarrierTrackerTest.testTwoLastBarriersOneByOne(CheckpointBarrierTrackerTest.java:616)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] curcur merged pull request #17354: [FLINK-24200][streaming] Calculating maximum alignment time rather than using the constant value
curcur merged pull request #17354: URL: https://github.com/apache/flink/pull/17354 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API
flinkbot edited a comment on pull request #17360: URL: https://github.com/apache/flink/pull/17360#issuecomment-927544424 ## CI report: * 72a98b6c81ca3972146961a485a37c48cbe2033e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24635) * 3bbe36d1af9b8830e4a8605dced3b1442c53d64e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24688) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API
flinkbot edited a comment on pull request #17360: URL: https://github.com/apache/flink/pull/17360#issuecomment-927544424 ## CI report: * 72a98b6c81ca3972146961a485a37c48cbe2033e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24635) * 3bbe36d1af9b8830e4a8605dced3b1442c53d64e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] becketqin commented on a change in pull request #5: [FLINK-1][iteration] Adds the iteration module and API
becketqin commented on a change in pull request #5:
URL: https://github.com/apache/flink-ml/pull/5#discussion_r719152625
##
File path:
flink-ml-iteration/src/main/java/org/apache/flink/ml/iteration/DataStreamList.java
##
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.iteration;
+
+import org.apache.flink.streaming.api.datastream.DataStream;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+/** An utility class to maintain a list of {@link DataStream}, which might
have different types. */
Review comment:
A utility class
##
File path:
flink-ml-iteration/src/main/java/org/apache/flink/ml/iteration/IterationBodyResult.java
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.iteration;
+
+import org.apache.flink.streaming.api.datastream.DataStream;
+
+import javax.annotation.Nullable;
+
+/** The result of an iteration, specifying the feedbacks and the outputs. */
+public class IterationBodyResult {
+
+/**
+ * A list of feedback variable streams. These streams will only be used
during the iteration
+ * execution and will not be returned to the caller of the iteration body.
It is assumed that
+ * the method which executes the iteration body will feed the records of
the feedback variable
+ * streams back to the corresponding input variable streams.
+ */
+private final DataStreamList feedbackVariableStreams;
+
+/**
+ * A list of output streams. These streams will be returned to the caller
of the methods that
+ * execute the iteration body.
+ */
+private final DataStreamList outputStreams;
+
+/**
+ * An optional termination criteria stream. If this stream is not null, it
will be used together
+ * with the feedback variable streams to determine when the iteration
should terminate.
+ */
+private final @Nullable DataStream terminationCriteria;
+
+public IterationBodyResult(
+DataStreamList feedbackVariableStreams, DataStreamList
outputStreams) {
+this(feedbackVariableStreams, outputStreams, null);
+}
+
+public IterationBodyResult(
+DataStreamList feedbackVariableStreams,
+DataStreamList outputStreams,
+@Nullable DataStream terminationCriteria) {
+this.feedbackVariableStreams = feedbackVariableStreams;
+this.outputStreams = outputStreams;
+this.terminationCriteria = terminationCriteria;
+}
+
+public DataStreamList getFeedbackVariableStreams() {
Review comment:
I guess this is probably more of a code convention of Flink. So getter
method seems fine here.
##
File path:
flink-ml-iteration/src/main/java/org/apache/flink/ml/iteration/Iterations.java
##
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+
[GitHub] [flink] jherico commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API
jherico commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r719847453
##
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##
@@ -0,0 +1,191 @@
+---
+title: AWS Glue Avro
+weight: 4
+type: docs
+aliases:
+ - /dev/table/connectors/formats/avro-glue.html
+---
+
+
+# AWS Glue Avro Format
+
+{{< label "Format: Serialization Schema" >}}
+{{< label "Format: Deserialization Schema" >}}
+
+The Glue Schema Registry (``avro-glue``) format allows you to read records
that were serialized by the
``com.amazonaws.services.schemaregistry.serializers.avro.AWSKafkaAvroSerializer``
and to write records that can in turn be read by the
``com.amazonaws.services.schemaregistry.deserializers.avro.AWSKafkaAvroDeserializer``.
These records have their schemas stored out-of-band in a configured registry
provided by the AWS Glue Schema Registry
[service](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas).
Review comment:
will fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] jherico commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API
jherico commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r719847389
##
File path:
flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroFormatFactory.java
##
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static
org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AUTO_REGISTRATION;
+import static
org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AWS_REGION;
+import static
org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_SIZE;
+import static
org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_TTL_MS;
+import static
org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPATIBILITY;
+import static
org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPRESSION_TYPE;
+import static
org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.ENDPOINT;
+import static
org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.RECORD_TYPE;
+import static
org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.REGISTRY_NAME;
+import static
org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+@Internal
+public class GlueSchemaRegistryAvroFormatFactory implements
DeserializationFormatFactory, SerializationFormatFactory {
+public static final String IDENTIFIER = "avro-glue";
+
+@Override
+public DecodingFormat>
createDecodingFormat(DynamicTableFactory.Context context,
+ReadableConfig formatOptions) {
+FactoryUtil.validateFactoryOptions(this, formatOptions);
+final Map configMap = buildConfigMap(formatOptions);
+
+return new DecodingFormat>() {
+@Override
+public DeserializationSchema
createRuntimeDecoder(DynamicTableSource.Context context,
+DataType producedDataType) {
+final RowType rowType = (RowType)
producedDataType.getLogicalType();
+final TypeInformation rowDataTypeInfo =
context.createTypeInformation(producedDataType);
+return new AvroRowDataDeserializationSchema(
+
[GitHub] [flink] jherico commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API
jherico commented on a change in pull request #17360: URL: https://github.com/apache/flink/pull/17360#discussion_r719846371 ## File path: docs/data/sql_connectors.yml ## @@ -48,6 +48,12 @@ avro-confluent: category: format sql_url: https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-avro-confluent-registry/$version/flink-sql-avro-confluent-registry-$version.jar +avro-glue: +name: Avro AWS Glue Schema Registry +maven: flink-avro-glue-schema-registry +category: format +sql_url: https://repo.maven.apache.org/maven2/org/apache/flink/flink-avro-glue-schema-registry/$version/flink-avro-glue-schema-registry-$version.jar Review comment: The module is pure java and the only dependencies that it has that are scala-binary-versioned are `test` or `provided` scope. The module is currently located [here](https://repo1.maven.org/maven2/org/apache/flink/flink-avro-glue-schema-registry/) in the central repository, so I don't think this is the case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] jherico commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API
jherico commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r719845596
##
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##
@@ -0,0 +1,191 @@
+---
+title: AWS Glue Avro
+weight: 4
+type: docs
+aliases:
+ - /dev/table/connectors/formats/avro-glue.html
+---
+
+
+# AWS Glue Avro Format
+
+{{< label "Format: Serialization Schema" >}}
+{{< label "Format: Deserialization Schema" >}}
+
+The Glue Schema Registry (``avro-glue``) format allows you to read records
that were serialized by the
``com.amazonaws.services.schemaregistry.serializers.avro.AWSKafkaAvroSerializer``
and to write records that can in turn be read by the
``com.amazonaws.services.schemaregistry.deserializers.avro.AWSKafkaAvroDeserializer``.
These records have their schemas stored out-of-band in a configured registry
provided by the AWS Glue Schema Registry
[service](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas).
+
+When reading (deserializing) a record with this format the Avro writer schema
is fetched from the configured AWS Glue Schema Registry based on the schema
version id encoded in the record while the reader schema is inferred from table
schema.
+
+When writing (serializing) a record with this format the Avro schema is
inferred from the table schema and used to retrieve a schema id to be encoded
with the data. The lookup is performed against the configured AWS Glue Schema
Registry under the
[value](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas)
given in `avro-glue.schema-name`.
+
+The Avro Glue Schema Registry format can only be used in conjunction with the
[Apache Kafka SQL connector]({{< ref "docs/connectors/table/kafka" >}}) or the
[Upsert Kafka SQL Connector]({{< ref "docs/connectors/table/upsert-kafka" >}}).
+
+Dependencies
+
+
+{{< sql_download_table "avro-glue" >}}
+
+How to create tables with Avro-Glue format
+--
+
+Example of a table using raw UTF-8 string as Kafka key and Avro records
registered in the Schema Registry as Kafka values:
+
+```sql
+CREATE TABLE user_created (
+
+ -- one column mapped to the Kafka raw UTF-8 key
+ the_kafka_key STRING,
+
+ -- a few columns mapped to the Avro fields of the Kafka value
+ id STRING,
+ name STRING,
+ email STRING
+
+) WITH (
+
+ 'connector' = 'kafka',
+ 'topic' = 'user_events_example1',
+ 'properties.bootstrap.servers' = 'localhost:9092',
+
+ -- UTF-8 string as Kafka keys, using the 'the_kafka_key' table column
+ 'key.format' = 'raw',
+ 'key.fields' = 'the_kafka_key',
+
+ 'value.format' = 'avro-glue',
+ 'value.avro-glue.region' = 'us-east-1',
+ 'value.avro-glue.registry.name' = 'my-schema-registry',
+ 'value.avro-glue.schema-name' = 'my-schema-name',
+ 'value.fields-include' = 'EXCEPT_KEY'
+)
+```
+
+Format Options
+
+
+Yes, these options have inconsistent naming convnetions. No, I can't fix it.
This is for consistentcy with the existing [AWS Glue client
code](https://github.com/awslabs/aws-glue-schema-registry/blob/master/common/src/main/java/com/amazonaws/services/schemaregistry/utils/AWSSchemaRegistryConstants.java#L20).
Review comment:
I'm doing another pass on the naming of the options and will have it in
the next commit. Should be more inline with other formats, and as you suggest
I'm doing the translation inside the factory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Comment Edited] (FLINK-24427) Hide Scala in flink-table-planner from API
[ https://issues.apache.org/jira/browse/FLINK-24427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17423039#comment-17423039 ] Seth Wiesman edited comment on FLINK-24427 at 9/30/21, 11:01 PM: - Hi [~eloisant], The goal is to hide the planner from the user code classloader so that user code can use any scala version without the community having to cross build the planner. There are a few ways to achieve this in practice; and I'm not sure what specifically Timo has in mind. One option is to use Flink's plugin mechanism, which is used to load file systems and metric reporters, and hides all dependencies in dedicated classloaders. Another is what we recently did to hide akka from flink-runtime, making it effectively scala free FLINK-18783 was (Author: sjwiesman): Hi [~eloisant], The goal is to hide the planner from the user code classloader so that user code can use any scala version without the community having to cross build the planner. There are a few ways to achieve this in practice; and I'm not sure what specifically Timo has in mind. One option is to use Flink's plugin mechanism, which is used to load file systems and metric reporters, and hides all dependencies in a dedicated classloader. Another is what we recently did to hide akka from flink-runtime, making it effectively scala free FLINK-18783 > Hide Scala in flink-table-planner from API > -- > > Key: FLINK-24427 > URL: https://issues.apache.org/jira/browse/FLINK-24427 > Project: Flink > Issue Type: Sub-task > Components: API / Scala, Table SQL / API, Table SQL / Planner >Reporter: Timo Walther >Priority: Major > > FLIP-32 has decoupled the planner from the API. However, the planner code > base is still the largest Scala code base we have in Flink that we cannot > port to Java easily. > In order to allow arbitrary Scala versions in the user API, we suggest to > hide the Scala version of the planner from the Scala version of the API. The > API is mostly developed in Java and contains only a couple of classes > implemented in Scala. Those should be easier to maintain for various Scala > versions. The planner is already discovered via Java SPI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-24427) Hide Scala in flink-table-planner from API
[ https://issues.apache.org/jira/browse/FLINK-24427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17423039#comment-17423039 ] Seth Wiesman commented on FLINK-24427: -- Hi [~eloisant], The goal is to hide the planner from the user code classloader so that user code can use any scala version without the community having to cross build the planner. There are a few ways to achieve this in practice; and I'm not sure what specifically Timo has in mind. One option is to use Flink's plugin mechanism, which is used to load file systems and metric reporters, and hides all dependencies in a dedicated classloader. Another is what we recently did to hide akka from flink-runtime, making it effectively scala free FLINK-18783 > Hide Scala in flink-table-planner from API > -- > > Key: FLINK-24427 > URL: https://issues.apache.org/jira/browse/FLINK-24427 > Project: Flink > Issue Type: Sub-task > Components: API / Scala, Table SQL / API, Table SQL / Planner >Reporter: Timo Walther >Priority: Major > > FLIP-32 has decoupled the planner from the API. However, the planner code > base is still the largest Scala code base we have in Flink that we cannot > port to Java easily. > In order to allow arbitrary Scala versions in the user API, we suggest to > hide the Scala version of the planner from the Scala version of the API. The > API is mostly developed in Java and contains only a couple of classes > implemented in Scala. Those should be easier to maintain for various Scala > versions. The planner is already discovered via Java SPI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #17400: [FLINK-24432][rocksdb] RocksIteratorWrapper.seekToLast() logic typo
flinkbot edited a comment on pull request #17400: URL: https://github.com/apache/flink/pull/17400#issuecomment-931731823 ## CI report: * be875f4f2424814aa5a2953e21fbd3fa151bb879 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24686) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17400: [FLINK-24432][rocksdb] RocksIteratorWrapper.seekToLast() logic typo
flinkbot commented on pull request #17400: URL: https://github.com/apache/flink/pull/17400#issuecomment-931731823 ## CI report: * be875f4f2424814aa5a2953e21fbd3fa151bb879 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17400: [FLINK-24432][rocksdb] RocksIteratorWrapper.seekToLast() logic typo
flinkbot commented on pull request #17400: URL: https://github.com/apache/flink/pull/17400#issuecomment-931719917 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit be875f4f2424814aa5a2953e21fbd3fa151bb879 (Thu Sep 30 21:31:38 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-24432).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-24432) RocksIteratorWrapper.seekToLast() calls the wrong RocksIterator method
[
https://issues.apache.org/jira/browse/FLINK-24432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-24432:
---
Labels: pull-request-available (was: )
> RocksIteratorWrapper.seekToLast() calls the wrong RocksIterator method
> --
>
> Key: FLINK-24432
> URL: https://issues.apache.org/jira/browse/FLINK-24432
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.14.0
>Reporter: Victor Xu
>Priority: Minor
> Labels: pull-request-available
>
> The RocksIteratorWrapper is a wrapper of RocksIterator to do additional
> status check for all the methods. However, there's a typo that
> RocksIteratorWrapper.*seekToLast*() method calls RocksIterator's
> *seekToFirst*(), which is obviously wrong. I guess this issue wasn't found
> before as it was only referenced in the
> RocksTransformingIteratorWrapper.seekToLast() method and nowhere else.
> {code:java}
> @Override
> public void seekToFirst() {
> iterator.seekToFirst();
> status();
> }
> @Override
> public void seekToLast() {
> iterator.seekToFirst();
> status();
> }{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] victorunique opened a new pull request #17400: [FLINK-24432][rocksdb] RocksIteratorWrapper.seekToLast() logic typo
victorunique opened a new pull request #17400: URL: https://github.com/apache/flink/pull/17400 ## What is the purpose of the change The RocksIteratorWrapper is a wrapper of RocksIterator to do additional status check for all the methods. However, there's a typo that RocksIteratorWrapper.seekToLast() method calls RocksIterator's seekToFirst(), which is obviously wrong. I guess this issue wasn't found before as it was only referenced in the RocksTransformingIteratorWrapper.seekToLast() method and nowhere else. ## Brief change log *(for example:)* - Replace the *iterator.seekToFirst()* to the *iterator.seekToLast()* in the seekToLast method; ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-24432) RocksIteratorWrapper.seekToLast() calls the wrong RocksIterator method
Victor Xu created FLINK-24432:
-
Summary: RocksIteratorWrapper.seekToLast() calls the wrong
RocksIterator method
Key: FLINK-24432
URL: https://issues.apache.org/jira/browse/FLINK-24432
Project: Flink
Issue Type: Bug
Components: Runtime / State Backends
Affects Versions: 1.14.0
Reporter: Victor Xu
The RocksIteratorWrapper is a wrapper of RocksIterator to do additional status
check for all the methods. However, there's a typo that
RocksIteratorWrapper.*seekToLast*() method calls RocksIterator's
*seekToFirst*(), which is obviously wrong. I guess this issue wasn't found
before as it was only referenced in the
RocksTransformingIteratorWrapper.seekToLast() method and nowhere else.
{code:java}
@Override
public void seekToFirst() {
iterator.seekToFirst();
status();
}
@Override
public void seekToLast() {
iterator.seekToFirst();
status();
}{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #17139: [BP-1.13][FLINK-21853][e2e] Harden common_ha.sh#ha_tm_watchdog
flinkbot edited a comment on pull request #17139: URL: https://github.com/apache/flink/pull/17139#issuecomment-912546148 ## CI report: * fc899518ebc87cee5afce13d6ec3301de2b26c51 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24681) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17138: [BP-1.14][FLINK-21853][e2e] Harden common_ha.sh#ha_tm_watchdog
flinkbot edited a comment on pull request #17138: URL: https://github.com/apache/flink/pull/17138#issuecomment-912546041 ## CI report: * e2ff9e3ca2f4aa94c6d7666b83771012835ccec0 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24680) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17137: [FLINK-21853][e2e] Harden common_ha.sh#ha_tm_watchdog
flinkbot edited a comment on pull request #17137: URL: https://github.com/apache/flink/pull/17137#issuecomment-912545933 ## CI report: * b836a9082563539235d97241c4f44aff87bb5a7d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24679) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17399: [BP-1.14][FLINK-24357][tests] Harden ZooKeeperLeaderElectionConnectionHandlingTest
flinkbot edited a comment on pull request #17399: URL: https://github.com/apache/flink/pull/17399#issuecomment-931472290 ## CI report: * 238bb66a8a84f1545c971b0a9eb87b65fec835bc Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24676) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17398: [FLINK-24357][tests] Harden ZooKeeperLeaderElectionConnectionHandlingTest
flinkbot edited a comment on pull request #17398: URL: https://github.com/apache/flink/pull/17398#issuecomment-931472187 ## CI report: * 665b7b1db2c9d0c83329fe3d0ce66e0178c3becc Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24675) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17397: [BP-1.14][FLINK-24117][HA]Remove unHandledErrorListener in ZooKeeperLeaderElectionDriver and ZooKeeperLeaderRetrievalDriver
flinkbot edited a comment on pull request #17397: URL: https://github.com/apache/flink/pull/17397#issuecomment-931434206 ## CI report: * 958ebb4485f9e0f45c65a362542fcb9c56c063de Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24674) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17381: [FLINK-24399][table-common] Make handling of DataType less verbose for connector developers
flinkbot edited a comment on pull request #17381: URL: https://github.com/apache/flink/pull/17381#issuecomment-930061717 ## CI report: * a986056b7d9cb08a4807e588865cacf3376259ee UNKNOWN * a9124ed79554c32c3f8c5c785b3369bf1a49cfe6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24673) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17150: [FLINK-24117][HA]Remove unHandledErrorListener in ZooKeeperLeaderElec…
flinkbot edited a comment on pull request #17150: URL: https://github.com/apache/flink/pull/17150#issuecomment-913084999 ## CI report: * cc062cf3688d7328bc9388900a566c7c82b33fb0 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24671) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-24427) Hide Scala in flink-table-planner from API
[ https://issues.apache.org/jira/browse/FLINK-24427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422979#comment-17422979 ] Erwan Loisant edited comment on FLINK-24427 at 9/30/21, 7:08 PM: - I'm not sure what's the proposed solution here, I don't think hiding the version would really make it possible to use code compiled for Scala 2.12 in 2.13. So I guess either we cross-compile to multiple version (like it was done for 2.11 and 2.12 until now), or we migrate all the code to Java? was (Author: eloisant): I'm not sure what's the proposed solution here, I don't think hiding the version would really make it possible to use code compiled for Scala 2.12 in 2.13. So I guess either we cross-compile to multiple version (like it was done for 2.11 and 2.12 until now), or we migrate all the code to Java? > Hide Scala in flink-table-planner from API > -- > > Key: FLINK-24427 > URL: https://issues.apache.org/jira/browse/FLINK-24427 > Project: Flink > Issue Type: Sub-task > Components: API / Scala, Table SQL / API, Table SQL / Planner >Reporter: Timo Walther >Priority: Major > > FLIP-32 has decoupled the planner from the API. However, the planner code > base is still the largest Scala code base we have in Flink that we cannot > port to Java easily. > In order to allow arbitrary Scala versions in the user API, we suggest to > hide the Scala version of the planner from the Scala version of the API. The > API is mostly developed in Java and contains only a couple of classes > implemented in Scala. Those should be easier to maintain for various Scala > versions. The planner is already discovered via Java SPI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13414) Add support for Scala 2.13
[ https://issues.apache.org/jira/browse/FLINK-13414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422981#comment-17422981 ] Nick Burkard commented on FLINK-13414: -- Looks like the two tickets are now assigned, if there's anything else to help out with for opting out of Scala let me know [~MartijnVisser]. > Add support for Scala 2.13 > -- > > Key: FLINK-13414 > URL: https://issues.apache.org/jira/browse/FLINK-13414 > Project: Flink > Issue Type: New Feature > Components: API / Scala >Reporter: Chaoran Yu >Priority: Minor > Labels: auto-deprioritized-major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-24427) Hide Scala in flink-table-planner from API
[ https://issues.apache.org/jira/browse/FLINK-24427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422979#comment-17422979 ] Erwan Loisant commented on FLINK-24427: --- I'm not sure what's the proposed solution here, I don't think hiding the version would really make it possible to use code compiled for Scala 2.12 in 2.13. So I guess either we cross-compile to multiple version (like it was done for 2.11 and 2.12 until now), or we migrate all the code to Java? > Hide Scala in flink-table-planner from API > -- > > Key: FLINK-24427 > URL: https://issues.apache.org/jira/browse/FLINK-24427 > Project: Flink > Issue Type: Sub-task > Components: API / Scala, Table SQL / API, Table SQL / Planner >Reporter: Timo Walther >Priority: Major > > FLIP-32 has decoupled the planner from the API. However, the planner code > base is still the largest Scala code base we have in Flink that we cannot > port to Java easily. > In order to allow arbitrary Scala versions in the user API, we suggest to > hide the Scala version of the planner from the Scala version of the API. The > API is mostly developed in Java and contains only a couple of classes > implemented in Scala. Those should be easier to maintain for various Scala > versions. The planner is already discovered via Java SPI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13414) Add support for Scala 2.13
[ https://issues.apache.org/jira/browse/FLINK-13414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422974#comment-17422974 ] Erwan Loisant commented on FLINK-13414: --- Hi [~MartijnVisser] , I'll give a try to hiding Scala version for flink-table-planner! > Add support for Scala 2.13 > -- > > Key: FLINK-13414 > URL: https://issues.apache.org/jira/browse/FLINK-13414 > Project: Flink > Issue Type: New Feature > Components: API / Scala >Reporter: Chaoran Yu >Priority: Minor > Labels: auto-deprioritized-major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #17396: [FLINK-24393][test] Add CAST tests for type combinations
flinkbot edited a comment on pull request #17396: URL: https://github.com/apache/flink/pull/17396#issuecomment-931316432 ## CI report: * 446362585c664a26aeedf7864f1639134eb90fa9 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24670) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17369: [FLINK-19792][table-planner] Fix IntervalJoin with time equality predicate
flinkbot edited a comment on pull request #17369: URL: https://github.com/apache/flink/pull/17369#issuecomment-929030154 ## CI report: * ae309e02fd4e8221ce56b848934ab700b6a9c304 UNKNOWN * 54c6684aff5bce94116153d15145e6fd43c754bb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24672) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-13414) Add support for Scala 2.13
[ https://issues.apache.org/jira/browse/FLINK-13414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422943#comment-17422943 ] Martijn Visser commented on FLINK-13414: With the help of [~NickBurkard] we managed to drop Scala 2.11 today, after we previously dropped the Scala Shell as well. There are 2 tickets related to opting-out of Scala, one is focussing on tasks that are needed to allow Scala users that use the Java DataStream APIs to use whichever Scala version they want to use. That's https://issues.apache.org/jira/browse/FLINK-23986. One of the sub-tasks of that ticket is to also make this possible for the Java Table API. That's https://issues.apache.org/jira/browse/FLINK-24427. If there's any of you who's interested and willing to help out on those, do let us know. When these two tickets are done, then we'll be in a much better shape and we can then take the next steps for improving the Scala support in Flink. If you can or are willing to help, do let me know. CC [~NickBurkard] [~yuchaoran2011] [~etspaceman] [~eloisant] [~ariskoliopoulos] [~Zhen-hao] [~guenterh] > Add support for Scala 2.13 > -- > > Key: FLINK-13414 > URL: https://issues.apache.org/jira/browse/FLINK-13414 > Project: Flink > Issue Type: New Feature > Components: API / Scala >Reporter: Chaoran Yu >Priority: Minor > Labels: auto-deprioritized-major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #17369: [FLINK-19792][table-planner] Fix IntervalJoin with time equality predicate
flinkbot edited a comment on pull request #17369: URL: https://github.com/apache/flink/pull/17369#issuecomment-929030154 ## CI report: * ae309e02fd4e8221ce56b848934ab700b6a9c304 UNKNOWN * 25cdb2478665779912631293636547e4b73b Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24665) * 54c6684aff5bce94116153d15145e6fd43c754bb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24672) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17384: [FLINK-24388][table] Modules can provide a table source/sink factory
flinkbot edited a comment on pull request #17384: URL: https://github.com/apache/flink/pull/17384#issuecomment-930147821 ## CI report: * f26b7576ac1fe73208acfe3dc424941c39c5a8c2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24666) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17138: [BP-1.14][FLINK-21853][e2e] Harden common_ha.sh#ha_tm_watchdog
flinkbot edited a comment on pull request #17138: URL: https://github.com/apache/flink/pull/17138#issuecomment-912546041 ## CI report: * 2fe1214aa91b0e4016bae9d213187fcae3136f8a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23509) * e2ff9e3ca2f4aa94c6d7666b83771012835ccec0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24680) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17139: [BP-1.13][FLINK-21853][e2e] Harden common_ha.sh#ha_tm_watchdog
flinkbot edited a comment on pull request #17139: URL: https://github.com/apache/flink/pull/17139#issuecomment-912546148 ## CI report: * 777d90a47aa243f9b6fa4a5de8dcf15fc8e4fa40 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23510) * fc899518ebc87cee5afce13d6ec3301de2b26c51 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24681) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17137: [FLINK-21853][e2e] Harden common_ha.sh#ha_tm_watchdog
flinkbot edited a comment on pull request #17137: URL: https://github.com/apache/flink/pull/17137#issuecomment-912545933 ## CI report: * c8c132cb9688244505ed35ad1e61a42db5c8a1ab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23600) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23508) * b836a9082563539235d97241c4f44aff87bb5a7d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24679) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17398: [FLINK-24357][tests] Harden ZooKeeperLeaderElectionConnectionHandlingTest
flinkbot edited a comment on pull request #17398: URL: https://github.com/apache/flink/pull/17398#issuecomment-931472187 ## CI report: * 665b7b1db2c9d0c83329fe3d0ce66e0178c3becc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24675) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17399: [BP-1.14][FLINK-24357][tests] Harden ZooKeeperLeaderElectionConnectionHandlingTest
flinkbot edited a comment on pull request #17399: URL: https://github.com/apache/flink/pull/17399#issuecomment-931472290 ## CI report: * 238bb66a8a84f1545c971b0a9eb87b65fec835bc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24676) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17396: [FLINK-24393][test] Add CAST tests for type combinations
flinkbot edited a comment on pull request #17396: URL: https://github.com/apache/flink/pull/17396#issuecomment-931316432 ## CI report: * 268f980ade47dbc2e0b95f7a734298a0c5823cb2 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24664) * 446362585c664a26aeedf7864f1639134eb90fa9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24670) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17369: [FLINK-19792][table-planner] Fix IntervalJoin with time equality predicate
flinkbot edited a comment on pull request #17369: URL: https://github.com/apache/flink/pull/17369#issuecomment-929030154 ## CI report: * cb02a4b37c6c11ef3b0950d4eb57a7b9bd5f972b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24584) * ae309e02fd4e8221ce56b848934ab700b6a9c304 UNKNOWN * 25cdb2478665779912631293636547e4b73b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24665) * 54c6684aff5bce94116153d15145e6fd43c754bb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24672) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17381: [FLINK-24399][table-common] Make handling of DataType less verbose for connector developers
flinkbot edited a comment on pull request #17381: URL: https://github.com/apache/flink/pull/17381#issuecomment-930061717 ## CI report: * dde53f8081d364a04f69c58d08460f2235ed8258 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24653) * a986056b7d9cb08a4807e588865cacf3376259ee UNKNOWN * a9124ed79554c32c3f8c5c785b3369bf1a49cfe6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24673) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17139: [BP-1.13][FLINK-21853][e2e] Harden common_ha.sh#ha_tm_watchdog
flinkbot edited a comment on pull request #17139: URL: https://github.com/apache/flink/pull/17139#issuecomment-912546148 ## CI report: * 777d90a47aa243f9b6fa4a5de8dcf15fc8e4fa40 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23510) * fc899518ebc87cee5afce13d6ec3301de2b26c51 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17138: [BP-1.14][FLINK-21853][e2e] Harden common_ha.sh#ha_tm_watchdog
flinkbot edited a comment on pull request #17138: URL: https://github.com/apache/flink/pull/17138#issuecomment-912546041 ## CI report: * 2fe1214aa91b0e4016bae9d213187fcae3136f8a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23509) * e2ff9e3ca2f4aa94c6d7666b83771012835ccec0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17137: [FLINK-21853][e2e] Harden common_ha.sh#ha_tm_watchdog
flinkbot edited a comment on pull request #17137: URL: https://github.com/apache/flink/pull/17137#issuecomment-912545933 ## CI report: * c8c132cb9688244505ed35ad1e61a42db5c8a1ab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23600) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23508) * b836a9082563539235d97241c4f44aff87bb5a7d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-24148) Add bloom filter policy option in RocksDBConfiguredOptions
[ https://issues.apache.org/jira/browse/FLINK-24148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Tang resolved FLINK-24148. -- Resolution: Fixed merged in release-1.14: 7882d5e5e7d6c6bd0c12f659b4c92829703a2b29 > Add bloom filter policy option in RocksDBConfiguredOptions > -- > > Key: FLINK-24148 > URL: https://issues.apache.org/jira/browse/FLINK-24148 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Affects Versions: 1.13.2, 1.14.1 >Reporter: Jiayi Liao >Assignee: Jiayi Liao >Priority: Minor > Labels: pull-request-available > Fix For: 1.15.0, 1.14.1 > > > Bloom filter can efficiently enhance the read on RocksDB, especially for the > reading among L0 files. (more details see > https://github.com/facebook/rocksdb/wiki/RocksDB-Bloom-Filter) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-24279) Support withBroadcast with DataStream API in Flink ML Library
[
https://issues.apache.org/jira/browse/FLINK-24279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-24279:
---
Labels: pull-request-available (was: )
> Support withBroadcast with DataStream API in Flink ML Library
> --
>
> Key: FLINK-24279
> URL: https://issues.apache.org/jira/browse/FLINK-24279
> Project: Flink
> Issue Type: New Feature
> Components: Library / Machine Learning
>Reporter: Zhipeng Zhang
>Priority: Major
> Labels: pull-request-available
>
> When doing machine learning using DataStream, we found that DataStream lacks
> withBroadcast() function, which could be useful in machine learning.
>
> A DataSet-based demo is like:
> {code:java}
> DataSet d1 = ...;
> DataSet d2 = ...;
> d1.map(new RichMapFunction () {
> @Override
> public Object map(Object aLong) throws Exception{
> List elements = getRuntimeContext().getBroadcastVariable("d2");
> ...;
>}
> }).withBroadcastSet(d2, "d2");
> {code}
>
> The withBroadcast() function incurs priority-base data-consuming. For example
> in the above code snippet, we cannot consume any element from d1 before we
> consumed all of elements in d2.
>
> Thus when supporting withBroadcast() in DataStream, we also need
> priority-base data-consuming. This could probably lead to deadlock and
> DataStream does not provide a solution for deadlock.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink-ml] zhipeng93 opened a new pull request #15: [FLINK-24279] Support withBroadcast in DataStream by caching in static variables
zhipeng93 opened a new pull request #15: URL: https://github.com/apache/flink-ml/pull/15 This PR supports withBroadcast() function in DataStream by caching the broadcastInputs in static variables. Note that this PR is based on [[FLINK-5][iteration]](https://github.com/apache/flink-ml/pull/11) and also upgraded the flink version to 1.14.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-23519) Aggregate State Backend Latency by State Level
[ https://issues.apache.org/jira/browse/FLINK-23519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Tang resolved FLINK-23519. -- Resolution: Fixed Merged in release-1.14: b5fab64432c9839f956c9c169f617edf1026669f > Aggregate State Backend Latency by State Level > -- > > Key: FLINK-23519 > URL: https://issues.apache.org/jira/browse/FLINK-23519 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics, Runtime / State Backends >Affects Versions: 1.13.0 >Reporter: Mason Chen >Assignee: Yun Tang >Priority: Minor > Labels: pull-request-available > Fix For: 1.13.3, 1.15.0, 1.14.1 > > > To make metrics aggregation easier, there should be a config to expose > something like `state.backend.rocksdb.metrics.column-family-as-variable` that > rocksdb provides to do aggregation across column families > ([https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/#state-backend-rocksdb-metrics-column-family-as-variable]). > > In this case of state backend latency, the variable exposed would be state > level instead column family. This makes it easier to aggregate by the various > state levels that are reported. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #17399: [BP-1.14][FLINK-24357][tests] Harden ZooKeeperLeaderElectionConnectionHandlingTest
flinkbot commented on pull request #17399: URL: https://github.com/apache/flink/pull/17399#issuecomment-931472290 ## CI report: * 238bb66a8a84f1545c971b0a9eb87b65fec835bc UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17398: [FLINK-24357][tests] Harden ZooKeeperLeaderElectionConnectionHandlingTest
flinkbot commented on pull request #17398: URL: https://github.com/apache/flink/pull/17398#issuecomment-931472187 ## CI report: * 665b7b1db2c9d0c83329fe3d0ce66e0178c3becc UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17384: [FLINK-24388][table] Modules can provide a table source/sink factory
flinkbot edited a comment on pull request #17384: URL: https://github.com/apache/flink/pull/17384#issuecomment-930147821 ## CI report: * 1c3a1b045072bcaa0983fc045c79abb3d50ea437 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24662) * f26b7576ac1fe73208acfe3dc424941c39c5a8c2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24666) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17381: [FLINK-24399][table-common] Make handling of DataType less verbose for connector developers
flinkbot edited a comment on pull request #17381: URL: https://github.com/apache/flink/pull/17381#issuecomment-930061717 ## CI report: * dde53f8081d364a04f69c58d08460f2235ed8258 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24653) * a986056b7d9cb08a4807e588865cacf3376259ee UNKNOWN * a9124ed79554c32c3f8c5c785b3369bf1a49cfe6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17395: [FLINK-24410] Upgrade Confluent Platform OSS version in end-to-end tests
flinkbot edited a comment on pull request #17395: URL: https://github.com/apache/flink/pull/17395#issuecomment-931075115 ## CI report: * fd152be7499a7cbc2f31dbea7b543de9fc138424 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24663) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17397: [BP-1.14][FLINK-24117][HA]Remove unHandledErrorListener in ZooKeeperLeaderElectionDriver and ZooKeeperLeaderRetrievalDriver
flinkbot edited a comment on pull request #17397: URL: https://github.com/apache/flink/pull/17397#issuecomment-931434206 ## CI report: * 958ebb4485f9e0f45c65a362542fcb9c56c063de Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24674) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17369: [FLINK-19792][table-planner] Fix IntervalJoin with time equality predicate
flinkbot edited a comment on pull request #17369: URL: https://github.com/apache/flink/pull/17369#issuecomment-929030154 ## CI report: * cb02a4b37c6c11ef3b0950d4eb57a7b9bd5f972b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24584) * ae309e02fd4e8221ce56b848934ab700b6a9c304 UNKNOWN * 25cdb2478665779912631293636547e4b73b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24665) * 54c6684aff5bce94116153d15145e6fd43c754bb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17150: [FLINK-24117][HA]Remove unHandledErrorListener in ZooKeeperLeaderElec…
flinkbot edited a comment on pull request #17150: URL: https://github.com/apache/flink/pull/17150#issuecomment-913084999 ## CI report: * 2abf1ee7133d78e17f6c8ca3481b7d2aa690abd8 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23552) * cc062cf3688d7328bc9388900a566c7c82b33fb0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24671) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] AHeise commented on pull request #17061: [FLINK-23977][elasticsearch] Added DynamicElasticsearchSink for Dynamic ES Cluster Routing
AHeise commented on pull request #17061: URL: https://github.com/apache/flink/pull/17061#issuecomment-931463319 It's very similar to [HybridSource](https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/hybrid/HybridSource.java#L89-L89), and as such should live in flink-connector-base. I'm not entirely sure how to call it. Technically it's a `DemultiplexingSink` (it demultiplexes one stream into several sinks by some criteria). It could be a bit simpler called `SplittingSink`. However, both names sound like you could also implement it by using a static topology with several sinks by demultiplexing in a map with side outputs. So, actually `DynamicSink` captures the dynamic nature better but I'm missing a bit the splitting part... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-24431) [Kinesis][EFO] EAGER registration strategy does not work when job fails over
[
https://issues.apache.org/jira/browse/FLINK-24431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Cranmer reassigned FLINK-24431:
-
Assignee: Rudi Kershaw
> [Kinesis][EFO] EAGER registration strategy does not work when job fails over
>
>
> Key: FLINK-24431
> URL: https://issues.apache.org/jira/browse/FLINK-24431
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: 1.14.0, 1.12.5, 1.13.2
>Reporter: Danny Cranmer
>Assignee: Rudi Kershaw
>Priority: Major
> Fix For: 1.12.6, 1.13.3, 1.15.0, 1.14.1
>
>
> *Background*
> The EFO Kinesis connector will register and de-register stream consumers
> based on the configured [registration
> strategy|https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/datastream/kinesis/#efo-stream-consumer-registrationderegistration].
> When {{EAGER}} is used, the client (usually job manager) will register the
> consumer and then the task managers will de-register the consumer when job
> stops/fails. If the job is configured to restart on fail, then the consumer
> will not exist and the job will continuously fail over.
> *Solution*
> The proposal is to [not deregister the stream
> consumer|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/StreamConsumerRegistrarUtil.java#L88]
> when {{EAGER}} is used. The documentation should be updated to reflect this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24431) [Kinesis][EFO] EAGER registration strategy does not work when job fails over
[
https://issues.apache.org/jira/browse/FLINK-24431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Cranmer updated FLINK-24431:
--
Description:
*Background*
The EFO Kinesis connector will register and de-register stream consumers based
on the configured [registration
strategy|https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/datastream/kinesis/#efo-stream-consumer-registrationderegistration].
When {{EAGER}} is used, the client (usually job manager) will register the
consumer and then the task managers will de-register the consumer when job
stops/fails. If the job is configured to restart on fail, then the consumer
will not exist and the job will continuously fail over.
*Solution*
The proposal is to [not deregister the stream
consumer|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/StreamConsumerRegistrarUtil.java#L88]
when {{EAGER}} is used. The documentation should be updated to reflect this.
was:
*Background*
The EFO Kinesis connector will register and de-register stream consumers based
on the configured [registration
strategy|https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/datastream/kinesis/#efo-stream-consumer-registrationderegistration].
When {{EAGER}} is used, the client (usually job manager) will register the
consumer and then the task managers will de-register the consumer when job
stops/fails. If the job is configured to restart on fail, then the consumer
will not exist and the job will continuously fail over.
*Solution*
The proposal is to not deregister the stream consumer when {{EAGER}} is used.
The documentation should be updated to reflect this.
> [Kinesis][EFO] EAGER registration strategy does not work when job fails over
>
>
> Key: FLINK-24431
> URL: https://issues.apache.org/jira/browse/FLINK-24431
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: 1.14.0, 1.12.5, 1.13.2
>Reporter: Danny Cranmer
>Priority: Major
> Fix For: 1.12.6, 1.13.3, 1.15.0, 1.14.1
>
>
> *Background*
> The EFO Kinesis connector will register and de-register stream consumers
> based on the configured [registration
> strategy|https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/datastream/kinesis/#efo-stream-consumer-registrationderegistration].
> When {{EAGER}} is used, the client (usually job manager) will register the
> consumer and then the task managers will de-register the consumer when job
> stops/fails. If the job is configured to restart on fail, then the consumer
> will not exist and the job will continuously fail over.
> *Solution*
> The proposal is to [not deregister the stream
> consumer|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/util/StreamConsumerRegistrarUtil.java#L88]
> when {{EAGER}} is used. The documentation should be updated to reflect this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-24431) [Kinesis][EFO] EAGER registration strategy does not work when job fails over
Danny Cranmer created FLINK-24431:
-
Summary: [Kinesis][EFO] EAGER registration strategy does not work
when job fails over
Key: FLINK-24431
URL: https://issues.apache.org/jira/browse/FLINK-24431
Project: Flink
Issue Type: Bug
Components: Connectors / Kinesis
Affects Versions: 1.13.2, 1.12.5, 1.14.0
Reporter: Danny Cranmer
Fix For: 1.12.6, 1.13.3, 1.15.0, 1.14.1
*Background*
The EFO Kinesis connector will register and de-register stream consumers based
on the configured [registration
strategy|https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/datastream/kinesis/#efo-stream-consumer-registrationderegistration].
When {{EAGER}} is used, the client (usually job manager) will register the
consumer and then the task managers will de-register the consumer when job
stops/fails. If the job is configured to restart on fail, then the consumer
will not exist and the job will continuously fail over.
*Solution*
The proposal is to not deregister the stream consumer when {{EAGER}} is used.
The documentation should be updated to reflect this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] rionmonster commented on pull request #17061: [FLINK-23977][elasticsearch] Added DynamicElasticsearchSink for Dynamic ES Cluster Routing
rionmonster commented on pull request #17061: URL: https://github.com/apache/flink/pull/17061#issuecomment-931445551 @AHeise Yeah, I think I much prefer that. I can adjust the JIRA ticket to reflect the more generic approach and remove this effort as it doesn't make a ton of sense if we elect to adopt a more holistic approach. I'd love to start working on this as I'd imagine it'd be primarily interfaces (`DynamicSink?`, `SinkRouter`) along with some corresponding docs that explain basic usage. Where in the codebase would something like this live? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on pull request #17137: [FLINK-21853][e2e] Harden common_ha.sh#ha_tm_watchdog
tillrohrmann commented on pull request #17137: URL: https://github.com/apache/flink/pull/17137#issuecomment-931443588 Thanks for the review @dmvk. I've addressed your comment and rebased the PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #17137: [FLINK-21853][e2e] Harden common_ha.sh#ha_tm_watchdog
tillrohrmann commented on a change in pull request #17137:
URL: https://github.com/apache/flink/pull/17137#discussion_r719536807
##
File path: flink-end-to-end-tests/test-scripts/common_ha.sh
##
@@ -127,6 +127,18 @@ function kill_single {
echo "Killed JM @ ${PID}"
}
+function start_expected_num_tms() {
+ local EXPECTED_TMS=$1
+
+ local RUNNING_TMS=`jps | grep 'TaskManager' | wc -l`
+
+ while [ "${RUNNING_TMS}" -lt "${EXPECTED_TMS}" ]; do
+ echo "Starting new TM."
+ "$FLINK_DIR"/bin/taskmanager.sh start > /dev/null
+ RUNNING_TMS=$(( $RUNNING_TMS + 1 ))
Review comment:
Good catch. Thanks a lot!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17397: [BP-1.14][FLINK-24117][HA]Remove unHandledErrorListener in ZooKeeperLeaderElectionDriver and ZooKeeperLeaderRetrievalDriver
flinkbot commented on pull request #17397: URL: https://github.com/apache/flink/pull/17397#issuecomment-931434206 ## CI report: * 958ebb4485f9e0f45c65a362542fcb9c56c063de UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17396: [FLINK-24393][test] Add CAST tests for type combinations
flinkbot edited a comment on pull request #17396: URL: https://github.com/apache/flink/pull/17396#issuecomment-931316432 ## CI report: * 268f980ade47dbc2e0b95f7a734298a0c5823cb2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24664) * 446362585c664a26aeedf7864f1639134eb90fa9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24670) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17381: [FLINK-24399][table-common] Make handling of DataType less verbose for connector developers
flinkbot edited a comment on pull request #17381: URL: https://github.com/apache/flink/pull/17381#issuecomment-930061717 ## CI report: * dde53f8081d364a04f69c58d08460f2235ed8258 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24653) * a986056b7d9cb08a4807e588865cacf3376259ee UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-24357) ZooKeeperLeaderElectionConnectionHandlingTest#testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled fails with an Unhandled error
[
https://issues.apache.org/jira/browse/FLINK-24357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-24357:
---
Labels: pull-request-available test-stability (was: test-stability)
> ZooKeeperLeaderElectionConnectionHandlingTest#testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled
> fails with an Unhandled error
> -
>
> Key: FLINK-24357
> URL: https://issues.apache.org/jira/browse/FLINK-24357
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Piotr Nowojski
>Assignee: Till Rohrmann
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0, 1.14.1
>
> Attachments: logs-ci_build-test_ci_build_core-1632297723.zip
>
>
> In a [private
> azure|https://dev.azure.com/pnowojski/637f6470-2732-4605-a304-caebd40e284b/_apis/build/builds/517/logs/155]
> build when testing my own PR I've noticed the following error that looks
> unrelated to any of my changes (modifications to {{Task}} class
> error/cancellation handling logic):
> {noformat}
> 2021-09-22T08:09:16.6244936Z Sep 22 08:09:16 [ERROR]
> testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled
> Time elapsed: 28.753 s <<< FAILURE!
> 2021-09-22T08:09:16.6245821Z Sep 22 08:09:16 java.lang.AssertionError: The
> TestingFatalErrorHandler caught an exception.
> 2021-09-22T08:09:16.6246513Z Sep 22 08:09:16 at
> org.apache.flink.runtime.util.TestingFatalErrorHandlerResource.after(TestingFatalErrorHandlerResource.java:78)
> 2021-09-22T08:09:16.6247281Z Sep 22 08:09:16 at
> org.apache.flink.runtime.util.TestingFatalErrorHandlerResource.access$300(TestingFatalErrorHandlerResource.java:33)
> 2021-09-22T08:09:16.6248167Z Sep 22 08:09:16 at
> org.apache.flink.runtime.util.TestingFatalErrorHandlerResource$1.evaluate(TestingFatalErrorHandlerResource.java:57)
> 2021-09-22T08:09:16.6248862Z Sep 22 08:09:16 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2021-09-22T08:09:16.6249620Z Sep 22 08:09:16 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-09-22T08:09:16.6250210Z Sep 22 08:09:16 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2021-09-22T08:09:16.6250773Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-09-22T08:09:16.6251375Z Sep 22 08:09:16 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2021-09-22T08:09:16.6251951Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2021-09-22T08:09:16.6252562Z Sep 22 08:09:16 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2021-09-22T08:09:16.6253415Z Sep 22 08:09:16 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2021-09-22T08:09:16.6254469Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2021-09-22T08:09:16.6255039Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2021-09-22T08:09:16.6256238Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2021-09-22T08:09:16.6257109Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 2021-09-22T08:09:16.6257766Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> 2021-09-22T08:09:16.6258406Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-09-22T08:09:16.6259050Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> 2021-09-22T08:09:16.6259827Z Sep 22 08:09:16 at
> org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> 2021-09-22T08:09:16.6260963Z Sep 22 08:09:16 at
> org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> 2021-09-22T08:09:16.6261796Z Sep 22 08:09:16 at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
> 2021-09-22T08:09:16.6262428Z Sep 22 08:09:16 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> 2021-09-22T08:09:16.6263268Z Sep 22 08:09:16 at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> 2021-09-22T08:09:16.6263875Z Sep 22 08:09:16 at
> java.util.Iterator.forEachRemaining(Iterator.java:116)
> 2021-09-22T08:09:16.6265025Z Sep 22 08:09:16 at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> 2021-09-22T08:09:16.6265940Z Sep 22 08:09:16 at
>
[GitHub] [flink] flinkbot edited a comment on pull request #17150: [FLINK-24117][HA]Remove unHandledErrorListener in ZooKeeperLeaderElec…
flinkbot edited a comment on pull request #17150: URL: https://github.com/apache/flink/pull/17150#issuecomment-913084999 ## CI report: * 2abf1ee7133d78e17f6c8ca3481b7d2aa690abd8 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23552) * cc062cf3688d7328bc9388900a566c7c82b33fb0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17399: [BP-1.14][FLINK-24357][tests] Harden ZooKeeperLeaderElectionConnectionHandlingTest
flinkbot commented on pull request #17399: URL: https://github.com/apache/flink/pull/17399#issuecomment-931432441 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 238bb66a8a84f1545c971b0a9eb87b65fec835bc (Thu Sep 30 15:32:51 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17398: [FLINK-24357][tests] Harden ZooKeeperLeaderElectionConnectionHandlingTest
flinkbot commented on pull request #17398: URL: https://github.com/apache/flink/pull/17398#issuecomment-931432461 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 665b7b1db2c9d0c83329fe3d0ce66e0178c3becc (Thu Sep 30 15:32:53 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on pull request #15229: [FLINK-19142][runtime] Fix slot hijacking after task failover
tillrohrmann commented on pull request #15229: URL: https://github.com/apache/flink/pull/15229#issuecomment-931431652 Are you working on this problem @zhuzhurk? If not, then we should maybe close this PR and reopen it once you resume working on it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-24357) ZooKeeperLeaderElectionConnectionHandlingTest#testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled fails with an Unhandled error
[
https://issues.apache.org/jira/browse/FLINK-24357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-24357:
--
Labels: test-stability (was: pull-request-available)
> ZooKeeperLeaderElectionConnectionHandlingTest#testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled
> fails with an Unhandled error
> -
>
> Key: FLINK-24357
> URL: https://issues.apache.org/jira/browse/FLINK-24357
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Piotr Nowojski
>Assignee: Till Rohrmann
>Priority: Major
> Labels: test-stability
> Fix For: 1.15.0, 1.14.1
>
> Attachments: logs-ci_build-test_ci_build_core-1632297723.zip
>
>
> In a [private
> azure|https://dev.azure.com/pnowojski/637f6470-2732-4605-a304-caebd40e284b/_apis/build/builds/517/logs/155]
> build when testing my own PR I've noticed the following error that looks
> unrelated to any of my changes (modifications to {{Task}} class
> error/cancellation handling logic):
> {noformat}
> 2021-09-22T08:09:16.6244936Z Sep 22 08:09:16 [ERROR]
> testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled
> Time elapsed: 28.753 s <<< FAILURE!
> 2021-09-22T08:09:16.6245821Z Sep 22 08:09:16 java.lang.AssertionError: The
> TestingFatalErrorHandler caught an exception.
> 2021-09-22T08:09:16.6246513Z Sep 22 08:09:16 at
> org.apache.flink.runtime.util.TestingFatalErrorHandlerResource.after(TestingFatalErrorHandlerResource.java:78)
> 2021-09-22T08:09:16.6247281Z Sep 22 08:09:16 at
> org.apache.flink.runtime.util.TestingFatalErrorHandlerResource.access$300(TestingFatalErrorHandlerResource.java:33)
> 2021-09-22T08:09:16.6248167Z Sep 22 08:09:16 at
> org.apache.flink.runtime.util.TestingFatalErrorHandlerResource$1.evaluate(TestingFatalErrorHandlerResource.java:57)
> 2021-09-22T08:09:16.6248862Z Sep 22 08:09:16 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2021-09-22T08:09:16.6249620Z Sep 22 08:09:16 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-09-22T08:09:16.6250210Z Sep 22 08:09:16 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2021-09-22T08:09:16.6250773Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-09-22T08:09:16.6251375Z Sep 22 08:09:16 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2021-09-22T08:09:16.6251951Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2021-09-22T08:09:16.6252562Z Sep 22 08:09:16 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2021-09-22T08:09:16.6253415Z Sep 22 08:09:16 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2021-09-22T08:09:16.6254469Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2021-09-22T08:09:16.6255039Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2021-09-22T08:09:16.6256238Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2021-09-22T08:09:16.6257109Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 2021-09-22T08:09:16.6257766Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> 2021-09-22T08:09:16.6258406Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-09-22T08:09:16.6259050Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> 2021-09-22T08:09:16.6259827Z Sep 22 08:09:16 at
> org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> 2021-09-22T08:09:16.6260963Z Sep 22 08:09:16 at
> org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> 2021-09-22T08:09:16.6261796Z Sep 22 08:09:16 at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
> 2021-09-22T08:09:16.6262428Z Sep 22 08:09:16 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> 2021-09-22T08:09:16.6263268Z Sep 22 08:09:16 at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> 2021-09-22T08:09:16.6263875Z Sep 22 08:09:16 at
> java.util.Iterator.forEachRemaining(Iterator.java:116)
> 2021-09-22T08:09:16.6265025Z Sep 22 08:09:16 at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> 2021-09-22T08:09:16.6265940Z Sep 22 08:09:16 at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
>
[jira] [Updated] (FLINK-24357) ZooKeeperLeaderElectionConnectionHandlingTest#testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled fails with an Unhandled error
[
https://issues.apache.org/jira/browse/FLINK-24357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-24357:
---
Labels: pull-request-available (was: )
> ZooKeeperLeaderElectionConnectionHandlingTest#testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled
> fails with an Unhandled error
> -
>
> Key: FLINK-24357
> URL: https://issues.apache.org/jira/browse/FLINK-24357
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Piotr Nowojski
>Assignee: Till Rohrmann
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0, 1.14.1
>
> Attachments: logs-ci_build-test_ci_build_core-1632297723.zip
>
>
> In a [private
> azure|https://dev.azure.com/pnowojski/637f6470-2732-4605-a304-caebd40e284b/_apis/build/builds/517/logs/155]
> build when testing my own PR I've noticed the following error that looks
> unrelated to any of my changes (modifications to {{Task}} class
> error/cancellation handling logic):
> {noformat}
> 2021-09-22T08:09:16.6244936Z Sep 22 08:09:16 [ERROR]
> testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled
> Time elapsed: 28.753 s <<< FAILURE!
> 2021-09-22T08:09:16.6245821Z Sep 22 08:09:16 java.lang.AssertionError: The
> TestingFatalErrorHandler caught an exception.
> 2021-09-22T08:09:16.6246513Z Sep 22 08:09:16 at
> org.apache.flink.runtime.util.TestingFatalErrorHandlerResource.after(TestingFatalErrorHandlerResource.java:78)
> 2021-09-22T08:09:16.6247281Z Sep 22 08:09:16 at
> org.apache.flink.runtime.util.TestingFatalErrorHandlerResource.access$300(TestingFatalErrorHandlerResource.java:33)
> 2021-09-22T08:09:16.6248167Z Sep 22 08:09:16 at
> org.apache.flink.runtime.util.TestingFatalErrorHandlerResource$1.evaluate(TestingFatalErrorHandlerResource.java:57)
> 2021-09-22T08:09:16.6248862Z Sep 22 08:09:16 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2021-09-22T08:09:16.6249620Z Sep 22 08:09:16 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-09-22T08:09:16.6250210Z Sep 22 08:09:16 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2021-09-22T08:09:16.6250773Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-09-22T08:09:16.6251375Z Sep 22 08:09:16 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2021-09-22T08:09:16.6251951Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2021-09-22T08:09:16.6252562Z Sep 22 08:09:16 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2021-09-22T08:09:16.6253415Z Sep 22 08:09:16 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2021-09-22T08:09:16.6254469Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2021-09-22T08:09:16.6255039Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2021-09-22T08:09:16.6256238Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2021-09-22T08:09:16.6257109Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 2021-09-22T08:09:16.6257766Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> 2021-09-22T08:09:16.6258406Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-09-22T08:09:16.6259050Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> 2021-09-22T08:09:16.6259827Z Sep 22 08:09:16 at
> org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> 2021-09-22T08:09:16.6260963Z Sep 22 08:09:16 at
> org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> 2021-09-22T08:09:16.6261796Z Sep 22 08:09:16 at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
> 2021-09-22T08:09:16.6262428Z Sep 22 08:09:16 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> 2021-09-22T08:09:16.6263268Z Sep 22 08:09:16 at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> 2021-09-22T08:09:16.6263875Z Sep 22 08:09:16 at
> java.util.Iterator.forEachRemaining(Iterator.java:116)
> 2021-09-22T08:09:16.6265025Z Sep 22 08:09:16 at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> 2021-09-22T08:09:16.6265940Z Sep 22 08:09:16 at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
>
[GitHub] [flink] tillrohrmann opened a new pull request #17399: [BP-1.14][FLINK-24357][tests] Harden ZooKeeperLeaderElectionConnectionHandlingTest
tillrohrmann opened a new pull request #17399: URL: https://github.com/apache/flink/pull/17399 Backport of #17398 to `release-1.14`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] AHeise commented on pull request #17061: [FLINK-23977][elasticsearch] Added DynamicElasticsearchSink for Dynamic ES Cluster Routing
AHeise commented on pull request #17061: URL: https://github.com/apache/flink/pull/17061#issuecomment-931429813 To open it to more use cases, we pretty much just need to replace the lower bounds with a more generic interface. You are not using much from the ESSink directly. However, we would first need to clarify if we want to switch to the new [`Sink` interface. ](https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/api/connector/sink/Sink.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann opened a new pull request #17398: [FLINK-24357][tests] Harden ZooKeeperLeaderElectionConnectionHandlingTest
tillrohrmann opened a new pull request #17398: URL: https://github.com/apache/flink/pull/17398 This commit hardens ZooKeeperLeaderElectionConnectionHandlingTest.testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled by allowing that exceptions can occur if the connection to ZooKeeper is lost. We do this by clearing the fatal error handler at the end of the test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-24357) ZooKeeperLeaderElectionConnectionHandlingTest#testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled fails with an Unhandled error
[
https://issues.apache.org/jira/browse/FLINK-24357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-24357:
--
Fix Version/s: 1.14.1
1.15.0
> ZooKeeperLeaderElectionConnectionHandlingTest#testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled
> fails with an Unhandled error
> -
>
> Key: FLINK-24357
> URL: https://issues.apache.org/jira/browse/FLINK-24357
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Piotr Nowojski
>Priority: Major
> Fix For: 1.15.0, 1.14.1
>
> Attachments: logs-ci_build-test_ci_build_core-1632297723.zip
>
>
> In a [private
> azure|https://dev.azure.com/pnowojski/637f6470-2732-4605-a304-caebd40e284b/_apis/build/builds/517/logs/155]
> build when testing my own PR I've noticed the following error that looks
> unrelated to any of my changes (modifications to {{Task}} class
> error/cancellation handling logic):
> {noformat}
> 2021-09-22T08:09:16.6244936Z Sep 22 08:09:16 [ERROR]
> testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled
> Time elapsed: 28.753 s <<< FAILURE!
> 2021-09-22T08:09:16.6245821Z Sep 22 08:09:16 java.lang.AssertionError: The
> TestingFatalErrorHandler caught an exception.
> 2021-09-22T08:09:16.6246513Z Sep 22 08:09:16 at
> org.apache.flink.runtime.util.TestingFatalErrorHandlerResource.after(TestingFatalErrorHandlerResource.java:78)
> 2021-09-22T08:09:16.6247281Z Sep 22 08:09:16 at
> org.apache.flink.runtime.util.TestingFatalErrorHandlerResource.access$300(TestingFatalErrorHandlerResource.java:33)
> 2021-09-22T08:09:16.6248167Z Sep 22 08:09:16 at
> org.apache.flink.runtime.util.TestingFatalErrorHandlerResource$1.evaluate(TestingFatalErrorHandlerResource.java:57)
> 2021-09-22T08:09:16.6248862Z Sep 22 08:09:16 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2021-09-22T08:09:16.6249620Z Sep 22 08:09:16 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-09-22T08:09:16.6250210Z Sep 22 08:09:16 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2021-09-22T08:09:16.6250773Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-09-22T08:09:16.6251375Z Sep 22 08:09:16 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2021-09-22T08:09:16.6251951Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2021-09-22T08:09:16.6252562Z Sep 22 08:09:16 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2021-09-22T08:09:16.6253415Z Sep 22 08:09:16 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2021-09-22T08:09:16.6254469Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2021-09-22T08:09:16.6255039Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2021-09-22T08:09:16.6256238Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2021-09-22T08:09:16.6257109Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 2021-09-22T08:09:16.6257766Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> 2021-09-22T08:09:16.6258406Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-09-22T08:09:16.6259050Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> 2021-09-22T08:09:16.6259827Z Sep 22 08:09:16 at
> org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> 2021-09-22T08:09:16.6260963Z Sep 22 08:09:16 at
> org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> 2021-09-22T08:09:16.6261796Z Sep 22 08:09:16 at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
> 2021-09-22T08:09:16.6262428Z Sep 22 08:09:16 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> 2021-09-22T08:09:16.6263268Z Sep 22 08:09:16 at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> 2021-09-22T08:09:16.6263875Z Sep 22 08:09:16 at
> java.util.Iterator.forEachRemaining(Iterator.java:116)
> 2021-09-22T08:09:16.6265025Z Sep 22 08:09:16 at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> 2021-09-22T08:09:16.6265940Z Sep 22 08:09:16 at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> 2021-09-22T08:09:16.6266767Z Sep 22 08:09:16 at
>
[jira] [Assigned] (FLINK-24357) ZooKeeperLeaderElectionConnectionHandlingTest#testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled fails with an Unhandled error
[
https://issues.apache.org/jira/browse/FLINK-24357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann reassigned FLINK-24357:
-
Assignee: Till Rohrmann
> ZooKeeperLeaderElectionConnectionHandlingTest#testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled
> fails with an Unhandled error
> -
>
> Key: FLINK-24357
> URL: https://issues.apache.org/jira/browse/FLINK-24357
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Piotr Nowojski
>Assignee: Till Rohrmann
>Priority: Major
> Fix For: 1.15.0, 1.14.1
>
> Attachments: logs-ci_build-test_ci_build_core-1632297723.zip
>
>
> In a [private
> azure|https://dev.azure.com/pnowojski/637f6470-2732-4605-a304-caebd40e284b/_apis/build/builds/517/logs/155]
> build when testing my own PR I've noticed the following error that looks
> unrelated to any of my changes (modifications to {{Task}} class
> error/cancellation handling logic):
> {noformat}
> 2021-09-22T08:09:16.6244936Z Sep 22 08:09:16 [ERROR]
> testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled
> Time elapsed: 28.753 s <<< FAILURE!
> 2021-09-22T08:09:16.6245821Z Sep 22 08:09:16 java.lang.AssertionError: The
> TestingFatalErrorHandler caught an exception.
> 2021-09-22T08:09:16.6246513Z Sep 22 08:09:16 at
> org.apache.flink.runtime.util.TestingFatalErrorHandlerResource.after(TestingFatalErrorHandlerResource.java:78)
> 2021-09-22T08:09:16.6247281Z Sep 22 08:09:16 at
> org.apache.flink.runtime.util.TestingFatalErrorHandlerResource.access$300(TestingFatalErrorHandlerResource.java:33)
> 2021-09-22T08:09:16.6248167Z Sep 22 08:09:16 at
> org.apache.flink.runtime.util.TestingFatalErrorHandlerResource$1.evaluate(TestingFatalErrorHandlerResource.java:57)
> 2021-09-22T08:09:16.6248862Z Sep 22 08:09:16 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2021-09-22T08:09:16.6249620Z Sep 22 08:09:16 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-09-22T08:09:16.6250210Z Sep 22 08:09:16 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2021-09-22T08:09:16.6250773Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-09-22T08:09:16.6251375Z Sep 22 08:09:16 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2021-09-22T08:09:16.6251951Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2021-09-22T08:09:16.6252562Z Sep 22 08:09:16 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2021-09-22T08:09:16.6253415Z Sep 22 08:09:16 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2021-09-22T08:09:16.6254469Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2021-09-22T08:09:16.6255039Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2021-09-22T08:09:16.6256238Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2021-09-22T08:09:16.6257109Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 2021-09-22T08:09:16.6257766Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> 2021-09-22T08:09:16.6258406Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-09-22T08:09:16.6259050Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> 2021-09-22T08:09:16.6259827Z Sep 22 08:09:16 at
> org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> 2021-09-22T08:09:16.6260963Z Sep 22 08:09:16 at
> org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> 2021-09-22T08:09:16.6261796Z Sep 22 08:09:16 at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
> 2021-09-22T08:09:16.6262428Z Sep 22 08:09:16 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> 2021-09-22T08:09:16.6263268Z Sep 22 08:09:16 at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> 2021-09-22T08:09:16.6263875Z Sep 22 08:09:16 at
> java.util.Iterator.forEachRemaining(Iterator.java:116)
> 2021-09-22T08:09:16.6265025Z Sep 22 08:09:16 at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> 2021-09-22T08:09:16.6265940Z Sep 22 08:09:16 at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> 2021-09-22T08:09:16.6266767Z Sep 22 08:09:16 at
>
[GitHub] [flink] rmetzger commented on pull request #16166: [FLINK-23000] Add log bundler to collect and compress logs
rmetzger commented on pull request #16166: URL: https://github.com/apache/flink/pull/16166#issuecomment-931423456 FYI: I was hoping to find some time to address the last outstanding comments before my parental leave, but I won't be able to. Maybe I find time in December to look into this again. If somebody else wants to take over this work, they are more than welcome to do so. I'm happy to occasionally respond to questions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] slinkydeveloper commented on a change in pull request #17369: [FLINK-19792][table-planner] Fix IntervalJoin with time equality predicate
slinkydeveloper commented on a change in pull request #17369:
URL: https://github.com/apache/flink/pull/17369#discussion_r719517375
##
File path:
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/IntervalJoinJsonPlanTest.java
##
@@ -82,18 +82,19 @@ public void testProcessingTimeInnerJoinWithOnClause() {
@Test
public void testRowTimeInnerJoinWithOnClause() {
-String sinkTableDdl =
-"CREATE TABLE MySink (\n"
-+ " a int,\n"
-+ " b varchar\n"
-+ ") with (\n"
-+ " 'connector' = 'values',\n"
-+ " 'table-sink-class' = 'DEFAULT')";
-tEnv.executeSql(sinkTableDdl);
util.verifyJsonPlan(
"INSERT INTO MySink "
+ "SELECT t1.a, t2.b FROM A t1 JOIN B t2 ON\n"
+ " t1.a = t2.a AND\n"
+ " t1.rowtime BETWEEN t2.rowtime - INTERVAL '10'
SECOND AND t2.rowtime + INTERVAL '1' HOUR");
}
+
+@Test
+public void testRowTimeEquiJoin() {
Review comment:
We have a test for that in the xml tests
https://github.com/apache/flink/pull/17369/files#diff-32daba6c5665d7991e66e5f5b3754c519e565557e7dafb11109a1cfea8e72dffR199.
The json test is removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-24387) Support a JSON_STRING
[
https://issues.apache.org/jira/browse/FLINK-24387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422843#comment-17422843
]
Timo Walther commented on FLINK-24387:
--
+1 to JSON_STRING
> Support a JSON_STRING
> -
>
> Key: FLINK-24387
> URL: https://issues.apache.org/jira/browse/FLINK-24387
> Project: Flink
> Issue Type: Sub-task
>Reporter: Ingo Bürk
>Assignee: Ingo Bürk
>Priority: Major
>
> We should consider adding a JSON_STRING function which can simply convert
> some (supported) type directly into a JSON representation, e.g. the
> equivalent of doing
> {code:java}
> JSON_QUERY(JSON_ARRAY(x), '$.[0]'){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-24357) ZooKeeperLeaderElectionConnectionHandlingTest#testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled fails with an Unhandled error
[
https://issues.apache.org/jira/browse/FLINK-24357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422838#comment-17422838
]
Till Rohrmann commented on FLINK-24357:
---
I think the problem is a test instability. In the test we stop the ZooKeeper
server in order to provoke a lost connection. This of course can result into
some {{KeeperException$ConnectionLossException}} if we did not manage to create
the directories for the {{TreeCache}} in the meantime. This should not be
problematic and should be handled in the test case.
> ZooKeeperLeaderElectionConnectionHandlingTest#testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled
> fails with an Unhandled error
> -
>
> Key: FLINK-24357
> URL: https://issues.apache.org/jira/browse/FLINK-24357
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Piotr Nowojski
>Priority: Major
> Attachments: logs-ci_build-test_ci_build_core-1632297723.zip
>
>
> In a [private
> azure|https://dev.azure.com/pnowojski/637f6470-2732-4605-a304-caebd40e284b/_apis/build/builds/517/logs/155]
> build when testing my own PR I've noticed the following error that looks
> unrelated to any of my changes (modifications to {{Task}} class
> error/cancellation handling logic):
> {noformat}
> 2021-09-22T08:09:16.6244936Z Sep 22 08:09:16 [ERROR]
> testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled
> Time elapsed: 28.753 s <<< FAILURE!
> 2021-09-22T08:09:16.6245821Z Sep 22 08:09:16 java.lang.AssertionError: The
> TestingFatalErrorHandler caught an exception.
> 2021-09-22T08:09:16.6246513Z Sep 22 08:09:16 at
> org.apache.flink.runtime.util.TestingFatalErrorHandlerResource.after(TestingFatalErrorHandlerResource.java:78)
> 2021-09-22T08:09:16.6247281Z Sep 22 08:09:16 at
> org.apache.flink.runtime.util.TestingFatalErrorHandlerResource.access$300(TestingFatalErrorHandlerResource.java:33)
> 2021-09-22T08:09:16.6248167Z Sep 22 08:09:16 at
> org.apache.flink.runtime.util.TestingFatalErrorHandlerResource$1.evaluate(TestingFatalErrorHandlerResource.java:57)
> 2021-09-22T08:09:16.6248862Z Sep 22 08:09:16 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2021-09-22T08:09:16.6249620Z Sep 22 08:09:16 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-09-22T08:09:16.6250210Z Sep 22 08:09:16 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2021-09-22T08:09:16.6250773Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-09-22T08:09:16.6251375Z Sep 22 08:09:16 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2021-09-22T08:09:16.6251951Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2021-09-22T08:09:16.6252562Z Sep 22 08:09:16 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2021-09-22T08:09:16.6253415Z Sep 22 08:09:16 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2021-09-22T08:09:16.6254469Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2021-09-22T08:09:16.6255039Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2021-09-22T08:09:16.6256238Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2021-09-22T08:09:16.6257109Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 2021-09-22T08:09:16.6257766Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> 2021-09-22T08:09:16.6258406Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-09-22T08:09:16.6259050Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> 2021-09-22T08:09:16.6259827Z Sep 22 08:09:16 at
> org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> 2021-09-22T08:09:16.6260963Z Sep 22 08:09:16 at
> org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> 2021-09-22T08:09:16.6261796Z Sep 22 08:09:16 at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
> 2021-09-22T08:09:16.6262428Z Sep 22 08:09:16 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> 2021-09-22T08:09:16.6263268Z Sep 22 08:09:16 at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> 2021-09-22T08:09:16.6263875Z Sep 22 08:09:16 at
> java.util.Iterator.forEachRemaining(Iterator.java:116)
> 2021-09-22T08:09:16.6265025Z
[GitHub] [flink] twalthr closed pull request #10838: [hotfix][table-common] Simplify formatted exceptions
twalthr closed pull request #10838: URL: https://github.com/apache/flink/pull/10838 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] twalthr commented on pull request #10838: [hotfix][table-common] Simplify formatted exceptions
twalthr commented on pull request #10838: URL: https://github.com/apache/flink/pull/10838#issuecomment-931414163 Closing this as it has never got enough attention. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] twalthr commented on a change in pull request #17369: [FLINK-19792][table-planner] Fix IntervalJoin with time equality predicate
twalthr commented on a change in pull request #17369:
URL: https://github.com/apache/flink/pull/17369#discussion_r719504153
##
File path:
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/IntervalJoinJsonPlanTest.java
##
@@ -82,18 +82,19 @@ public void testProcessingTimeInnerJoinWithOnClause() {
@Test
public void testRowTimeInnerJoinWithOnClause() {
-String sinkTableDdl =
-"CREATE TABLE MySink (\n"
-+ " a int,\n"
-+ " b varchar\n"
-+ ") with (\n"
-+ " 'connector' = 'values',\n"
-+ " 'table-sink-class' = 'DEFAULT')";
-tEnv.executeSql(sinkTableDdl);
util.verifyJsonPlan(
"INSERT INTO MySink "
+ "SELECT t1.a, t2.b FROM A t1 JOIN B t2 ON\n"
+ " t1.a = t2.a AND\n"
+ " t1.rowtime BETWEEN t2.rowtime - INTERVAL '10'
SECOND AND t2.rowtime + INTERVAL '1' HOUR");
}
+
+@Test
+public void testRowTimeEquiJoin() {
Review comment:
We can drop this test. A regular plan test via xml should be sufficient
compared to a full JSON plan. If not already present, you can add this to the
test that verifies `IntervalJoinTest.xml`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17397: [BP-1.14][FLINK-24117][HA]Remove unHandledErrorListener in ZooKeeperLeaderElectionDriver and ZooKeeperLeaderRetrievalDriver
flinkbot commented on pull request #17397: URL: https://github.com/apache/flink/pull/17397#issuecomment-931406025 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 958ebb4485f9e0f45c65a362542fcb9c56c063de (Thu Sep 30 15:04:10 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] akalash commented on a change in pull request #17354: [FLINK-24200][streaming] Calculating maximum alignment time rather than using the constant value
akalash commented on a change in pull request #17354:
URL: https://github.com/apache/flink/pull/17354#discussion_r719499492
##
File path:
flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/io/CheckpointBarrierTrackerTest.java
##
@@ -607,13 +606,26 @@ public void testTwoLastBarriersOneByOne() throws
Exception {
ValidatingCheckpointHandler validator = new
ValidatingCheckpointHandler();
inputGate = createCheckpointedInputGate(2, sequence, validator);
-for (BufferOrEvent boe : sequence) {
-assertEquals(boe, inputGate.pollNext().get());
-Thread.sleep(10);
-}
+// start checkpoint 1
+assertEquals(sequence[0], inputGate.pollNext().get());
+Thread.sleep(10);
+
+// start checkpoint 2
+long start = System.currentTimeMillis();
+assertEquals(sequence[1], inputGate.pollNext().get());
+Thread.sleep(1);
Review comment:
We don't need the high sleep time here. I use 1 ms as protection against
zero alignment time. So I just want to have an alignment time of more than zero
and 1 ms is enough
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
