[GitHub] [flink] flinkbot edited a comment on pull request #17610: [hotfix][docs] Fix the outdated upgrading instruction for Kafka Connector
flinkbot edited a comment on pull request #17610: URL: https://github.com/apache/flink/pull/17610#issuecomment-955131667 ## CI report: * 00214af0b13e7695db3b7c30d5b53edafae1f63a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25665) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17608: [FLINK-24550][rpc] Use ContextClassLoader for message deserialization
flinkbot edited a comment on pull request #17608: URL: https://github.com/apache/flink/pull/17608#issuecomment-954803172 ## CI report: * e6d48a2a8d555b42b1720db653674b73eb1b8ba0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25662) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-24646) Add operator wrapper for all-round iterations
[ https://issues.apache.org/jira/browse/FLINK-24646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17435839#comment-17435839 ] Yun Gao edited comment on FLINK-24646 at 10/30/21, 3:24 AM: Fixed on master via a1800b983b8851cc9aecd1fe115d90d88f3c6a58..9baf6bdaf50bd3c9a1cf2048845841a756547a24 was (Author: gaoyunhaii): Fixed on master via a1800b983b8851cc9aecd1fe115d90d88f3c6a58..824202490d3cf281942b04e66724705942580df4 > Add operator wrapper for all-round iterations > - > > Key: FLINK-24646 > URL: https://issues.apache.org/jira/browse/FLINK-24646 > Project: Flink > Issue Type: Sub-task > Components: Library / Machine Learning >Reporter: Yun Gao >Assignee: Yun Gao >Priority: Major > Labels: pull-request-available > Fix For: 0.1.0 > > > Inside the iteration, we will also broadcast the special events to mark the > end of rounds. To process these events, all the operators inside the > iteration is wrapped with a specialized wrapper operator. > There are two kinds of wrappers: the first wrapper would not recreate the > users' operator for each round, and the second one would. This issue would > implement the first kind. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink-ml] gaoyunhaii commented on a change in pull request #12: [FLINK-24650][iteration] add unbounded iteration.
gaoyunhaii commented on a change in pull request #12:
URL: https://github.com/apache/flink-ml/pull/12#discussion_r739600581
##
File path:
flink-ml-iteration/src/main/java/org/apache/flink/iteration/IterationFactory.java
##
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.iteration;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.iteration.compile.DraftExecutionEnvironment;
+import org.apache.flink.iteration.operator.HeadOperatorFactory;
+import org.apache.flink.iteration.operator.InputOperator;
+import org.apache.flink.iteration.operator.OperatorWrapper;
+import org.apache.flink.iteration.operator.OutputOperator;
+import org.apache.flink.iteration.operator.TailOperator;
+import org.apache.flink.iteration.typeinfo.IterationRecordTypeInfo;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.transformations.OneInputTransformation;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.function.BiFunction;
+import java.util.function.Function;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/** Creates iteration in a job. */
+@Internal
+public class IterationFactory {
+
+@SuppressWarnings({"unchecked", "rawtypes"})
+public static DataStreamList createIteration(
+DataStreamList initVariableStreams,
+DataStreamList dataStreams,
+IterationBody body,
+OperatorWrapper> initialOperatorWrapper,
+boolean mayHaveCriteria) {
+checkState(initVariableStreams.size() > 0, "There should be at least
one variable stream");
+
+IterationID iterationId = new IterationID();
+
+List> initVariableTypeInfos =
getTypeInfos(initVariableStreams);
+List> dataStreamTypeInfos =
getTypeInfos(dataStreams);
+
+// Add heads and inputs
+int totalInitVariableParallelism =
+map(initVariableStreams, DataStream::getParallelism).stream()
+.mapToInt(i -> i)
+.sum();
+DataStreamList initVariableInputs = addInputs(initVariableStreams,
false);
+DataStreamList headStreams =
+addHeads(
+initVariableStreams,
+initVariableInputs,
+iterationId,
+totalInitVariableParallelism,
+false,
+0);
+
+DataStreamList dataStreamInputs = addInputs(dataStreams, true);
+
+// Create the iteration body.
+StreamExecutionEnvironment env =
initVariableStreams.get(0).getExecutionEnvironment();
+DraftExecutionEnvironment draftEnv =
+new DraftExecutionEnvironment(env, initialOperatorWrapper);
+DataStreamList draftHeadStreams =
+addDraftSources(headStreams, draftEnv, initVariableTypeInfos);
+DataStreamList draftDataStreamInputs =
+addDraftSources(dataStreamInputs, draftEnv,
dataStreamTypeInfos);
+
+IterationBodyResult iterationBodyResult =
+body.process(draftHeadStreams, draftDataStreamInputs);
+
ensuresTransformationAdded(iterationBodyResult.getFeedbackVariableStreams(),
draftEnv);
+ensuresTransformationAdded(iterationBodyResult.getOutputStreams(),
draftEnv);
+draftEnv.copyToActualEnvironment();
+
+// Add tails and co-locate them with the heads.
+DataStreamList feedbackStreams =
+
getActualDataStreams(iterationBodyResult.getFeedbackVariableStreams(),
draftEnv);
+checkState(
+feedbackStreams.size() == initVariableStreams.size(),
+"The number of feedback streams "
++ feedbackStreams.size()
++ " does not match the initialized one "
++ initVariableStreams.size());
+DataStreamList tails =
[GitHub] [flink] baisui1981 commented on pull request #17521: [FLINK-24558][API/DataStream]make parent ClassLoader variable which c…
baisui1981 commented on pull request #17521: URL: https://github.com/apache/flink/pull/17521#issuecomment-955134599 @zentol @AHeise Thanks to both of you, have already understood me and have understood what I want to express, We want to build a product similar to [Jenkins](https://www.jenkins.io/) in the field of bigData, We will package various functions as Plugin as [Jenkins Plugin](https://plugins.jenkins.io/) and put it in the repository in advance. dirive by this mission, The **ISSUE** mentioned above was found in the process of building this product, Because building a Flink Job which is drived by user defined DSL in a production environment and submitting it is **fully automated**, if problems such as dependency conflicts between connectors are found in the process, they cannot be solved by manual intervention, so they just can be solved by facility of CL isolation. One of the ways I thought of this kind of problem. For FLink, only a new extension point is added, which has no effect on the existing functions of flink. For the OCP principle, I think this is a good way to implement it. And that will Added a new implementation option for users like me -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-24650) Support unbounded iteration.
[ https://issues.apache.org/jira/browse/FLINK-24650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-24650: --- Labels: pull-request-available (was: ) > Support unbounded iteration. > > > Key: FLINK-24650 > URL: https://issues.apache.org/jira/browse/FLINK-24650 > Project: Flink > Issue Type: Sub-task > Components: Library / Machine Learning >Reporter: Yun Gao >Assignee: Yun Gao >Priority: Major > Labels: pull-request-available > Fix For: 0.1.0 > > > Supports the unbounded iteration inside the flink-ml library. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink-ml] gaoyunhaii commented on a change in pull request #12: [FLINK-24650][iteration] add unbounded iteration.
gaoyunhaii commented on a change in pull request #12:
URL: https://github.com/apache/flink-ml/pull/12#discussion_r739599530
##
File path:
flink-ml-iteration/src/main/java/org/apache/flink/iteration/IterationFactory.java
##
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.iteration;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.iteration.compile.DraftExecutionEnvironment;
+import org.apache.flink.iteration.operator.HeadOperatorFactory;
+import org.apache.flink.iteration.operator.InputOperator;
+import org.apache.flink.iteration.operator.OperatorWrapper;
+import org.apache.flink.iteration.operator.OutputOperator;
+import org.apache.flink.iteration.operator.TailOperator;
+import org.apache.flink.iteration.typeinfo.IterationRecordTypeInfo;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.transformations.OneInputTransformation;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.function.BiFunction;
+import java.util.function.Function;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/** Creates iteration in a job. */
+@Internal
+public class IterationFactory {
+
+@SuppressWarnings({"unchecked", "rawtypes"})
+public static DataStreamList createIteration(
+DataStreamList initVariableStreams,
+DataStreamList dataStreams,
+IterationBody body,
+OperatorWrapper> initialOperatorWrapper,
+boolean mayHaveCriteria) {
+checkState(initVariableStreams.size() > 0, "There should be at least
one variable stream");
+
+IterationID iterationId = new IterationID();
+
+List> initVariableTypeInfos =
getTypeInfos(initVariableStreams);
+List> dataStreamTypeInfos =
getTypeInfos(dataStreams);
+
+// Add heads and inputs
+int totalInitVariableParallelism =
+map(initVariableStreams, DataStream::getParallelism).stream()
+.mapToInt(i -> i)
+.sum();
+DataStreamList initVariableInputs = addInputs(initVariableStreams,
false);
+DataStreamList headStreams =
+addHeads(
+initVariableStreams,
+initVariableInputs,
+iterationId,
+totalInitVariableParallelism,
+false,
+0);
+
+DataStreamList dataStreamInputs = addInputs(dataStreams, true);
+
+// Create the iteration body.
+StreamExecutionEnvironment env =
initVariableStreams.get(0).getExecutionEnvironment();
+DraftExecutionEnvironment draftEnv =
+new DraftExecutionEnvironment(env, initialOperatorWrapper);
+DataStreamList draftHeadStreams =
+addDraftSources(headStreams, draftEnv, initVariableTypeInfos);
+DataStreamList draftDataStreamInputs =
+addDraftSources(dataStreamInputs, draftEnv,
dataStreamTypeInfos);
+
+IterationBodyResult iterationBodyResult =
+body.process(draftHeadStreams, draftDataStreamInputs);
+
ensuresTransformationAdded(iterationBodyResult.getFeedbackVariableStreams(),
draftEnv);
Review comment:
Currently flink only adds transformation with operators into the env.
Some transformations like `SideOutTransformation` would not be added, but might
be used as the return values of the iteration body.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17610: [hotfix][docs] Fix the outdated upgrading instruction for Kafka Connector
flinkbot edited a comment on pull request #17610: URL: https://github.com/apache/flink/pull/17610#issuecomment-955131667 ## CI report: * 00214af0b13e7695db3b7c30d5b53edafae1f63a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25665) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17610: [hotfix][docs] Fix the outdated upgrading instruction for Kafka Connector
flinkbot commented on pull request #17610: URL: https://github.com/apache/flink/pull/17610#issuecomment-955131713 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 990862da26a5f5277b4470feeed3f7be0f84c98d (Sat Oct 30 02:33:55 UTC 2021) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17610: [hotfix][docs] Fix the outdated upgrading instruction for Kafka Connector
flinkbot commented on pull request #17610: URL: https://github.com/apache/flink/pull/17610#issuecomment-955131667 ## CI report: * 00214af0b13e7695db3b7c30d5b53edafae1f63a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17345: [FLINK-24227][connectors] Added Kinesis Data Streams Sink i…
flinkbot edited a comment on pull request #17345: URL: https://github.com/apache/flink/pull/17345#issuecomment-926109717 ## CI report: * d9ef87451762a897b00adf72926071be0d822342 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25663) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gaoyunhaii opened a new pull request #17610: [hotfix][docs] Fix the outdated upgrading instruction for Kafka connector
gaoyunhaii opened a new pull request #17610: URL: https://github.com/apache/flink/pull/17610 ## What is the purpose of the change This PR fixes the outdated instruction on upgrading to the latest version for Kafka Connector. ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): **no** - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no** - The serializers: **no** - The runtime per-record code paths (performance sensitive): **no** - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: **no** - The S3 file system connector: **no** ## Documentation - Does this pull request introduce a new feature? **no** - If yes, how is the feature documented? **not applicable** -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] baisui1981 commented on pull request #17521: [FLINK-24558][API/DataStream]make parent ClassLoader variable which c…
baisui1981 commented on pull request #17521: URL: https://github.com/apache/flink/pull/17521#issuecomment-955126616 > But the user-jar is the same for all tasks. You can't use a manifest in said jar for differentiating which classes should now be loaded / made accessible. You either a) need the CL factory to create a task-specific classloader (which you can't because jars are the same and CLs are shared) or b) need the cooperation from the user-code to load classes in a specific way. @zentol Please forgive me for my negligence, I didn’t explain the process clearly. Before submit the job to Flink cluster, will package a jar dynamically, at then,the metadata info for the manifest which is relevant to the job will be put in the jar togother: code as below [FlinkTaskNodeController.java](https://github.com/qlangtech/plugins/blob/864f5bf01fad567fbc09a217f4bbf17c95910f94/tis-incr/tis-realtime-flink/src/main/java/com/qlangtech/plugins/incr/flink/launch/FlinkTaskNodeController.java#L129) ``` java Manifest manifest = new Manifest(); Map entries = manifest.getEntries(); Attributes attrs = new Attributes(); attrs.put(new Attributes.Name(collection.getName()), String.valueOf(timestamp)); // put Flink job name entries.put(TISFlinkCDCStart.TIS_APP_NAME, attrs); ``` when submit to the server side, in my customized extend point implementation method [ClassLoaderFactoryBuilder.buildServerLoaderFactory](https://github.com/qlangtech/plugins/blob/864f5bf01fad567fbc09a217f4bbf17c95910f94/tis-incr/tis-flink-extends/src/main/java/com/qlangtech/plugins/incr/flink/TISFlinClassLoaderFactory.java#L101),will extract the param from the submitted jar manifest, at then pull the plugin bundles from TIS plugin repository with http protocol, and initialize the PluginManager which is responsible for load class . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-24663) PyFlink failed to get the site packege path because of SyntaxError in shell command
[
https://issues.apache.org/jira/browse/FLINK-24663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jackwangcs updated FLINK-24663:
---
Labels: (was: pull-request-available)
> PyFlink failed to get the site packege path because of SyntaxError in shell
> command
> ---
>
> Key: FLINK-24663
> URL: https://issues.apache.org/jira/browse/FLINK-24663
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.12.3
>Reporter: jackwangcs
>Priority: Major
>
> Flink throws an exception when it tries to install 3rd party dependencies:
> {code:java}
> Caused by: java.io.IOException: Failed to execute the command: python -c
> import sys;from distutils.dist import Distribution;install_obj =
> Distribution().get_command_obj('install', create=True);install_obj.prefix =
> sys.argv[1];install_obj.finalize_options();installed_dir =
> [install_obj.install_purelib];install_obj.install_purelib !=
> install_obj.install_platlib and
> installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> > 1 and print(installed_dir[1])
> /mnt/yarn/usercache/hadoop/appcache/application_1629776785656_0100/python-dist-fb549cea-0857-4b11-9eb7-7818eaa3f561/python-requirements
> output: File "", line 1
> import sys;from distutils.dist import Distribution;install_obj =
> Distribution().get_command_obj('install', create=True);install_obj.prefix =
> sys.argv[1];install_obj.finalize_options();installed_dir =
> [install_obj.install_purelib];install_obj.install_purelib !=
> install_obj.install_platlib and
> installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> > 1 and print(installed_dir[1])
>
>
>
>
> ^SyntaxError: invalid syntax
> at
> org.apache.flink.python.util.PythonEnvironmentManagerUtils.execute(PythonEnvironmentManagerUtils.java:211)
> at
> org.apache.flink.python.util.PythonEnvironmentManagerUtils.getSitePackagesPath(PythonEnvironmentManagerUtils.java:171)
> at
> org.apache.flink.python.util.PythonEnvironmentManagerUtils.pipInstallRequirements(PythonEnvironmentManagerUtils.java:99)
> at
> org.apache.flink.python.env.beam.ProcessPythonEnvironmentManager.createEnvironment(ProcessPythonEnvironmentManager.java:169)
> at
> org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.createPythonExecutionEnvironment(BeamPythonFunctionRunner.java:339)
> {code}
> This can be reproduced by running the python script in a bash shell:
> {code:java}
> python3 -c import sys;from distutils.dist import Distribution;install_obj =
> Distribution().get_command_obj('install',
> create=True);print(sys.argv[1]);install_obj.prefix =
> sys.argv[1];install_obj.finalize_options();installed_dir =
> [install_obj.install_purelib];install_obj.install_purelib !=
> install_obj.install_platlib and
> installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> > 1 and print(installed_dir[1]) /tmp/requirements
> -bash: syntax error near unexpected token `(' {code}
> The solution is to quote all argements to execute:
> {code:java}
> python3 "-c" "import sys;from distutils.dist import Distribution;install_obj
> = Distribution().get_command_obj('install',
> create=True);print(sys.argv[1]);install_obj.prefix =
> sys.argv[1];install_obj.finalize_options();installed_dir =
> [install_obj.install_purelib];install_obj.install_purelib !=
> install_obj.install_platlib and
> installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> > 1 and print(installed_dir[1])" "/tmp/requirements"
> /tmp/requirements
> /tmp/requirements/lib/python3.6/site-packages{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] JackWangCS closed pull request #17609: [FLINK-24663][Python] Quote bash argument for PythonEnvironmentManagerUtils
JackWangCS closed pull request #17609: URL: https://github.com/apache/flink/pull/17609 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17106: [FLINK-20443][API/DataStream] ContinuousProcessingTimeTrigger doesn't fire at the end of the window
flinkbot edited a comment on pull request #17106: URL: https://github.com/apache/flink/pull/17106#issuecomment-910662440 ## CI report: * bc06f66402036c872851c92a4d64a27a7bef918b UNKNOWN * 651832650f3de249e2f9398bb87927da3d261dbc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25644) * 5443a0cb5c553f09e98218d2e120121378275d62 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25664) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17106: [FLINK-20443][API/DataStream] ContinuousProcessingTimeTrigger doesn't fire at the end of the window
flinkbot edited a comment on pull request #17106: URL: https://github.com/apache/flink/pull/17106#issuecomment-910662440 ## CI report: * bc06f66402036c872851c92a4d64a27a7bef918b UNKNOWN * 651832650f3de249e2f9398bb87927da3d261dbc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25644) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17106: [FLINK-20443][API/DataStream] ContinuousProcessingTimeTrigger doesn't fire at the end of the window
flinkbot edited a comment on pull request #17106: URL: https://github.com/apache/flink/pull/17106#issuecomment-910662440 ## CI report: * bc06f66402036c872851c92a4d64a27a7bef918b UNKNOWN * 651832650f3de249e2f9398bb87927da3d261dbc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25644) * 5443a0cb5c553f09e98218d2e120121378275d62 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17106: [FLINK-20443][API/DataStream] ContinuousProcessingTimeTrigger doesn't fire at the end of the window
flinkbot edited a comment on pull request #17106: URL: https://github.com/apache/flink/pull/17106#issuecomment-910662440 ## CI report: * bc06f66402036c872851c92a4d64a27a7bef918b UNKNOWN * 651832650f3de249e2f9398bb87927da3d261dbc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25644) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17106: [FLINK-20443][API/DataStream] ContinuousProcessingTimeTrigger doesn't fire at the end of the window
flinkbot edited a comment on pull request #17106: URL: https://github.com/apache/flink/pull/17106#issuecomment-910662440 ## CI report: * bc06f66402036c872851c92a4d64a27a7bef918b UNKNOWN * 651832650f3de249e2f9398bb87927da3d261dbc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25644) * 5443a0cb5c553f09e98218d2e120121378275d62 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20870) FlinkKafkaSink
[
https://issues.apache.org/jira/browse/FLINK-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-20870:
---
Labels: auto-deprioritized-major stale-minor (was:
auto-deprioritized-major)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> FlinkKafkaSink
> --
>
> Key: FLINK-20870
> URL: https://issues.apache.org/jira/browse/FLINK-20870
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.12.0
> Environment: flink :1.12.0
> kafka 2.2.1
>Reporter: xx chai
>Priority: Minor
> Labels: auto-deprioritized-major, stale-minor
> Attachments: 1610089153(1).png
>
>
> I consum from kafka sink to kafka,Then I split the message into then pieces.I
> guess the ten message should in one transaction. When the fifth message is
> sink kafka ,I throw a exception.but the first four are already in kafka.
> I set some parameters :
>properties.setProperty("transactional.id", "cxx");
> properties.setProperty("ack", "all");
> properties.put("enable.idempotence",true);
> properties.put("max.in.flight.requests.per.connection",5);
> properties.put("retries", 2);
> properties.setProperty("client.id", "producer-syn-2");
> properties.put("isolation.level","read_committed");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20830) Add a type of HEADLESS_CLUSTER_IP for rest service type
[ https://issues.apache.org/jira/browse/FLINK-20830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-20830: --- Labels: auto-deprioritized-major stale-minor (was: auto-deprioritized-major) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Add a type of HEADLESS_CLUSTER_IP for rest service type > --- > > Key: FLINK-20830 > URL: https://issues.apache.org/jira/browse/FLINK-20830 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes >Reporter: Aitozi >Priority: Minor > Labels: auto-deprioritized-major, stale-minor > > Now we can choose ClusterIP or NodePort or LoadBalancer as rest service type. > But in our internal kubernetes cluster, there is no kube-proxy, and ClusterIP > mode rely on kube-proxy. So I think can we support another type of > HEADLESS_CLUSTER_IP to directly talk to jobmanager pod? cc [~fly_in_gis] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20880) Turn Json String into Row
[
https://issues.apache.org/jira/browse/FLINK-20880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-20880:
---
Labels: auto-deprioritized-major stale-minor (was:
auto-deprioritized-major)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Turn Json String into Row
> --
>
> Key: FLINK-20880
> URL: https://issues.apache.org/jira/browse/FLINK-20880
> Project: Flink
> Issue Type: Improvement
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.12.0
>Reporter: tonychan
>Priority: Minor
> Labels: auto-deprioritized-major, stale-minor
>
> schema has three params ,a,b,c
> message a：\{ "a": 1, "b": 1, "c": 2}
> message b：\{ "a": 1, "b": null, "c": 2}
> message c：\{ "a": 1, "c": 2}
> the schema is a,b,c running the code as bleow
> Row row = new Row(RowKind.INSERT,3);
> row.setField(0,1);
> row.setField(1,2);
> row.setField(2,3);
> System.out.println(row);
> a:1,2,3
> b:1,null,3
> c:1,null,3
> turn message into row , message a is ok, but we cant distinguish message b
> and message c if we use null, when deal message c ,we must use like NaN
> row(2,NaN), i think row need a easy way to show message b and message c
> with row
>
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20795) add a parameter to decide whether print dirty record when `ignore-parse-errors` is true
[ https://issues.apache.org/jira/browse/FLINK-20795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-20795: --- Labels: auto-deprioritized-major stale-minor (was: auto-deprioritized-major) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > add a parameter to decide whether print dirty record when > `ignore-parse-errors` is true > --- > > Key: FLINK-20795 > URL: https://issues.apache.org/jira/browse/FLINK-20795 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / Ecosystem >Affects Versions: 1.13.0 >Reporter: zoucao >Priority: Minor > Labels: auto-deprioritized-major, stale-minor > > add a parameter to decide whether print dirty data when > `ignore-parse-errors`=true, some users want to make his task stability and > know the dirty record to fix the upstream, too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20827) Just read record correlating to join key in FilesystemLookUpFunc
[ https://issues.apache.org/jira/browse/FLINK-20827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-20827: --- Labels: auto-deprioritized-major stale-minor (was: auto-deprioritized-major) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Just read record correlating to join key in FilesystemLookUpFunc > > > Key: FLINK-20827 > URL: https://issues.apache.org/jira/browse/FLINK-20827 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem, Connectors / Hive >Reporter: zoucao >Priority: Minor > Labels: auto-deprioritized-major, stale-minor > > When using Temporal table join, all hive tables' records will be loaded into > cache. But sometimes, the size of hive temporal table is larger than > expected, and users can't know exactly how big it is in memory. In this > situation, some error will occur, for example, `GC overhead limit exceeded`, > `the heartbeat of TaskManager timeout (caused by gc)`. > Maybe we can optimize the number of records readed from hive table? If the > upstream records can be hashed only by using `Join key`, then we only need > to load the part of records into cache, whose value of join key after being > hashed, is equal to one fixed hash value. If it can be done, the whole table > can be divided by the number of parallelism. I don't know whether it could > come true in the upstream under the existing framework, but It is easy to > support in `FileSystemLookupFunction` > If not, we can add some logs to tell others the size of cache to help them to > set MemorySize or other parameter of TM. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22984) UnsupportedOperationException when using Python UDF to generate watermark
[
https://issues.apache.org/jira/browse/FLINK-22984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22984:
---
Labels: auto-deprioritized-critical auto-deprioritized-major (was:
auto-deprioritized-critical stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Major, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> UnsupportedOperationException when using Python UDF to generate watermark
> -
>
> Key: FLINK-22984
> URL: https://issues.apache.org/jira/browse/FLINK-22984
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.13.0, 1.13.1
>Reporter: Maciej Bryński
>Priority: Minor
> Labels: auto-deprioritized-critical, auto-deprioritized-major
>
> Hi,
> I'm trying to use output of Python UDF (parse_data) to set watermark for the
> table
> {code:java}
> CREATE TABLE test (
> data BYTES,
> ts as parse_data(data).ts,
> WATERMARK for ts as ts
> ) WITH (
>'connector' = 'kafka',
>'topic' = 'test',
>'properties.bootstrap.servers' = 'localhost:9092',
>'properties.group.id' = 'flink',
>'scan.startup.mode' = 'earliest-offset',
>'format' = 'raw'
> ){code}
> Then running SELECT on this table gives me exception
> {code:java}
> Py4JJavaError: An error occurred while calling o311.hasNext.
> : java.lang.RuntimeException: Failed to fetch next result
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:109)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
> at
> org.apache.flink.table.api.internal.TableResultImpl$CloseableRowIteratorWrapper.hasNext(TableResultImpl.java:370)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.flink.api.python.shaded.py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
> at
> org.apache.flink.api.python.shaded.py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
> at
> org.apache.flink.api.python.shaded.py4j.Gateway.invoke(Gateway.java:282)
> at
> org.apache.flink.api.python.shaded.py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
> at
> org.apache.flink.api.python.shaded.py4j.commands.CallCommand.execute(CallCommand.java:79)
> at
> org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:238)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: java.io.IOException: Failed to fetch job execution result
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.getAccumulatorResults(CollectResultFetcher.java:177)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:120)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:106)
> ... 13 more
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> at
> java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
> at
> java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2022)
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.getAccumulatorResults(CollectResultFetcher.java:175)
> ... 15 more
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job
> execution failed.
> at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
> at
> org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:137)
> at
> java.base/java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:680)
> at
> java.base/java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:658)
> at
> java.base/java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:2094)
> at
> org.apache.flink.runtime.minicluster.MiniClusterJobClient.getJobExecutionResult(MiniClusterJobClient.java:134)
[jira] [Updated] (FLINK-20819) flink : Connectors : JDBC test failing on Red Hat 8.x PowerPC Linux
[ https://issues.apache.org/jira/browse/FLINK-20819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-20819: --- Labels: auto-deprioritized-major stale-minor (was: auto-deprioritized-major) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > flink : Connectors : JDBC test failing on Red Hat 8.x PowerPC Linux > --- > > Key: FLINK-20819 > URL: https://issues.apache.org/jira/browse/FLINK-20819 > Project: Flink > Issue Type: Bug > Components: Tests >Affects Versions: 1.11.3 > Environment: NAME="Red Hat Enterprise Linux" > VERSION="8.3 (Ootpa)" > Java > **openjdk version "1.8.0_275" > OpenJDK Runtime Environment (build 1.8.0_275-b01) > OpenJDK 64-Bit Server VM (build 25.275-b01, mixed mode) >Reporter: Priya >Priority: Minor > Labels: auto-deprioritized-major, stale-minor > > git clone [https://github.com/apache/flink.git] > cd flink > mvn clean package > > [ERROR] > org.apache.flink.connector.jdbc.catalog.factory.JdbcCatalogFactoryTest Time > elapsed: 1.028 s <<< ERROR! > java.lang.IllegalStateException: No Postgres binary found for Linux / ppc64le -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20962) Rewrite the example in 'flink-python/pyflink/shell.py'
[ https://issues.apache.org/jira/browse/FLINK-20962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-20962: --- Labels: auto-deprioritized-major pull-request-available stale-minor (was: auto-deprioritized-major pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Rewrite the example in 'flink-python/pyflink/shell.py' > -- > > Key: FLINK-20962 > URL: https://issues.apache.org/jira/browse/FLINK-20962 > Project: Flink > Issue Type: Improvement > Components: API / Python >Reporter: Wei Zhong >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available, > stale-minor > > Currently the pyflink example in 'flink-python/pyflink/shell.py' was added in > version 1.9 and has not been updated since. We need to rewrite it with the > latest recommended API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20912) Increase Log and Metric: Time consumed by Checkpoint Restore
[ https://issues.apache.org/jira/browse/FLINK-20912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-20912: --- Labels: auto-deprioritized-major stale-minor (was: auto-deprioritized-major) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Increase Log and Metric: Time consumed by Checkpoint Restore > > > Key: FLINK-20912 > URL: https://issues.apache.org/jira/browse/FLINK-20912 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.12.1, 1.13.0 >Reporter: future >Priority: Minor > Labels: auto-deprioritized-major, stale-minor > > In a production environment, some jobs with higher SLAs need to be restarted > quickly if failover occurs. Checkpoint restore is an important part of task > start. When the Flink task starts slowly, the related Log and Metric should > be added to facilitate troubleshooting. > For example: ByteDance shared in FFA 2020: They made OperatorState > parallelized restore. Without these metrics, there will be two problems: > 1. It is not easy to find the problem. If the task starts slowly, it is not > known whether the root cause is the slow Checkpoint restore. > 2. If optimized, how much speed has been improved for restore? Need to be > quantified. > I believe that many companies have made relevant metrics in their internal > Flink versions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20881) flink sql and flink sink must be run in one stream
[ https://issues.apache.org/jira/browse/FLINK-20881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-20881: --- Labels: auto-deprioritized-major stale-minor (was: auto-deprioritized-major) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > flink sql and flink sink must be run in one stream > -- > > Key: FLINK-20881 > URL: https://issues.apache.org/jira/browse/FLINK-20881 > Project: Flink > Issue Type: New Feature > Components: API / DataStream >Reporter: donglei >Priority: Minor > Labels: auto-deprioritized-major, stale-minor > Attachments: image-2021-01-07-17-34-09-195.png > > > i have many sink written by code and we need to use hive to store data > so flink sql execute and flink data stream sink must be support by one job > !image-2021-01-07-17-34-09-195.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20820) Rename o.a.f.table.runtime.generated package in blink runtime
[
https://issues.apache.org/jira/browse/FLINK-20820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-20820:
---
Labels: auto-deprioritized-major stale-minor (was:
auto-deprioritized-major)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Rename o.a.f.table.runtime.generated package in blink runtime
> -
>
> Key: FLINK-20820
> URL: https://issues.apache.org/jira/browse/FLINK-20820
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
>Reporter: Chesnay Schepler
>Priority: Minor
> Labels: auto-deprioritized-major, stale-minor
>
> The {{org/apache/flink/table/runtime/generated}} in flink-table-runtime-blink
> does not contain generate code and hence should be renamed to avoid confusion.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20886) Add the option to get a threaddump on checkpoint timeouts
[ https://issues.apache.org/jira/browse/FLINK-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-20886: --- Labels: auto-deprioritized-major stale-minor usability (was: auto-deprioritized-major usability) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Add the option to get a threaddump on checkpoint timeouts > - > > Key: FLINK-20886 > URL: https://issues.apache.org/jira/browse/FLINK-20886 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Reporter: Nico Kruber >Priority: Minor > Labels: auto-deprioritized-major, stale-minor, usability > > For debugging checkpoint timeouts, I was thinking about the following > addition to Flink: > When a checkpoint times out and the async thread is still running, create a > thread dump [1] and either add this to the checkpoint stats, log it, or write > it out. > This may help identifying where the checkpoint is stuck (maybe a lock, could > also be in a third party lib like the FS connectors,...). It would give us > some insights into what the thread is currently doing. > Limiting the scope of the threads would be nice but may not be possible in > the general case since additional threads (spawned by the FS connector lib, > or otherwise connected) may interact with the async thread(s) by e.g. going > through the same locks. Maybe we can reduce the thread dumps to all async > threads of the failed checkpoint + all thready that interact with it, e.g. > via locks? > I'm also not sure whether the ability to have thread dumps or not should be > user-configurable (Could it contain sensitive information from other jobs if > you run a session cluster? Is that even relevant since we don't give > isolation guarantees anyway?). If it is configurable, it should be on by > default. > [1] https://crunchify.com/how-to-generate-java-thread-dump-programmatically/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #17345: [FLINK-24227][connectors] Added Kinesis Data Streams Sink i…
flinkbot edited a comment on pull request #17345: URL: https://github.com/apache/flink/pull/17345#issuecomment-926109717 ## CI report: * e74755211c68fe2bce5a5088c7ea00a4a4b7237f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25654) * d9ef87451762a897b00adf72926071be0d822342 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25663) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17345: [FLINK-24227][connectors] Added Kinesis Data Streams Sink i…
flinkbot edited a comment on pull request #17345: URL: https://github.com/apache/flink/pull/17345#issuecomment-926109717 ## CI report: * e74755211c68fe2bce5a5088c7ea00a4a4b7237f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25654) * d9ef87451762a897b00adf72926071be0d822342 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17606: [FLINK-24706][rpc] Forward deserialization errors to returned future
flinkbot edited a comment on pull request #17606: URL: https://github.com/apache/flink/pull/17606#issuecomment-954752065 ## CI report: * 3652a3b41d02f84d54924930705ae710d7f7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25657) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on pull request #17606: [FLINK-24706][rpc] Forward deserialization errors to returned future
zentol commented on pull request #17606: URL: https://github.com/apache/flink/pull/17606#issuecomment-955057857 hmm...the `TaskManagerProcessFailureBatchRecoveryITCase` is failing pretty reliably now. I'm wondering if this issue was masking another failure as well... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17606: [FLINK-24706][rpc] Forward deserialization errors to returned future
flinkbot edited a comment on pull request #17606: URL: https://github.com/apache/flink/pull/17606#issuecomment-954752065 ## CI report: * d65fff260e684b231835a04583b3e9d600ae96d6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25645) * 3652a3b41d02f84d54924930705ae710d7f7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25657) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on pull request #17606: [FLINK-24706][rpc] Forward deserialization errors to returned future
zentol commented on pull request #17606: URL: https://github.com/apache/flink/pull/17606#issuecomment-955045722 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17605: [FLINK-24704][table] Fix exception when the input record loses monotonicity on the sort key field of UpdatableTopNFunction
flinkbot edited a comment on pull request #17605: URL: https://github.com/apache/flink/pull/17605#issuecomment-954693023 ## CI report: * 2e9cea2efeb7e9f4722275a20f8551f27833fd95 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25653) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17608: [FLINK-24550][rpc] Use ContextClassLoader for message deserialization
flinkbot edited a comment on pull request #17608: URL: https://github.com/apache/flink/pull/17608#issuecomment-954803172 ## CI report: * 49487256bfb6ae8d304a42330454651924d001fb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25648) * e6d48a2a8d555b42b1720db653674b73eb1b8ba0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25662) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17608: [FLINK-24550][rpc] Use ContextClassLoader for message deserialization
flinkbot edited a comment on pull request #17608: URL: https://github.com/apache/flink/pull/17608#issuecomment-954803172 ## CI report: * 49487256bfb6ae8d304a42330454651924d001fb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25648) * e6d48a2a8d555b42b1720db653674b73eb1b8ba0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17606: [FLINK-24706][rpc] Forward deserialization errors to returned future
flinkbot edited a comment on pull request #17606: URL: https://github.com/apache/flink/pull/17606#issuecomment-954752065 ## CI report: * d65fff260e684b231835a04583b3e9d600ae96d6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25645) * 3652a3b41d02f84d54924930705ae710d7f7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25657) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17606: [FLINK-24706][rpc] Forward deserialization errors to returned future
flinkbot edited a comment on pull request #17606: URL: https://github.com/apache/flink/pull/17606#issuecomment-954752065 ## CI report: * d65fff260e684b231835a04583b3e9d600ae96d6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25645) * 3652a3b41d02f84d54924930705ae710d7f7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-24327) Integrate unified Elasticsearch 7 sink with Table API
[ https://issues.apache.org/jira/browse/FLINK-24327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise updated FLINK-24327: Summary: Integrate unified Elasticsearch 7 sink with Table API (was: Integrate Elasticsearch 7 sink with Table API) > Integrate unified Elasticsearch 7 sink with Table API > - > > Key: FLINK-24327 > URL: https://issues.apache.org/jira/browse/FLINK-24327 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Alexander Preuss >Assignee: Alexander Preuss >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-24327) Integrate Elasticsearch 7 sink with Table API
[ https://issues.apache.org/jira/browse/FLINK-24327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise resolved FLINK-24327. - Fix Version/s: 1.15.0 Resolution: Fixed Merged into master as 4335044245da4d1129e3abababfc7399cecfaf4e..980acdb665c08a8fa09f02b24755d607a21fc1c8. > Integrate Elasticsearch 7 sink with Table API > - > > Key: FLINK-24327 > URL: https://issues.apache.org/jira/browse/FLINK-24327 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Alexander Preuss >Assignee: Alexander Preuss >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] AHeise merged pull request #17374: [FLINK-24327][connectors/elasticsearch] Add Elasticsearch 7 sink for table API
AHeise merged pull request #17374: URL: https://github.com/apache/flink/pull/17374 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] AHeise merged pull request #17594: [FLINK-24612][connectors/kafka] Configure Kafka test container log levels accordingly to test logger
AHeise merged pull request #17594: URL: https://github.com/apache/flink/pull/17594 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17345: [FLINK-24227][connectors] Added Kinesis Data Streams Sink i…
flinkbot edited a comment on pull request #17345: URL: https://github.com/apache/flink/pull/17345#issuecomment-926109717 ## CI report: * e74755211c68fe2bce5a5088c7ea00a4a4b7237f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25654) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17608: [FLINK-24550][rpc] Use ContextClassLoader for message deserialization
flinkbot edited a comment on pull request #17608: URL: https://github.com/apache/flink/pull/17608#issuecomment-954803172 ## CI report: * 49487256bfb6ae8d304a42330454651924d001fb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25648) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17345: [FLINK-24227][connectors] Added Kinesis Data Streams Sink i…
flinkbot edited a comment on pull request #17345: URL: https://github.com/apache/flink/pull/17345#issuecomment-926109717 ## CI report: * 5c8ae3d4103b875b368cebb5ab9376c2cff21096 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25649) * e74755211c68fe2bce5a5088c7ea00a4a4b7237f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25654) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17345: [FLINK-24227][connectors] Added Kinesis Data Streams Sink i…
flinkbot edited a comment on pull request #17345: URL: https://github.com/apache/flink/pull/17345#issuecomment-926109717 ## CI report: * 4358f7215221955c9e6a3735c9bcd327a42c8c43 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25647) * 5c8ae3d4103b875b368cebb5ab9376c2cff21096 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25649) * e74755211c68fe2bce5a5088c7ea00a4a4b7237f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25654) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17345: [FLINK-24227][connectors] Added Kinesis Data Streams Sink i…
flinkbot edited a comment on pull request #17345: URL: https://github.com/apache/flink/pull/17345#issuecomment-926109717 ## CI report: * 4358f7215221955c9e6a3735c9bcd327a42c8c43 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25647) * 5c8ae3d4103b875b368cebb5ab9376c2cff21096 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25649) * e74755211c68fe2bce5a5088c7ea00a4a4b7237f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17106: [FLINK-20443][API/DataStream] ContinuousProcessingTimeTrigger doesn't fire at the end of the window
flinkbot edited a comment on pull request #17106: URL: https://github.com/apache/flink/pull/17106#issuecomment-910662440 ## CI report: * bc06f66402036c872851c92a4d64a27a7bef918b UNKNOWN * 651832650f3de249e2f9398bb87927da3d261dbc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25644) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17605: [FLINK-24704][table] Fix exception when the input record loses monotonicity on the sort key field of UpdatableTopNFunction
flinkbot edited a comment on pull request #17605: URL: https://github.com/apache/flink/pull/17605#issuecomment-954693023 ## CI report: * c208976e1df1e97ab7951395e8be0c1f205a170a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25643) * 2e9cea2efeb7e9f4722275a20f8551f27833fd95 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25653) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17605: [FLINK-24704][table] Fix exception when the input record loses monotonicity on the sort key field of UpdatableTopNFunction
flinkbot edited a comment on pull request #17605: URL: https://github.com/apache/flink/pull/17605#issuecomment-954693023 ## CI report: * c208976e1df1e97ab7951395e8be0c1f205a170a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25643) * 2e9cea2efeb7e9f4722275a20f8551f27833fd95 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lincoln-lil edited a comment on pull request #17605: [FLINK-24704][table] Fix exception when the input record loses monotonicity on the sort key field of UpdatableTopNFunction
lincoln-lil edited a comment on pull request #17605: URL: https://github.com/apache/flink/pull/17605#issuecomment-954909178 Some tests failed, I'll fix it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lincoln-lil commented on pull request #17605: [FLINK-24704][table] Fix exception when the input record loses monotonicity on the sort key field of UpdatableTopNFunction
lincoln-lil commented on pull request #17605: URL: https://github.com/apache/flink/pull/17605#issuecomment-954909178 Tests failure because runtime context not initialized, I'll fix it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17609: [FLINK-24663][Python] Quote bash argument for PythonEnvironmentManagerUtils
flinkbot edited a comment on pull request #17609: URL: https://github.com/apache/flink/pull/17609#issuecomment-954829138 ## CI report: * 6f0fdb55ef06c57617fec52c42a0c650d938e371 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25650) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17606: [FLINK-24706][rpc] Forward deserialization errors to returned future
flinkbot edited a comment on pull request #17606: URL: https://github.com/apache/flink/pull/17606#issuecomment-954752065 ## CI report: * d65fff260e684b231835a04583b3e9d600ae96d6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25645) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17607: [FLINK-24543][runtime] Avoid possible inconsistencies of ZookeeperSta…
flinkbot edited a comment on pull request #17607: URL: https://github.com/apache/flink/pull/17607#issuecomment-954762085 ## CI report: * 7ccc4b0240e733447c42bbfa48911f768afac57d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25646) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17605: [FLINK-24704][table] Fix exception when the input record loses monotonicity on the sort key field of UpdatableTopNFunction
flinkbot edited a comment on pull request #17605: URL: https://github.com/apache/flink/pull/17605#issuecomment-954693023 ## CI report: * c208976e1df1e97ab7951395e8be0c1f205a170a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25643) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17571: [FLINK-23015][table-runtime-blink] Implement streaming window Deduplicate operator
flinkbot edited a comment on pull request #17571: URL: https://github.com/apache/flink/pull/17571#issuecomment-951900326 ## CI report: * a49b4c966901159870a7f0014b2f4520fc35425e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25629) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17571: [FLINK-23015][table-runtime-blink] Implement streaming window Deduplicate operator
flinkbot edited a comment on pull request #17571: URL: https://github.com/apache/flink/pull/17571#issuecomment-951900326 ## CI report: * a49b4c966901159870a7f0014b2f4520fc35425e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25629) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17571: [FLINK-23015][table-runtime-blink] Implement streaming window Deduplicate operator
flinkbot edited a comment on pull request #17571: URL: https://github.com/apache/flink/pull/17571#issuecomment-951900326 ## CI report: * a49b4c966901159870a7f0014b2f4520fc35425e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25629) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] beyond1920 commented on pull request #17571: [FLINK-23015][table-runtime-blink] Implement streaming window Deduplicate operator
beyond1920 commented on pull request #17571: URL: https://github.com/apache/flink/pull/17571#issuecomment-954862143 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24708) `ConvertToNotInOrInRule` has a bug which leads to wrong result
[
https://issues.apache.org/jira/browse/FLINK-24708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17436049#comment-17436049
]
JING ZHANG commented on FLINK-24708:
[~TsReaper] cc [~godfrey] [~godfreyhe] It seems to be a bug in
`ConvertToNotInOrInRule`.
For query 1 and query 7, after apply `ConvertToNotInOrInRule` in
'default_rewrite' phase, 'abc','bcd','cde' would be cast to CHAR(4) which leads
to empty result, because 'bcd' != 'bcd '.
!image-2021-10-29-23-59-48-074.png!
> `ConvertToNotInOrInRule` has a bug which leads to wrong result
> --
>
> Key: FLINK-24708
> URL: https://issues.apache.org/jira/browse/FLINK-24708
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: JING ZHANG
>Priority: Major
> Attachments: image-2021-10-29-23-59-48-074.png
>
>
> A user report this bug in maillist, I paste the content here.
> We are in the process of upgrading from Flink 1.9.3 to 1.13.3. We have
> noticed that statements with either where UPPER(field) or LOWER(field) in
> combination with an IN do not always evaluate correctly.
>
> The following test case highlights this problem.
>
>
> {code:java}
> import org.apache.flink.streaming.api.datastream.DataStream;
> import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
> import org.apache.flink.table.api.Schema;
> import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
> public class TestCase {
> public static void main(String[] args) throws Exception
> { final StreamExecutionEnvironment env =
> StreamExecutionEnvironment._getExecutionEnvironment_();
> env.setParallelism(1); TestData testData = new TestData();
> testData.setField1("bcd"); DataStream stream =
> env.fromElements(testData); stream.print(); // To prevent 'No
> operators' error final StreamTableEnvironment tableEnvironment =
> StreamTableEnvironment._create_(env);
> tableEnvironment.createTemporaryView("testTable", stream,
> Schema._newBuilder_().build()); // Fails because abcd is larger than
> abc tableEnvironment.executeSql("select *, '1' as run from testTable
> WHERE lower(field1) IN ('abcd', 'abc', 'bcd', 'cde')").print(); //
> Succeeds because lower was removed
> tableEnvironment.executeSql("select *, '2' as run from testTable WHERE field1
> IN ('abcd', 'abc', 'bcd', 'cde')").print(); // These 4 succeed
> because the smallest literal is before abcd
> tableEnvironment.executeSql("select *, '3' as run from testTable WHERE
> lower(field1) IN ('abc', 'abcd', 'bcd', 'cde')").print();
> tableEnvironment.executeSql("select *, '4' as run from testTable WHERE
> lower(field1) IN ('abc', 'bcd', 'abhi', 'cde')").print();
> tableEnvironment.executeSql("select *, '5' as run from testTable WHERE
> lower(field1) IN ('cde', 'abcd', 'abc', 'bcd')").print();
> tableEnvironment.executeSql("select *, '6' as run from testTable WHERE
> lower(field1) IN ('cde', 'abc', 'abcd', 'bcd')").print(); // Fails
> because smallest is not first tableEnvironment.executeSql("select *,
> '7' as run from testTable WHERE lower(field1) IN ('cdef', 'abce', 'abcd',
> 'ab', 'bcd')").print(); // Succeeds
> tableEnvironment.executeSql("select *, '8' as run from testTable WHERE
> lower(field1) IN ('ab', 'cdef', 'abce', 'abcdefgh', 'bcd')").print();
> env.execute("TestCase"); }
> public static class TestData {
> private String field1;
> public String getField1()
> { return field1; }
> public void setField1(String field1)
> { this.field1 = field1; }
> }
> }
>
> {code}
>
> The job produces the following output:
> Empty set
> +-+---++
> |op| field1| run|
> +-+---++
> |+I| bcd| 2|
> +-+---++
> 1 row in set
> +-+---++
> |op| field1| run|
> +-+---++
> |+I| bcd| 3|
> +-+---++
> 1 row in set
> +-+---++
> |op| field1| run|
>
[jira] [Updated] (FLINK-24708) `ConvertToNotInOrInRule` has a bug which leads to wrong result
[
https://issues.apache.org/jira/browse/FLINK-24708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
JING ZHANG updated FLINK-24708:
---
Attachment: image-2021-10-29-23-59-48-074.png
> `ConvertToNotInOrInRule` has a bug which leads to wrong result
> --
>
> Key: FLINK-24708
> URL: https://issues.apache.org/jira/browse/FLINK-24708
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: JING ZHANG
>Priority: Major
> Attachments: image-2021-10-29-23-59-48-074.png
>
>
> A user report this bug in maillist, I paste the content here.
> We are in the process of upgrading from Flink 1.9.3 to 1.13.3. We have
> noticed that statements with either where UPPER(field) or LOWER(field) in
> combination with an IN do not always evaluate correctly.
>
> The following test case highlights this problem.
>
>
> {code:java}
> import org.apache.flink.streaming.api.datastream.DataStream;
> import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
> import org.apache.flink.table.api.Schema;
> import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
> public class TestCase {
> public static void main(String[] args) throws Exception
> { final StreamExecutionEnvironment env =
> StreamExecutionEnvironment._getExecutionEnvironment_();
> env.setParallelism(1); TestData testData = new TestData();
> testData.setField1("bcd"); DataStream stream =
> env.fromElements(testData); stream.print(); // To prevent 'No
> operators' error final StreamTableEnvironment tableEnvironment =
> StreamTableEnvironment._create_(env);
> tableEnvironment.createTemporaryView("testTable", stream,
> Schema._newBuilder_().build()); // Fails because abcd is larger than
> abc tableEnvironment.executeSql("select *, '1' as run from testTable
> WHERE lower(field1) IN ('abcd', 'abc', 'bcd', 'cde')").print(); //
> Succeeds because lower was removed
> tableEnvironment.executeSql("select *, '2' as run from testTable WHERE field1
> IN ('abcd', 'abc', 'bcd', 'cde')").print(); // These 4 succeed
> because the smallest literal is before abcd
> tableEnvironment.executeSql("select *, '3' as run from testTable WHERE
> lower(field1) IN ('abc', 'abcd', 'bcd', 'cde')").print();
> tableEnvironment.executeSql("select *, '4' as run from testTable WHERE
> lower(field1) IN ('abc', 'bcd', 'abhi', 'cde')").print();
> tableEnvironment.executeSql("select *, '5' as run from testTable WHERE
> lower(field1) IN ('cde', 'abcd', 'abc', 'bcd')").print();
> tableEnvironment.executeSql("select *, '6' as run from testTable WHERE
> lower(field1) IN ('cde', 'abc', 'abcd', 'bcd')").print(); // Fails
> because smallest is not first tableEnvironment.executeSql("select *,
> '7' as run from testTable WHERE lower(field1) IN ('cdef', 'abce', 'abcd',
> 'ab', 'bcd')").print(); // Succeeds
> tableEnvironment.executeSql("select *, '8' as run from testTable WHERE
> lower(field1) IN ('ab', 'cdef', 'abce', 'abcdefgh', 'bcd')").print();
> env.execute("TestCase"); }
> public static class TestData {
> private String field1;
> public String getField1()
> { return field1; }
> public void setField1(String field1)
> { this.field1 = field1; }
> }
> }
>
> {code}
>
> The job produces the following output:
> Empty set
> +-+---++
> |op| field1| run|
> +-+---++
> |+I| bcd| 2|
> +-+---++
> 1 row in set
> +-+---++
> |op| field1| run|
> +-+---++
> |+I| bcd| 3|
> +-+---++
> 1 row in set
> +-+---++
> |op| field1| run|
> +-+---++
> |+I| bcd| 4|
> +-+---++
> 1 row in set
> +-+---++
> |op| field1|
[jira] [Created] (FLINK-24708) `ConvertToNotInOrInRule` has a bug which leads to wrong result
JING ZHANG created FLINK-24708:
--
Summary: `ConvertToNotInOrInRule` has a bug which leads to wrong
result
Key: FLINK-24708
URL: https://issues.apache.org/jira/browse/FLINK-24708
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Reporter: JING ZHANG
A user report this bug in maillist, I paste the content here.
We are in the process of upgrading from Flink 1.9.3 to 1.13.3. We have noticed
that statements with either where UPPER(field) or LOWER(field) in combination
with an IN do not always evaluate correctly.
The following test case highlights this problem.
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.table.api.Schema;
import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
public class TestCase {
public static void main(String[] args) throws Exception {
final StreamExecutionEnvironment env =
StreamExecutionEnvironment._getExecutionEnvironment_();
env.setParallelism(1);
TestData testData = new TestData();
testData.setField1("bcd");
DataStream stream = env.fromElements(testData);
stream.print(); // To prevent 'No operators' error
final StreamTableEnvironment tableEnvironment =
StreamTableEnvironment._create_(env);
tableEnvironment.createTemporaryView("testTable", stream,
Schema._newBuilder_().build());
// Fails because abcd is larger than abc
tableEnvironment.executeSql("select *, '1' as run from testTable WHERE
lower(field1) IN ('abcd', 'abc', 'bcd', 'cde')").print();
// Succeeds because lower was removed
tableEnvironment.executeSql("select *, '2' as run from testTable WHERE
field1 IN ('abcd', 'abc', 'bcd', 'cde')").print();
// These 4 succeed because the smallest literal is before abcd
tableEnvironment.executeSql("select *, '3' as run from testTable WHERE
lower(field1) IN ('abc', 'abcd', 'bcd', 'cde')").print();
tableEnvironment.executeSql("select *, '4' as run from testTable WHERE
lower(field1) IN ('abc', 'bcd', 'abhi', 'cde')").print();
tableEnvironment.executeSql("select *, '5' as run from testTable WHERE
lower(field1) IN ('cde', 'abcd', 'abc', 'bcd')").print();
tableEnvironment.executeSql("select *, '6' as run from testTable WHERE
lower(field1) IN ('cde', 'abc', 'abcd', 'bcd')").print();
// Fails because smallest is not first
tableEnvironment.executeSql("select *, '7' as run from testTable WHERE
lower(field1) IN ('cdef', 'abce', 'abcd', 'ab', 'bcd')").print();
// Succeeds
tableEnvironment.executeSql("select *, '8' as run from testTable WHERE
lower(field1) IN ('ab', 'cdef', 'abce', 'abcdefgh', 'bcd')").print();
env.execute("TestCase");
}
public static class TestData {
private String field1;
public String getField1() {
return field1;
}
public void setField1(String field1) {
this.field1 = field1;
}
}
}
The job produces the following output:
Empty set
++++
| op | field1 | run |
++++
| +I | bcd | 2 |
++++
1 row in set
++++
| op | field1 | run |
++++
| +I | bcd | 3 |
++++
1 row in set
++++
| op | field1 | run |
++++
| +I | bcd | 4 |
++++
1 row in set
++++
| op | field1 | run |
++++
| +I | bcd | 5 |
++++
1 row in set
++++
| op | field1 | run |
[jira] [Updated] (FLINK-24708) `ConvertToNotInOrInRule` has a bug which leads to wrong result
[
https://issues.apache.org/jira/browse/FLINK-24708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
JING ZHANG updated FLINK-24708:
---
Description:
A user report this bug in maillist, I paste the content here.
We are in the process of upgrading from Flink 1.9.3 to 1.13.3. We have noticed
that statements with either where UPPER(field) or LOWER(field) in combination
with an IN do not always evaluate correctly.
The following test case highlights this problem.
{code:java}
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.table.api.Schema;
import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
public class TestCase {
public static void main(String[] args) throws Exception
{ final StreamExecutionEnvironment env =
StreamExecutionEnvironment._getExecutionEnvironment_();
env.setParallelism(1); TestData testData = new TestData();
testData.setField1("bcd"); DataStream stream =
env.fromElements(testData); stream.print(); // To prevent 'No
operators' error final StreamTableEnvironment tableEnvironment =
StreamTableEnvironment._create_(env);
tableEnvironment.createTemporaryView("testTable", stream,
Schema._newBuilder_().build()); // Fails because abcd is larger than
abc tableEnvironment.executeSql("select *, '1' as run from testTable
WHERE lower(field1) IN ('abcd', 'abc', 'bcd', 'cde')").print(); //
Succeeds because lower was removed tableEnvironment.executeSql("select
*, '2' as run from testTable WHERE field1 IN ('abcd', 'abc', 'bcd',
'cde')").print(); // These 4 succeed because the smallest literal is
before abcd tableEnvironment.executeSql("select *, '3' as run from
testTable WHERE lower(field1) IN ('abc', 'abcd', 'bcd', 'cde')").print();
tableEnvironment.executeSql("select *, '4' as run from testTable WHERE
lower(field1) IN ('abc', 'bcd', 'abhi', 'cde')").print();
tableEnvironment.executeSql("select *, '5' as run from testTable WHERE
lower(field1) IN ('cde', 'abcd', 'abc', 'bcd')").print();
tableEnvironment.executeSql("select *, '6' as run from testTable WHERE
lower(field1) IN ('cde', 'abc', 'abcd', 'bcd')").print(); // Fails
because smallest is not first tableEnvironment.executeSql("select *,
'7' as run from testTable WHERE lower(field1) IN ('cdef', 'abce', 'abcd', 'ab',
'bcd')").print(); // Succeeds
tableEnvironment.executeSql("select *, '8' as run from testTable WHERE
lower(field1) IN ('ab', 'cdef', 'abce', 'abcdefgh', 'bcd')").print();
env.execute("TestCase"); }
public static class TestData {
private String field1;
public String getField1()
{ return field1; }
public void setField1(String field1)
{ this.field1 = field1; }
}
}
{code}
The job produces the following output:
Empty set
+-+---++
|op| field1| run|
+-+---++
|+I| bcd| 2|
+-+---++
1 row in set
+-+---++
|op| field1| run|
+-+---++
|+I| bcd| 3|
+-+---++
1 row in set
+-+---++
|op| field1| run|
+-+---++
|+I| bcd| 4|
+-+---++
1 row in set
+-+---++
|op| field1| run|
+-+---++
|+I| bcd| 5|
+-+---++
1 row in set
+-+---++
|op| field1| run|
+-+---++
|+I| bcd| 6|
+-+---++
1 row in set
[GitHub] [flink] flinkbot edited a comment on pull request #17605: [FLINK-24704][table] Fix exception when the input record loses monotonicity on the sort key field of UpdatableTopNFunction
flinkbot edited a comment on pull request #17605: URL: https://github.com/apache/flink/pull/17605#issuecomment-954693023 ## CI report: * 10ca9cf225390e479797cac385c89c3b45eb2cb1 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25641) * c208976e1df1e97ab7951395e8be0c1f205a170a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25643) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17609: [FLINK-24663][Python] Quote bash argument for PythonEnvironmentManagerUtils
flinkbot edited a comment on pull request #17609: URL: https://github.com/apache/flink/pull/17609#issuecomment-954829138 ## CI report: * 6f0fdb55ef06c57617fec52c42a0c650d938e371 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25650) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17609: [FLINK-24663][Python] Quote bash argument for PythonEnvironmentManagerUtils
flinkbot commented on pull request #17609: URL: https://github.com/apache/flink/pull/17609#issuecomment-954831151 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 6f0fdb55ef06c57617fec52c42a0c650d938e371 (Fri Oct 29 15:23:15 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-24663).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17609: [FLINK-24663][Python] Quote bash argument for PythonEnvironmentManagerUtils
flinkbot commented on pull request #17609: URL: https://github.com/apache/flink/pull/17609#issuecomment-954829138 ## CI report: * 6f0fdb55ef06c57617fec52c42a0c650d938e371 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-24663) PyFlink failed to get the site packege path because of SyntaxError in shell command
[
https://issues.apache.org/jira/browse/FLINK-24663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-24663:
---
Labels: pull-request-available (was: )
> PyFlink failed to get the site packege path because of SyntaxError in shell
> command
> ---
>
> Key: FLINK-24663
> URL: https://issues.apache.org/jira/browse/FLINK-24663
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.12.3
>Reporter: jackwangcs
>Priority: Major
> Labels: pull-request-available
>
> Flink throws an exception when it tries to install 3rd party dependencies:
> {code:java}
> Caused by: java.io.IOException: Failed to execute the command: python -c
> import sys;from distutils.dist import Distribution;install_obj =
> Distribution().get_command_obj('install', create=True);install_obj.prefix =
> sys.argv[1];install_obj.finalize_options();installed_dir =
> [install_obj.install_purelib];install_obj.install_purelib !=
> install_obj.install_platlib and
> installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> > 1 and print(installed_dir[1])
> /mnt/yarn/usercache/hadoop/appcache/application_1629776785656_0100/python-dist-fb549cea-0857-4b11-9eb7-7818eaa3f561/python-requirements
> output: File "", line 1
> import sys;from distutils.dist import Distribution;install_obj =
> Distribution().get_command_obj('install', create=True);install_obj.prefix =
> sys.argv[1];install_obj.finalize_options();installed_dir =
> [install_obj.install_purelib];install_obj.install_purelib !=
> install_obj.install_platlib and
> installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> > 1 and print(installed_dir[1])
>
>
>
>
> ^SyntaxError: invalid syntax
> at
> org.apache.flink.python.util.PythonEnvironmentManagerUtils.execute(PythonEnvironmentManagerUtils.java:211)
> at
> org.apache.flink.python.util.PythonEnvironmentManagerUtils.getSitePackagesPath(PythonEnvironmentManagerUtils.java:171)
> at
> org.apache.flink.python.util.PythonEnvironmentManagerUtils.pipInstallRequirements(PythonEnvironmentManagerUtils.java:99)
> at
> org.apache.flink.python.env.beam.ProcessPythonEnvironmentManager.createEnvironment(ProcessPythonEnvironmentManager.java:169)
> at
> org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.createPythonExecutionEnvironment(BeamPythonFunctionRunner.java:339)
> {code}
> This can be reproduced by running the python script in a bash shell:
> {code:java}
> python3 -c import sys;from distutils.dist import Distribution;install_obj =
> Distribution().get_command_obj('install',
> create=True);print(sys.argv[1]);install_obj.prefix =
> sys.argv[1];install_obj.finalize_options();installed_dir =
> [install_obj.install_purelib];install_obj.install_purelib !=
> install_obj.install_platlib and
> installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> > 1 and print(installed_dir[1]) /tmp/requirements
> -bash: syntax error near unexpected token `(' {code}
> The solution is to quote all argements to execute:
> {code:java}
> python3 "-c" "import sys;from distutils.dist import Distribution;install_obj
> = Distribution().get_command_obj('install',
> create=True);print(sys.argv[1]);install_obj.prefix =
> sys.argv[1];install_obj.finalize_options();installed_dir =
> [install_obj.install_purelib];install_obj.install_purelib !=
> install_obj.install_platlib and
> installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> > 1 and print(installed_dir[1])" "/tmp/requirements"
> /tmp/requirements
> /tmp/requirements/lib/python3.6/site-packages{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] JackWangCS opened a new pull request #17609: [FLINK-24663][Python] Quote bash argument for PythonEnvironmentManagerUtils
JackWangCS opened a new pull request #17609: URL: https://github.com/apache/flink/pull/17609 ## What is the purpose of the change PyFlink failed to get the site packege path and thus failed install 3rd party dependencies because of SyntaxError in shell command. This PR quotes long bash argument for PythonEnvironmentManagerUtils to solve this issue. ## Brief change log Quote long python bash argument for PythonEnvironmentManagerUtils ## Verifying this change This change added tests and can be verified as follows: - *Add new test `testBashQuoteString()` in PythonEnvironmentManagerUtilsTest to test the string quoting. * The functionality of `PythonEnvironmentManagerUtils` already be covered by existing tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] gaoyunhaii commented on a change in pull request #11: [FLINK-24649][iteration] Add DraftExecutionEnvironment to support wrapping operators during compile time
gaoyunhaii commented on a change in pull request #11:
URL: https://github.com/apache/flink-ml/pull/11#discussion_r739320492
##
File path:
flink-ml-iteration/src/main/java/org/apache/flink/iteration/compile/DraftExecutionEnvironment.java
##
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.iteration.compile;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.JobExecutionResult;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.core.execution.DefaultExecutorServiceLoader;
+import
org.apache.flink.iteration.compile.translator.BroadcastStateTransformationTranslator;
+import
org.apache.flink.iteration.compile.translator.KeyedBroadcastStateTransformationTranslator;
+import
org.apache.flink.iteration.compile.translator.MultipleInputTransformationTranslator;
+import
org.apache.flink.iteration.compile.translator.OneInputTransformationTranslator;
+import
org.apache.flink.iteration.compile.translator.PartitionTransformationTranslator;
+import
org.apache.flink.iteration.compile.translator.ReduceTransformationTranslator;
+import
org.apache.flink.iteration.compile.translator.SideOutputTransformationTranslator;
+import
org.apache.flink.iteration.compile.translator.TwoInputTransformationTranslator;
+import
org.apache.flink.iteration.compile.translator.UnionTransformationTranslator;
+import org.apache.flink.iteration.operator.OperatorWrapper;
+import org.apache.flink.iteration.utils.ReflectionUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import
org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction;
+import org.apache.flink.streaming.api.graph.StreamGraph;
+import
org.apache.flink.streaming.api.transformations.BroadcastStateTransformation;
+import
org.apache.flink.streaming.api.transformations.KeyedBroadcastStateTransformation;
+import
org.apache.flink.streaming.api.transformations.KeyedMultipleInputTransformation;
+import
org.apache.flink.streaming.api.transformations.MultipleInputTransformation;
+import org.apache.flink.streaming.api.transformations.OneInputTransformation;
+import org.apache.flink.streaming.api.transformations.PartitionTransformation;
+import org.apache.flink.streaming.api.transformations.ReduceTransformation;
+import org.apache.flink.streaming.api.transformations.SideOutputTransformation;
+import org.apache.flink.streaming.api.transformations.TwoInputTransformation;
+import org.apache.flink.streaming.api.transformations.UnionTransformation;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A specialized stream execution environment that allows users to first
construct a subgraph and
+ * later copy the transformations into the actual environment. During the
copying it could apply
+ * some kinds of {@link OperatorWrapper} to change the operators in each
transformation.
+ */
+public class DraftExecutionEnvironment extends StreamExecutionEnvironment {
+
+@SuppressWarnings("rawtypes")
+private static final Map,
DraftTransformationTranslator>
+translators = new HashMap<>();
+
+static {
Review comment:
Currently all the limitation is in the doc of `IterationBody` and refer
from `Iterations`, I added the new document there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-24663) PyFlink failed to get the site packege path because of SyntaxError in shell command
[
https://issues.apache.org/jira/browse/FLINK-24663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17436028#comment-17436028
]
jackwangcs commented on FLINK-24663:
In fact, we can only quote the long python command part like below:
{code:java}
python3 -c 'import sys;from distutils.dist import Distribution;install_obj =
Distribution().get_command_obj('\''install'\'',
create=True);print(sys.argv[1]);install_obj.prefix =
sys.argv[1];install_obj.finalize_options();installed_dir =
[install_obj.install_purelib];install_obj.install_purelib !=
install_obj.install_platlib and
installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> 1 and print(installed_dir[1])' /tmp/requirements
/tmp/requirements
/tmp/requirements/lib/python3.6/site-packages {code}
I will try to fix this by quoting the python argument.
> PyFlink failed to get the site packege path because of SyntaxError in shell
> command
> ---
>
> Key: FLINK-24663
> URL: https://issues.apache.org/jira/browse/FLINK-24663
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.12.3
>Reporter: jackwangcs
>Priority: Major
>
> Flink throws an exception when it tries to install 3rd party dependencies:
> {code:java}
> Caused by: java.io.IOException: Failed to execute the command: python -c
> import sys;from distutils.dist import Distribution;install_obj =
> Distribution().get_command_obj('install', create=True);install_obj.prefix =
> sys.argv[1];install_obj.finalize_options();installed_dir =
> [install_obj.install_purelib];install_obj.install_purelib !=
> install_obj.install_platlib and
> installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> > 1 and print(installed_dir[1])
> /mnt/yarn/usercache/hadoop/appcache/application_1629776785656_0100/python-dist-fb549cea-0857-4b11-9eb7-7818eaa3f561/python-requirements
> output: File "", line 1
> import sys;from distutils.dist import Distribution;install_obj =
> Distribution().get_command_obj('install', create=True);install_obj.prefix =
> sys.argv[1];install_obj.finalize_options();installed_dir =
> [install_obj.install_purelib];install_obj.install_purelib !=
> install_obj.install_platlib and
> installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> > 1 and print(installed_dir[1])
>
>
>
>
> ^SyntaxError: invalid syntax
> at
> org.apache.flink.python.util.PythonEnvironmentManagerUtils.execute(PythonEnvironmentManagerUtils.java:211)
> at
> org.apache.flink.python.util.PythonEnvironmentManagerUtils.getSitePackagesPath(PythonEnvironmentManagerUtils.java:171)
> at
> org.apache.flink.python.util.PythonEnvironmentManagerUtils.pipInstallRequirements(PythonEnvironmentManagerUtils.java:99)
> at
> org.apache.flink.python.env.beam.ProcessPythonEnvironmentManager.createEnvironment(ProcessPythonEnvironmentManager.java:169)
> at
> org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.createPythonExecutionEnvironment(BeamPythonFunctionRunner.java:339)
> {code}
> This can be reproduced by running the python script in a bash shell:
> {code:java}
> python3 -c import sys;from distutils.dist import Distribution;install_obj =
> Distribution().get_command_obj('install',
> create=True);print(sys.argv[1]);install_obj.prefix =
> sys.argv[1];install_obj.finalize_options();installed_dir =
> [install_obj.install_purelib];install_obj.install_purelib !=
> install_obj.install_platlib and
> installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> > 1 and print(installed_dir[1]) /tmp/requirements
> -bash: syntax error near unexpected token `(' {code}
> The solution is to quote all argements to execute:
> {code:java}
> python3 "-c" "import sys;from distutils.dist import Distribution;install_obj
> = Distribution().get_command_obj('install',
> create=True);print(sys.argv[1]);install_obj.prefix =
> sys.argv[1];install_obj.finalize_options();installed_dir =
> [install_obj.install_purelib];install_obj.install_purelib !=
> install_obj.install_platlib and
> installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> > 1 and print(installed_dir[1])" "/tmp/requirements"
> /tmp/requirements
> /tmp/requirements/lib/python3.6/site-packages{code}
--
[GitHub] [flink] flinkbot edited a comment on pull request #17345: [FLINK-24227][connectors] Added Kinesis Data Streams Sink i…
flinkbot edited a comment on pull request #17345: URL: https://github.com/apache/flink/pull/17345#issuecomment-926109717 ## CI report: * 4358f7215221955c9e6a3735c9bcd327a42c8c43 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25647) * 5c8ae3d4103b875b368cebb5ab9376c2cff21096 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25649) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17345: [FLINK-24227][connectors] Added Kinesis Data Streams Sink i…
flinkbot edited a comment on pull request #17345: URL: https://github.com/apache/flink/pull/17345#issuecomment-926109717 ## CI report: * 4358f7215221955c9e6a3735c9bcd327a42c8c43 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25647) * 5c8ae3d4103b875b368cebb5ab9376c2cff21096 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] gaoyunhaii commented on a change in pull request #11: [FLINK-24649][iteration] Add DraftExecutionEnvironment to support wrapping operators during compile time
gaoyunhaii commented on a change in pull request #11:
URL: https://github.com/apache/flink-ml/pull/11#discussion_r739312559
##
File path:
flink-ml-iteration/src/main/java/org/apache/flink/iteration/compile/DraftExecutionEnvironment.java
##
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.iteration.compile;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.JobExecutionResult;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.core.execution.DefaultExecutorServiceLoader;
+import
org.apache.flink.iteration.compile.translator.BroadcastStateTransformationTranslator;
+import
org.apache.flink.iteration.compile.translator.KeyedBroadcastStateTransformationTranslator;
+import
org.apache.flink.iteration.compile.translator.MultipleInputTransformationTranslator;
+import
org.apache.flink.iteration.compile.translator.OneInputTransformationTranslator;
+import
org.apache.flink.iteration.compile.translator.PartitionTransformationTranslator;
+import
org.apache.flink.iteration.compile.translator.ReduceTransformationTranslator;
+import
org.apache.flink.iteration.compile.translator.SideOutputTransformationTranslator;
+import
org.apache.flink.iteration.compile.translator.TwoInputTransformationTranslator;
+import
org.apache.flink.iteration.compile.translator.UnionTransformationTranslator;
+import org.apache.flink.iteration.operator.OperatorWrapper;
+import org.apache.flink.iteration.utils.ReflectionUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import
org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction;
+import org.apache.flink.streaming.api.graph.StreamGraph;
+import
org.apache.flink.streaming.api.transformations.BroadcastStateTransformation;
+import
org.apache.flink.streaming.api.transformations.KeyedBroadcastStateTransformation;
+import
org.apache.flink.streaming.api.transformations.KeyedMultipleInputTransformation;
+import
org.apache.flink.streaming.api.transformations.MultipleInputTransformation;
+import org.apache.flink.streaming.api.transformations.OneInputTransformation;
+import org.apache.flink.streaming.api.transformations.PartitionTransformation;
+import org.apache.flink.streaming.api.transformations.ReduceTransformation;
+import org.apache.flink.streaming.api.transformations.SideOutputTransformation;
+import org.apache.flink.streaming.api.transformations.TwoInputTransformation;
+import org.apache.flink.streaming.api.transformations.UnionTransformation;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A specialized stream execution environment that allows users to first
construct a subgraph and
+ * later copy the transformations into the actual environment. During the
copying it could apply
+ * some kinds of {@link OperatorWrapper} to change the operators in each
transformation.
+ */
+public class DraftExecutionEnvironment extends StreamExecutionEnvironment {
+
+@SuppressWarnings("rawtypes")
+private static final Map,
DraftTransformationTranslator>
+translators = new HashMap<>();
+
+static {
+translators.put(
+BroadcastStateTransformation.class, new
BroadcastStateTransformationTranslator());
+translators.put(
+KeyedBroadcastStateTransformation.class,
+new KeyedBroadcastStateTransformationTranslator());
+translators.put(
+KeyedMultipleInputTransformation.class,
+new KeyedBroadcastStateTransformationTranslator());
+translators.put(
+MultipleInputTransformation.class, new
MultipleInputTransformationTranslator());
+translators.put(OneInputTransformation.class, new
OneInputTransformationTranslator());
+translators.put(PartitionTransformation.class, new
PartitionTransformationTranslator());
+
[GitHub] [flink-ml] gaoyunhaii commented on a change in pull request #11: [FLINK-24649][iteration] Add DraftExecutionEnvironment to support wrapping operators during compile time
gaoyunhaii commented on a change in pull request #11:
URL: https://github.com/apache/flink-ml/pull/11#discussion_r739311463
##
File path:
flink-ml-iteration/src/main/java/org/apache/flink/iteration/compile/DraftTransformationTranslator.java
##
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.iteration.compile;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.iteration.operator.OperatorWrapper;
+
+/** Creates the actual transformation according to the draft transformation. */
+public interface DraftTransformationTranslator> {
+
+Transformation translate(
+TF draftTransformation, OperatorWrapper operatorWrapper,
Context context);
+
+/** The context for {@link DraftTransformationTranslator}. */
+interface Context {
+
+Transformation getActualTransformation(int draftId);
+
+ExecutionConfig getExecutionConfig();
+
+default Transformation copyProperties(
Review comment:
We should have set the parallelism~
And for resources, it seems currently these interface is already outdated
and not used~?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17345: [FLINK-24227][connectors] Added Kinesis Data Streams Sink i…
flinkbot edited a comment on pull request #17345: URL: https://github.com/apache/flink/pull/17345#issuecomment-926109717 ## CI report: * 4358f7215221955c9e6a3735c9bcd327a42c8c43 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25647) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] gaoyunhaii commented on a change in pull request #11: [FLINK-24649][iteration] Add DraftExecutionEnvironment to support wrapping operators during compile time
gaoyunhaii commented on a change in pull request #11:
URL: https://github.com/apache/flink-ml/pull/11#discussion_r739309491
##
File path:
flink-ml-iteration/src/main/java/org/apache/flink/iteration/compile/DraftTransformationTranslator.java
##
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.iteration.compile;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.iteration.operator.OperatorWrapper;
+
+/** Creates the actual transformation according to the draft transformation. */
+public interface DraftTransformationTranslator> {
+
+Transformation translate(
+TF draftTransformation, OperatorWrapper operatorWrapper,
Context context);
+
+/** The context for {@link DraftTransformationTranslator}. */
+interface Context {
+
+Transformation getActualTransformation(int draftId);
+
+ExecutionConfig getExecutionConfig();
+
+default Transformation copyProperties(
+Transformation actual, Transformation draft) {
+actual.setName(draft.getName());
+actual.setParallelism(draft.getParallelism());
+
+if (draft.getMaxParallelism() > 0) {
+actual.setMaxParallelism(draft.getMaxParallelism());
+}
+
+if (draft.getBufferTimeout() > 0) {
+actual.setBufferTimeout(draft.getBufferTimeout());
+}
+
+if (draft.getSlotSharingGroup().isPresent()) {
+actual.setSlotSharingGroup(draft.getSlotSharingGroup().get());
+}
+actual.setCoLocationGroupKey(draft.getCoLocationGroupKey());
+
+actual.setUid(draft.getUid());
+if (draft.getUserProvidedNodeHash() != null) {
Review comment:
We might not directly set null since it has `checkNotNull` checks ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17345: [FLINK-24227][connectors] Added Kinesis Data Streams Sink i…
flinkbot edited a comment on pull request #17345: URL: https://github.com/apache/flink/pull/17345#issuecomment-926109717 ## CI report: * 4358f7215221955c9e6a3735c9bcd327a42c8c43 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25647) * 5c8ae3d4103b875b368cebb5ab9376c2cff21096 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17345: [FLINK-24227][connectors] Added Kinesis Data Streams Sink i…
flinkbot edited a comment on pull request #17345: URL: https://github.com/apache/flink/pull/17345#issuecomment-926109717 ## CI report: * 4358f7215221955c9e6a3735c9bcd327a42c8c43 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25647) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17345: [FLINK-24227][connectors] Added Kinesis Data Streams Sink i…
flinkbot edited a comment on pull request #17345: URL: https://github.com/apache/flink/pull/17345#issuecomment-926109717 ## CI report: * 4358f7215221955c9e6a3735c9bcd327a42c8c43 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25647) * 5c8ae3d4103b875b368cebb5ab9376c2cff21096 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17608: [FLINK-24550][rpc] Use ContextClassLoader for message deserialization
flinkbot edited a comment on pull request #17608: URL: https://github.com/apache/flink/pull/17608#issuecomment-954803172 ## CI report: * 49487256bfb6ae8d304a42330454651924d001fb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25648) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17345: [FLINK-24227][connectors] Added Kinesis Data Streams Sink i…
flinkbot edited a comment on pull request #17345: URL: https://github.com/apache/flink/pull/17345#issuecomment-926109717 ## CI report: * 4358f7215221955c9e6a3735c9bcd327a42c8c43 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25647) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17604: [FLINK-23919][table-planner-blink] Fix field name conflict bug in `WindowUtil`
flinkbot edited a comment on pull request #17604: URL: https://github.com/apache/flink/pull/17604#issuecomment-954664725 ## CI report: * db26794d5bd076c1389715d25df346ae8f0edf72 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25639) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17608: [FLINK-24550][rpc] Use ContextClassLoader for message deserialization
flinkbot commented on pull request #17608: URL: https://github.com/apache/flink/pull/17608#issuecomment-954803417 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 49487256bfb6ae8d304a42330454651924d001fb (Fri Oct 29 14:46:10 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17608: [FLINK-24550][rpc] Use ContextClassLoader for message deserialization
flinkbot commented on pull request #17608: URL: https://github.com/apache/flink/pull/17608#issuecomment-954803172 ## CI report: * 49487256bfb6ae8d304a42330454651924d001fb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17345: [FLINK-24227][connectors] Added Kinesis Data Streams Sink i…
flinkbot edited a comment on pull request #17345: URL: https://github.com/apache/flink/pull/17345#issuecomment-926109717 ## CI report: * 4358f7215221955c9e6a3735c9bcd327a42c8c43 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25647) * 5c8ae3d4103b875b368cebb5ab9376c2cff21096 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-24550) Can not access job information from a standby jobmanager UI
[
https://issues.apache.org/jira/browse/FLINK-24550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-24550:
---
Labels: pull-request-available (was: )
> Can not access job information from a standby jobmanager UI
> ---
>
> Key: FLINK-24550
> URL: https://issues.apache.org/jira/browse/FLINK-24550
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination, Runtime / Web Frontend
>Affects Versions: 1.14.0
>Reporter: Dawid Wysakowicz
>Assignee: Chesnay Schepler
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.15.0, 1.14.1
>
>
> One can not access the "running jobs" section (if a job is running) or if the
> job is completed it can not access the job page. Moreover the overview
> section does not work in the standby manager if a job is running. The active
> jobmanager UI works just fine.
> {code}
> 2021-10-14 15:45:11,483 ERROR
> org.apache.flink.runtime.rest.handler.job.JobExceptionsHandler [] - Unhandled
> exception.
> java.util.concurrent.CancellationException: null
> at
> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2263)
> ~[?:1.8.0_231]
> at
> org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraphInternal(DefaultExecutionGraphCache.java:98)
> ~[flink-dist_2.12-1.14.0.jar:1.14.0]
> at
> org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraphInfo(DefaultExecutionGraphCache.java:67)
> ~[flink-dist_2.12-1.14.0.jar:1.14.0]
> at
> org.apache.flink.runtime.rest.handler.job.AbstractExecutionGraphHandler.handleRequest(AbstractExecutionGraphHandler.java:81)
> ~[flink-dist_2.12-1.14.0.jar:1.14.0]
> at
> org.apache.flink.runtime.rest.handler.AbstractRestHandler.respondToRequest(AbstractRestHandler.java:83)
> ~[flink-dist_2.12-1.14.0.jar:1.14.0]
> at
> org.apache.flink.runtime.rest.handler.AbstractHandler.respondAsLeader(AbstractHandler.java:195)
> ~[flink-dist_2.12-1.14.0.jar:1.14.0]
> at
> org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.lambda$channelRead0$0(LeaderRetrievalHandler.java:83)
> ~[flink-dist_2.12-1.14.0.jar:1.14.0]
> at java.util.Optional.ifPresent(Optional.java:159) [?:1.8.0_231]
> at
> org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer.java:45)
> [flink-dist_2.12-1.14.0.jar:1.14.0]
> at
> org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:80)
> [flink-dist_2.12-1.14.0.jar:1.14.0]
> at
> org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:49)
> [flink-dist_2.12-1.14.0.jar:1.14.0]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
> [flink-dist_2.12-1.14.0.jar:1.14.0]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> [flink-dist_2.12-1.14.0.jar:1.14.0]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> [flink-dist_2.12-1.14.0.jar:1.14.0]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> [flink-dist_2.12-1.14.0.jar:1.14.0]
> at
> org.apache.flink.runtime.rest.handler.router.RouterHandler.routed(RouterHandler.java:115)
> [flink-dist_2.12-1.14.0.jar:1.14.0]
> at
> org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:94)
> [flink-dist_2.12-1.14.0.jar:1.14.0]
> at
> org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:55)
> [flink-dist_2.12-1.14.0.jar:1.14.0]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
> [flink-dist_2.12-1.14.0.jar:1.14.0]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> [flink-dist_2.12-1.14.0.jar:1.14.0]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> [flink-dist_2.12-1.14.0.jar:1.14.0]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> [flink-dist_2.12-1.14.0.jar:1.14.0]
> at
>
[GitHub] [flink] flinkbot edited a comment on pull request #17345: [FLINK-24227][connectors] Added Kinesis Data Streams Sink i…
flinkbot edited a comment on pull request #17345: URL: https://github.com/apache/flink/pull/17345#issuecomment-926109717 ## CI report: * 4358f7215221955c9e6a3735c9bcd327a42c8c43 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25647) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol opened a new pull request #17608: [FLINK-24550][rpc] Use ContextClassLoader for message deserialization
zentol opened a new pull request #17608: URL: https://github.com/apache/flink/pull/17608 Based on #17606. We now use the Flink ClassLoader to deserialize the message payload. This should be safe because this serialization only occurs for results from the Flink layer, not the Akka RPC layer (where the Akka classloader would be required for deserialization). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-benchmarks] pnowojski merged pull request #38: [FLINK-24018] Remove Scala dependencies from flink dependencies
pnowojski merged pull request #38: URL: https://github.com/apache/flink-benchmarks/pull/38 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17345: [FLINK-24227][connectors] Added Kinesis Data Streams Sink i…
flinkbot edited a comment on pull request #17345: URL: https://github.com/apache/flink/pull/17345#issuecomment-926109717 ## CI report: * 4358f7215221955c9e6a3735c9bcd327a42c8c43 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25647) * 5c8ae3d4103b875b368cebb5ab9376c2cff21096 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17345: [FLINK-24227][connectors] Added Kinesis Data Streams Sink i…
flinkbot edited a comment on pull request #17345: URL: https://github.com/apache/flink/pull/17345#issuecomment-926109717 ## CI report: * 4358f7215221955c9e6a3735c9bcd327a42c8c43 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25647) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] gaoyunhaii commented on a change in pull request #11: [FLINK-24649][iteration] Add DraftExecutionEnvironment to support wrapping operators during compile time
gaoyunhaii commented on a change in pull request #11:
URL: https://github.com/apache/flink-ml/pull/11#discussion_r739284296
##
File path:
flink-ml-iteration/src/main/java/org/apache/flink/iteration/compile/DraftExecutionEnvironment.java
##
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.iteration.compile;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.JobExecutionResult;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.core.execution.DefaultExecutorServiceLoader;
+import
org.apache.flink.iteration.compile.translator.BroadcastStateTransformationTranslator;
+import
org.apache.flink.iteration.compile.translator.KeyedBroadcastStateTransformationTranslator;
+import
org.apache.flink.iteration.compile.translator.MultipleInputTransformationTranslator;
+import
org.apache.flink.iteration.compile.translator.OneInputTransformationTranslator;
+import
org.apache.flink.iteration.compile.translator.PartitionTransformationTranslator;
+import
org.apache.flink.iteration.compile.translator.ReduceTransformationTranslator;
+import
org.apache.flink.iteration.compile.translator.SideOutputTransformationTranslator;
+import
org.apache.flink.iteration.compile.translator.TwoInputTransformationTranslator;
+import
org.apache.flink.iteration.compile.translator.UnionTransformationTranslator;
+import org.apache.flink.iteration.operator.OperatorWrapper;
+import org.apache.flink.iteration.utils.ReflectionUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import
org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction;
+import org.apache.flink.streaming.api.graph.StreamGraph;
+import
org.apache.flink.streaming.api.transformations.BroadcastStateTransformation;
+import
org.apache.flink.streaming.api.transformations.KeyedBroadcastStateTransformation;
+import
org.apache.flink.streaming.api.transformations.KeyedMultipleInputTransformation;
+import
org.apache.flink.streaming.api.transformations.MultipleInputTransformation;
+import org.apache.flink.streaming.api.transformations.OneInputTransformation;
+import org.apache.flink.streaming.api.transformations.PartitionTransformation;
+import org.apache.flink.streaming.api.transformations.ReduceTransformation;
+import org.apache.flink.streaming.api.transformations.SideOutputTransformation;
+import org.apache.flink.streaming.api.transformations.TwoInputTransformation;
+import org.apache.flink.streaming.api.transformations.UnionTransformation;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A specialized stream execution environment that allows users to first
construct a subgraph and
+ * later copy the transformations into the actual environment. During the
copying it could apply
+ * some kinds of {@link OperatorWrapper} to change the operators in each
transformation.
+ */
+public class DraftExecutionEnvironment extends StreamExecutionEnvironment {
+
+@SuppressWarnings("rawtypes")
+private static final Map,
DraftTransformationTranslator>
+translators = new HashMap<>();
+
+static {
+translators.put(
+BroadcastStateTransformation.class, new
BroadcastStateTransformationTranslator());
+translators.put(
+KeyedBroadcastStateTransformation.class,
+new KeyedBroadcastStateTransformationTranslator());
+translators.put(
+KeyedMultipleInputTransformation.class,
+new KeyedBroadcastStateTransformationTranslator());
+translators.put(
+MultipleInputTransformation.class, new
MultipleInputTransformationTranslator());
+translators.put(OneInputTransformation.class, new
OneInputTransformationTranslator());
+translators.put(PartitionTransformation.class, new
PartitionTransformationTranslator());
+
[GitHub] [flink] flinkbot edited a comment on pull request #17598: [WIP][FLINK-24703][connectors][formats] Add FileSource support for reading CSV files.
flinkbot edited a comment on pull request #17598: URL: https://github.com/apache/flink/pull/17598#issuecomment-954282485 ## CI report: * ee8fa611b135924ad3a083ff71dbcbc2e1895351 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25635) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
