[GitHub] [flink] flinkbot commented on issue #8419: bugfix: increased distance in 100ms is not rigorous enough
flinkbot commented on issue #8419: bugfix: increased distance in 100ms is not rigorous enough URL: https://github.com/apache/flink/pull/8419#issuecomment-491564526 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhang-xiang-career opened a new pull request #8419: bugfix: increased distance in 100ms is not rigorous enough
zhang-xiang-career opened a new pull request #8419: bugfix: increased distance in 100ms is not rigorous enough URL: https://github.com/apache/flink/pull/8419 In the official streaming example of `org.apache.flink.streaming.examples.windowing.TopSpeedWindowing` , increased distance in 100ms is not rigorous enough with the value of speed/3.6d, it should be the value of speed/36d. Checking process like this: >incremental distance = ( km.h * 1000 / 3600) * 0.1 (s) ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluser with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] Aitozi commented on issue #8280: [FLINK-12297]Harden ClosureCleaner to handle the wrapped function
Aitozi commented on issue #8280: [FLINK-12297]Harden ClosureCleaner to handle the wrapped function URL: https://github.com/apache/flink/pull/8280#issuecomment-491563946 Hi, @StephanEwen I have followed your suggestion to add some fix. Still two have to confim: 1. check at the top level with the condition you mentioned may skip some class wrong, think about a class has no enclosingClass it may also have to be cleaned, so i don't add the check. 2. I found that the scala class clean is working with `ClosureCleaner.scala`. I think it should be consistent with the java clean action. I am not quiet familiar with the scala, but i will try to look into it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] walterddr commented on a change in pull request #8402: [FLINK-12473][ml] Add the interface of ML pipeline and ML lib
walterddr commented on a change in pull request #8402: [FLINK-12473][ml] Add
the interface of ML pipeline and ML lib
URL: https://github.com/apache/flink/pull/8402#discussion_r283117427
##
File path:
flink-ml/flink-ml-api/src/main/java/org/apache/flink/ml/api/core/Pipeline.java
##
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.api.core;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.ml.api.misc.param.Params;
+import org.apache.flink.ml.util.persist.MLStageFactory;
+import org.apache.flink.table.api.Table;
+
+import com.google.gson.Gson;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonObject;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * A pipeline is a linear workflow which chains {@link Estimator}s and {@link
Transformer}s to
+ * execute an algorithm.
+ *
+ * A pipeline itself can either act as an Estimator or a Transformer,
depending on the stages it
+ * includes. More specifically:
+ *
+ *
+ * If a Pipeline has an {@link Estimator}, one needs to call {@link
Pipeline#fit(Table)} before use
+ * the pipeline as a {@link Transformer}. In this case the Pipeline is an
{@link Estimator} and can
+ * produce a Pipeline as a {@link Model}.
+ *
+ *
+ * If a Pipeline has no {@link Estimator}, it is a {@link Transformer} and can
be applied to a Table
+ * directly. In this case, {@link Pipeline#fit(Table)} will simply return the
pipeline itself.
+ *
+ *
+ *
+ * In addition, a pipeline can also be used as a {@link PipelineStage} in
another pipeline, just
+ * like an ordinary {@link Estimator} or {@link Transformer} as describe above.
+ */
+@PublicEvolving
+public final class Pipeline implements Estimator,
Transformer,
+ Model {
+ private static final long serialVersionUID = 1L;
+ private List stages;
+ private Params params;
+
+ public Pipeline() {
+ this(new ArrayList<>());
+ }
+
+ public Pipeline(List stages) {
+ this.stages = stages;
+ this.params = new Params();
+ }
+
+ private static boolean isStageNeedFit(PipelineStage stage) {
+ return (stage instanceof Pipeline && ((Pipeline)
stage).needFit()) ||
+ (!(stage instanceof Pipeline) && stage instanceof
Estimator);
+ }
+
+ /**
+* Appends a PipelineStage to the tail of this pipeline.
+*
+* @param stage the stage to be appended
+*/
+ public Pipeline appendStage(PipelineStage stage) {
+ stages.add(stage);
+ return this;
+ }
+
+ /**
+* Returns a list of all stages in this pipeline in order.
+*
+* @return a list of all stages in this pipeline in order.
+*/
+ public List getStages() {
+ return stages;
+ }
+
+ /**
+* Check whether the pipeline acts as an {@link Estimator} or not. When
the return value is
+* true, that means this pipeline contains an {@link Estimator} and
thus users must invoke
+* {@link #fit(Table)} before they can use this pipeline as a {@link
Transformer}. Otherwise,
+* the pipeline can be used as a {@link Transformer} directly.
+*
+* @return {@code true} if this pipeline has an Estimator, {@code
false} otherwise
+*/
+ public boolean needFit() {
+ return this.getIndexOfLastEstimator() >= 0;
+ }
+
+ public Params getParams() {
+ return params;
+ }
+
+ //find the last Estimator or Pipeline that needs fit in stages, -1
stand for no Estimator in Pipeline
+ private int getIndexOfLastEstimator() {
+ int lastEstimatorIndex = -1;
+ for (int i = 0; i < stages.size(); i++) {
+ PipelineStage stage = stages.get(i);
+ lastEstimatorIndex = isStageNeedFit(stage) ? i :
lastEstimatorIndex;
+ }
+ return lastEstimatorIndex;
+ }
+
+ @Override
+ /**
+* Train the pipeline to fit on the records in the given {@link Table}.
+
[GitHub] [flink] walterddr commented on a change in pull request #8402: [FLINK-12473][ml] Add the interface of ML pipeline and ML lib
walterddr commented on a change in pull request #8402: [FLINK-12473][ml] Add
the interface of ML pipeline and ML lib
URL: https://github.com/apache/flink/pull/8402#discussion_r283117478
##
File path:
flink-ml/flink-ml-api/src/main/java/org/apache/flink/ml/api/misc/param/ParamInfo.java
##
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.api.misc.param;
+
+import org.apache.flink.annotation.PublicEvolving;
+
+/**
+ * Definition of a parameter, including name, type, default value, validator
and so on.
+ *
+ * This class is provided to unify the interaction with parameters.
+ *
+ * @param the type of the param value
+ */
+@PublicEvolving
+public class ParamInfo {
+ private final String name;
+ private final String description;
+ private final boolean isOptional;
+ private final boolean hasDefaultValue;
+ private final V defaultValue;
+ private final ParamValidator validator;
+ private final Class valueClass;
+
+ public ParamInfo(String name, String description, Class valueClass) {
+ this(name, description, true, false, null, null, valueClass);
+ }
+
+ public ParamInfo(String name, String description, V defaultValue,
Class valueClass) {
+ this(name, description, true, true, defaultValue, null,
valueClass);
+ }
+
+ public ParamInfo(String name, String description, boolean isOptional,
Class valueClass) {
+ this(name, description, isOptional, false, null, null,
valueClass);
+ }
+
+ public ParamInfo(String name, String description, boolean isOptional, V
defaultValue, Class valueClass) {
+ this(name, description, isOptional, true, defaultValue, null,
valueClass);
+ }
+
+ public ParamInfo(String name, String description, boolean isOptional, V
defaultValue, ParamValidator validator, Class valueClass) {
+ this(name, description, isOptional, true, defaultValue,
validator, valueClass);
+ }
+
+ private ParamInfo(String name, String description, boolean isOptional,
boolean hasDefaultValue, V defaultValue, ParamValidator validator, Class
valueClass) {
+ this.name = name;
+ this.description = description;
+ this.isOptional = isOptional;
+ this.hasDefaultValue = hasDefaultValue;
+ this.defaultValue = defaultValue;
+ this.validator = validator;
+ this.valueClass = valueClass;
+ }
+
+ /**
+* Returns the name of the parameter.
Review comment:
might be good to explain `name` needs to be unique within a pipeline (since
it is used for the paramMap as key)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] walterddr commented on a change in pull request #8402: [FLINK-12473][ml] Add the interface of ML pipeline and ML lib
walterddr commented on a change in pull request #8402: [FLINK-12473][ml] Add
the interface of ML pipeline and ML lib
URL: https://github.com/apache/flink/pull/8402#discussion_r283117318
##
File path:
flink-ml/flink-ml-api/src/main/java/org/apache/flink/ml/api/core/Pipeline.java
##
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.api.core;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.ml.api.misc.param.Params;
+import org.apache.flink.ml.util.persist.MLStageFactory;
+import org.apache.flink.table.api.Table;
+
+import com.google.gson.Gson;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonObject;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * A pipeline is a linear workflow which chains {@link Estimator}s and {@link
Transformer}s to
+ * execute an algorithm.
+ *
+ * A pipeline itself can either act as an Estimator or a Transformer,
depending on the stages it
+ * includes. More specifically:
+ *
+ *
+ * If a Pipeline has an {@link Estimator}, one needs to call {@link
Pipeline#fit(Table)} before use
+ * the pipeline as a {@link Transformer}. In this case the Pipeline is an
{@link Estimator} and can
+ * produce a Pipeline as a {@link Model}.
+ *
+ *
+ * If a Pipeline has no {@link Estimator}, it is a {@link Transformer} and can
be applied to a Table
+ * directly. In this case, {@link Pipeline#fit(Table)} will simply return the
pipeline itself.
+ *
+ *
+ *
+ * In addition, a pipeline can also be used as a {@link PipelineStage} in
another pipeline, just
+ * like an ordinary {@link Estimator} or {@link Transformer} as describe above.
+ */
+@PublicEvolving
+public final class Pipeline implements Estimator,
Transformer,
+ Model {
+ private static final long serialVersionUID = 1L;
+ private List stages;
+ private Params params;
+
+ public Pipeline() {
+ this(new ArrayList<>());
+ }
+
+ public Pipeline(List stages) {
+ this.stages = stages;
+ this.params = new Params();
+ }
+
+ private static boolean isStageNeedFit(PipelineStage stage) {
+ return (stage instanceof Pipeline && ((Pipeline)
stage).needFit()) ||
+ (!(stage instanceof Pipeline) && stage instanceof
Estimator);
+ }
+
+ /**
+* Appends a PipelineStage to the tail of this pipeline.
+*
+* @param stage the stage to be appended
+*/
+ public Pipeline appendStage(PipelineStage stage) {
+ stages.add(stage);
+ return this;
+ }
+
+ /**
+* Returns a list of all stages in this pipeline in order.
+*
+* @return a list of all stages in this pipeline in order.
+*/
+ public List getStages() {
+ return stages;
+ }
+
+ /**
+* Check whether the pipeline acts as an {@link Estimator} or not. When
the return value is
+* true, that means this pipeline contains an {@link Estimator} and
thus users must invoke
+* {@link #fit(Table)} before they can use this pipeline as a {@link
Transformer}. Otherwise,
+* the pipeline can be used as a {@link Transformer} directly.
+*
+* @return {@code true} if this pipeline has an Estimator, {@code
false} otherwise
+*/
+ public boolean needFit() {
+ return this.getIndexOfLastEstimator() >= 0;
+ }
+
+ public Params getParams() {
+ return params;
+ }
+
+ //find the last Estimator or Pipeline that needs fit in stages, -1
stand for no Estimator in Pipeline
+ private int getIndexOfLastEstimator() {
Review comment:
+1. should save some traverse time since it is also used in `needFit()` API.
Another way to think about it. since `needFit()` is used more often. we can
make it
```
public boolean needFit() {
stages.stream().filter(s ->isStageNeedFit(s)).findAny()
}
```
and just skip the use of last index in `public Pipeline fit(Table input)`.
both approaches save time in traverse in case of long
[GitHub] [flink] walterddr commented on a change in pull request #8402: [FLINK-12473][ml] Add the interface of ML pipeline and ML lib
walterddr commented on a change in pull request #8402: [FLINK-12473][ml] Add
the interface of ML pipeline and ML lib
URL: https://github.com/apache/flink/pull/8402#discussion_r283117176
##
File path:
flink-ml/flink-ml-api/src/main/java/org/apache/flink/ml/api/core/PipelineStage.java
##
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.api.core;
+
+import org.apache.flink.ml.api.misc.param.ParamInfo;
+import org.apache.flink.ml.api.misc.param.WithParams;
+import org.apache.flink.ml.api.misc.persist.Persistable;
+import org.apache.flink.ml.util.param.ExtractParamInfosUtil;
+import org.apache.flink.ml.util.persist.MLStageFactory;
+
+import com.google.gson.Gson;
+import com.google.gson.JsonObject;
+
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Base class for a stage in a pipeline. The interface is only a concept, and
does not have any
+ * actual functionality. Its subclasses must be either Estimator or
Transformer. No other classes
+ * should inherit this interface directly.
+ *
+ * Each pipeline stage is with parameters and meanwhile persistable.
+ *
+ * @param The class type of the PipelineStage implementation itself, used
by {@link
+ *org.apache.flink.ml.api.misc.param.WithParams}
+ * @see WithParams
+ */
+interface PipelineStage> extends WithParams,
Serializable,
+ Persistable {
+
+ default String toJson() {
Review comment:
How about using `Serialize` / `Deserialize` with type settings ? I think
many use cases require the flexibility to serialized the pipeline in a way that
fits the client system need.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] walterddr commented on a change in pull request #8402: [FLINK-12473][ml] Add the interface of ML pipeline and ML lib
walterddr commented on a change in pull request #8402: [FLINK-12473][ml] Add
the interface of ML pipeline and ML lib
URL: https://github.com/apache/flink/pull/8402#discussion_r283117513
##
File path:
flink-ml/flink-ml-api/src/main/java/org/apache/flink/ml/api/misc/param/Params.java
##
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.api.misc.param;
+
+import org.apache.flink.annotation.PublicEvolving;
+
+import com.google.gson.Gson;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * The map-like container class for parameter. This class is provided to unify
the interaction with
+ * parameters.
+ */
+@PublicEvolving
+public class Params implements Serializable {
+ private final Map paramMap = new HashMap<>();
+
+ /**
+* Returns the value of the specific parameter, or default value
defined in the {@code info} if
+* this Params doesn't contain the param.
+*
+* @param info the info of the specific parameter, usually with default
value
+* @param the type of the specific parameter
+* @return the value of the specific parameter, or default value
defined in the {@code info} if
+* this Params doesn't contain the parameter
+* @throws RuntimeException if the Params doesn't contains the specific
parameter, while the
+* param is not optional but has no default
value in the {@code info}
+*/
+ @SuppressWarnings("unchecked")
+ public V get(ParamInfo info) {
+ V value = (V) paramMap.getOrDefault(info.getName(),
info.getDefaultValue());
+ if (value == null && !info.isOptional() &&
info.getDefaultValue() == null) {
Review comment:
This is a bit confusing from your previous `hasDefaultValue()` javadoc. if
`hasDefaultValue()` is added and here defaultValue cannot be null if it is not
optional. it seems like:
```
info.hasDefaultValue() === (!info.isOptional() && info.getDefaultValue() ==
null)
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] walterddr commented on a change in pull request #8402: [FLINK-12473][ml] Add the interface of ML pipeline and ML lib
walterddr commented on a change in pull request #8402: [FLINK-12473][ml] Add
the interface of ML pipeline and ML lib
URL: https://github.com/apache/flink/pull/8402#discussion_r283117530
##
File path:
flink-ml/flink-ml-api/src/main/java/org/apache/flink/ml/util/persist/MLStageFactory.java
##
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.util.persist;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.ml.api.misc.persist.Persistable;
+
+import com.google.gson.Gson;
+import com.google.gson.JsonObject;
+
+/**
+ * Utility to restore a PipelineStage from a stage json.
+ */
+@PublicEvolving
+public class MLStageFactory {
Review comment:
How about just calling it `PipelineStageFactory` ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-12380) Add thread name in the log4j.properties
[
https://issues.apache.org/jira/browse/FLINK-12380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837888#comment-16837888
]
Andrey Zagrebin commented on FLINK-12380:
-
[~yunta] it could be just for historical reasons. As Stephan mentioned, current
default log setting files are already used a lot by users of both logging
systems, changing either format might be unexpected and force users to
reconsider logging settings. If somebody wanted thread name there, they have
probably already provided their custom settings.
> Add thread name in the log4j.properties
> ---
>
> Key: FLINK-12380
> URL: https://issues.apache.org/jira/browse/FLINK-12380
> Project: Flink
> Issue Type: Improvement
> Components: Build System
>Reporter: Yun Tang
>Assignee: Yun Tang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> This is inspired by FLINK-12368 when users want to add sub-task index
> information in the source code. We could add thread name, which already
> contains sub-task index information, in the logs to avoid have to change the
> source code.
> Moreover, I found existing {{logback.xml}} in Flink already contains {{thread
> name}} information. We should also add this in the {{log4j.properties.}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [flink] flinkbot commented on issue #8418: [FLINK-12491][docs][configuration] Fix incorrect javadoc for path sep…
flinkbot commented on issue #8418: [FLINK-12491][docs][configuration] Fix incorrect javadoc for path sep… URL: https://github.com/apache/flink/pull/8418#issuecomment-491518838 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-12491) Incorrect documentation for directory path separators of CoreOptions.TMP_DIRS
[
https://issues.apache.org/jira/browse/FLINK-12491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-12491:
---
Labels: pull-request-available (was: )
> Incorrect documentation for directory path separators of CoreOptions.TMP_DIRS
> -
>
> Key: FLINK-12491
> URL: https://issues.apache.org/jira/browse/FLINK-12491
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Configuration
>Affects Versions: 1.6.4, 1.7.2, 1.8.0, 1.9.0
>Reporter: Kezhu Wang
>Assignee: Kezhu Wang
>Priority: Minor
> Labels: pull-request-available
>
> {{CoreOptions.TMP_DIRS}} and {{ConfigConstants.TASK_MANAGER_TMP_DIR_KEY}}
> both say that:
> {quote}
> The config parameter defining the directories for temporary files, separated
> by
>* ",", "|", or the system's \{@link java.io.File#pathSeparator}.
> {quote}
> But the parsing phase uses {{String.split}} with argument {{",|" +
> File.pathSeparator}} eventually. However, in fact the sole parameter of
> {{String.split}} is a regular expression, so the directory path separators
> are "," or {{java.io.File#pathSeparator}}. After digging into history, I
> found that the documentation was introduced in commit
> {{a7c407ace4f6cbfbde3e247071cee5a755ae66db}} and inherited by
> {{76abcaa55d0d6ab704b7ab8164718e8e2dcae2c4}}. So, I think it is safe to drop
> "|" from documentation.
> {code:title=ConfigurationUtils.java}
> public class ConfigurationUtils {
> private static String[] splitPaths(@Nonnull String separatedPaths) {
> return separatedPaths.length() > 0 ? separatedPaths.split(",|"
> + File.pathSeparator) : EMPTY;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [flink] kezhuw opened a new pull request #8418: [FLINK-12491][docs][configuration] Fix incorrect javadoc for path sep…
kezhuw opened a new pull request #8418: [FLINK-12491][docs][configuration] Fix incorrect javadoc for path sep… URL: https://github.com/apache/flink/pull/8418 …arators of CoreOptions.TMP_DIRS ## What is the purpose of the change Fix incorrect javadoc for path separators part of `CoreOptions.TMP_DIRS` and `ConfigConstants.TASK_MANAGER_TMP_DIR_KEY` ## Brief change log * Fix incorrect javadoc for path separators part of `CoreOptions.TMP_DIRS` and `ConfigConstants.TASK_MANAGER_TMP_DIR_KEY`. * Refactor `ConfigurationUtils.splitPaths` so we can test both unix and windows path separators without demand on ci test matrix. ## Verifying this change This change added tests and can be verified as follows: * Added test for documented path separators. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-12371) Add support for converting (NOT) IN/ (NOT) EXISTS to SemiJoin, and generating optimized logical plan
[
https://issues.apache.org/jira/browse/FLINK-12371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kurt Young closed FLINK-12371.
--
Resolution: Implemented
Fix Version/s: 1.9.0
merged in 1.9.0: 76ae39a18ecb29c49a0ce9a205517b2104daa616
> Add support for converting (NOT) IN/ (NOT) EXISTS to SemiJoin, and generating
> optimized logical plan
>
>
> Key: FLINK-12371
> URL: https://issues.apache.org/jira/browse/FLINK-12371
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / Planner
>Reporter: godfrey he
>Assignee: godfrey he
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.9.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> This issue aims to convert IN/EXISTS to semi-join, and NOT IN/NOT EXISTS to
> anti-join.
> In Calcite,
> [SemiJoin|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/core/SemiJoin.java]
> only represents semi-join, (could not represent anti-join) and requires equi
> join condition. Queries like `select * from left where left.a1 in (select
> right.a2 from right where left.b1 > right.b2)` and `select * from left where
> not exists (select * from right)` could not be converted to Calcite SemiJoin
> operator.
> To solve the above problem, We need copy the {{SemiJoin}} class to Flink, and
> make the following changes:
> 1. make {{SemiJoin}} extending from {{Join}}, not from {{EquiJoin}}. (to
> support non-equi join condition)
> 2. add {{isAnti}} field attribute to represent anti-join.
> Currently, there are no rules to convert (NOT) IN/ (NOT) EXISTS to SemiJoin,
> so we need a whole new rule (named {{FlinkSubQueryRemoveRule}}) to meet our
> requirement.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [flink] KurtYoung merged pull request #8317: [FLINK-12371] [table-planner-blink] Add support for converting (NOT) IN/ (NOT) EXISTS to semi-join/anti-join, and generating optimized logical pla
KurtYoung merged pull request #8317: [FLINK-12371] [table-planner-blink] Add support for converting (NOT) IN/ (NOT) EXISTS to semi-join/anti-join, and generating optimized logical plan URL: https://github.com/apache/flink/pull/8317 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #8362: [FLINK-11391] Introduce shuffle master interface
flinkbot edited a comment on issue #8362: [FLINK-11391] Introduce shuffle master interface URL: https://github.com/apache/flink/pull/8362#issuecomment-490147915 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❗ 3. Needs [attention] from. - Needs attention by @tillrohrmann [PMC], @zhijiangW * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #8359: [FLINK-11051][table] Add streaming window FlatAggregate to Table API
flinkbot edited a comment on issue #8359: [FLINK-11051][table] Add streaming window FlatAggregate to Table API URL: https://github.com/apache/flink/pull/8359#issuecomment-490064489 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ❓ 1. The [description] looks good. * ✅ 2. There is [consensus] that the contribution should go into to Flink. - Approved by @sunjincheng121 [committer] * ❗ 3. Needs [attention] from. - Needs attention by @aljoscha [PMC], @dawidwys [committer], @twalthr [PMC] * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #8320: [FLINK-12201][network][metrics] Introduce InputGateWithMetrics in Task to increment numBytesIn metric
flinkbot edited a comment on issue #8320: [FLINK-12201][network][metrics] Introduce InputGateWithMetrics in Task to increment numBytesIn metric URL: https://github.com/apache/flink/pull/8320#issuecomment-487902387 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ✅ 1. The [description] looks good. - Approved by @azagrebin * ✅ 2. There is [consensus] that the contribution should go into to Flink. - Approved by @azagrebin * ❗ 3. Needs [attention] from. - Needs attention by @azagrebin * ✅ 4. The change fits into the overall [architecture]. - Approved by @azagrebin * ✅ 5. Overall code [quality] is good. - Approved by @azagrebin Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #8393: [hotfix][table] Fix typo for method name in FieldReferenceLookup
flinkbot edited a comment on issue #8393: [hotfix][table] Fix typo for method name in FieldReferenceLookup URL: https://github.com/apache/flink/pull/8393#issuecomment-491125622 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ✅ 1. The [description] looks good. - Approved by @sunjincheng121 [committer] * ✅ 2. There is [consensus] that the contribution should go into to Flink. - Approved by @sunjincheng121 [committer] * ❓ 3. Needs [attention] from. * ✅ 4. The change fits into the overall [architecture]. - Approved by @sunjincheng121 [committer] * ✅ 5. Overall code [quality] is good. - Approved by @sunjincheng121 [committer] Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #8389: [FLINK-12399][table] Fix FilterableTableSource does not change after applyPredicate
flinkbot edited a comment on issue #8389: [FLINK-12399][table] Fix FilterableTableSource does not change after applyPredicate URL: https://github.com/apache/flink/pull/8389#issuecomment-490965405 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❗ 3. Needs [attention] from. - Needs attention by @beyond1920, @godfreyhe * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #8347: [FLINK-12326][python] Add basic test framework for python api
flinkbot edited a comment on issue #8347: [FLINK-12326][python] Add basic test framework for python api URL: https://github.com/apache/flink/pull/8347#issuecomment-489486437 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ✅ 1. The [description] looks good. - Approved by @sunjincheng121 [committer] * ✅ 2. There is [consensus] that the contribution should go into to Flink. - Approved by @sunjincheng121 [committer] * ❓ 3. Needs [attention] from. * ✅ 4. The change fits into the overall [architecture]. - Approved by @sunjincheng121 [committer] * ✅ 5. Overall code [quality] is good. - Approved by @sunjincheng121 [committer] Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #8416: [FLINK-12331] Introduce partition/gate setup to decouple task registration with NetworkEnvironment
flinkbot edited a comment on issue #8416: [FLINK-12331] Introduce partition/gate setup to decouple task registration with NetworkEnvironment URL: https://github.com/apache/flink/pull/8416#issuecomment-491418683 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❗ 3. Needs [attention] from. - Needs attention by @zhijiangw * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #7805: [FLINK-11697] Add readMapReduceTextFile API for HadoopInputs.
flinkbot edited a comment on issue #7805: [FLINK-11697] Add readMapReduceTextFile API for HadoopInputs. URL: https://github.com/apache/flink/pull/7805#issuecomment-466323210 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #8377: [FLINK-12455][python] Move the packaging of pyflink to flink-dist
flinkbot edited a comment on issue #8377: [FLINK-12455][python] Move the packaging of pyflink to flink-dist URL: https://github.com/apache/flink/pull/8377#issuecomment-490752300 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ✅ 1. The [description] looks good. - Approved by @sunjincheng121 [committer] * ✅ 2. There is [consensus] that the contribution should go into to Flink. - Approved by @sunjincheng121 [committer] * ❓ 3. Needs [attention] from. * ✅ 4. The change fits into the overall [architecture]. - Approved by @sunjincheng121 [committer] * ✅ 5. Overall code [quality] is good. - Approved by @sunjincheng121 [committer] Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #8365: [FLINK-12417][table] Unify ReadableCatalog and ReadableWritableCatalog interfaces to Catalog interface
flinkbot edited a comment on issue #8365: [FLINK-12417][table] Unify ReadableCatalog and ReadableWritableCatalog interfaces to Catalog interface URL: https://github.com/apache/flink/pull/8365#issuecomment-490221019 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❗ 3. Needs [attention] from. - Needs attention by @dawidwys [committer] * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #8324: [FLINK-11921][table] Upgrade to calcite 1.19
flinkbot edited a comment on issue #8324: [FLINK-11921][table] Upgrade to calcite 1.19 URL: https://github.com/apache/flink/pull/8324#issuecomment-488032392 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❗ 3. Needs [attention] from. - Needs attention by @twalthr [PMC], @wuchong [committer] * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #8383: [FLINK-12454][python] Add -l(list) -i(include) and -e(exclude) option…
flinkbot edited a comment on issue #8383: [FLINK-12454][python] Add -l(list) -i(include) and -e(exclude) option… URL: https://github.com/apache/flink/pull/8383#issuecomment-490831303 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ✅ 1. The [description] looks good. - Approved by @sunjincheng121 [committer] * ✅ 2. There is [consensus] that the contribution should go into to Flink. - Approved by @sunjincheng121 [committer] * ❓ 3. Needs [attention] from. * ✅ 4. The change fits into the overall [architecture]. - Approved by @sunjincheng121 [committer] * ✅ 5. Overall code [quality] is good. - Approved by @sunjincheng121 [committer] Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #7810: [FLINK-11698] Add readMapRedTextFile API for HadoopInputs.
flinkbot edited a comment on issue #7810: [FLINK-11698] Add readMapRedTextFile API for HadoopInputs. URL: https://github.com/apache/flink/pull/7810#issuecomment-466377616 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #8417: [hotfix][javadocs] Correct the example of select for Interface Table.
flinkbot commented on issue #8417: [hotfix][javadocs] Correct the example of select for Interface Table. URL: https://github.com/apache/flink/pull/8417#issuecomment-491499131 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wenhuitang opened a new pull request #8417: [hotfix][javadocs] Correct the example of select for Interface Table.
wenhuitang opened a new pull request #8417: [hotfix][javadocs] Correct the
example of select for Interface Table.
URL: https://github.com/apache/flink/pull/8417
The example of select in Table.java is "tab.select("key, value.avg + ' The
average' as average")" which can causes ValidationException "Cannot resolve
field [key], input field list:[TMP_0]". So correct the example.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] yanghua commented on issue #8322: [FLINK-12364] Introduce a CheckpointFailureManager to centralized manage checkpoint failure
yanghua commented on issue #8322: [FLINK-12364] Introduce a CheckpointFailureManager to centralized manage checkpoint failure URL: https://github.com/apache/flink/pull/8322#issuecomment-491497620 @StefanRRichter Sorry, I still do not provide a final implementation of this issue. It's a temporary version just for discussion. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] yanghua commented on a change in pull request #8322: [FLINK-12364] Introduce a CheckpointFailureManager to centralized manage checkpoint failure
yanghua commented on a change in pull request #8322: [FLINK-12364] Introduce a
CheckpointFailureManager to centralized manage checkpoint failure
URL: https://github.com/apache/flink/pull/8322#discussion_r283090199
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointFailureManager.java
##
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint;
+
+import org.apache.flink.util.FlinkRuntimeException;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.TreeMap;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * The checkpoint failure manager which centralized manage checkpoint failure
processing logic.
+ */
+public class CheckpointFailureManager {
+
+ private final static int MAXIMUM_TOLERABLE_FAILURE_NUMBER =
Integer.MAX_VALUE;
+
+ private final static short IGNORE_FLAG = 0;
+ private final static short COUNT_FLAG = 1;
+ private final static short SUCCEED_FLAG = -1;
+
+ private final int tolerableCpFailureNumber;
+ private final FailJobCallback failureCallback;
+ private final TreeMap serialCheckpointResultTable;
+
+ public CheckpointFailureManager(int tolerableCpFailureNumber,
FailJobCallback failureCallback) {
+ checkArgument(tolerableCpFailureNumber >= 0
+ && tolerableCpFailureNumber <
MAXIMUM_TOLERABLE_FAILURE_NUMBER,
+ "The tolerable checkpoint failure number is illegal, " +
+ "it must be greater than or equal to 0 and less
than " + MAXIMUM_TOLERABLE_FAILURE_NUMBER + ".");
+ this.tolerableCpFailureNumber = tolerableCpFailureNumber;
+ this.failureCallback = checkNotNull(failureCallback);
+ this.serialCheckpointResultTable = new
TreeMap<>(Collections.reverseOrder());
+ }
+
+ /**
+* Handle checkpoint exception with a handler callback.
+*
+* @param exception the checkpoint exception.
+*/
+ public void handleCheckpointException(CheckpointException exception,
long checkpointId) {
Review comment:
Hi @StefanRRichter I have refactored the mechanism of counting based on your
suggestion, it considers the checkpoint id's sequence. But when I am
implementing, I meet a problem: the `CheckpointException` caused by **Trigger**
phase may not get the checkpoint id.
Currently, the method `triggerCheckpoint` has two results:
* Gets a pending checkpoint (can get the checkpoint id)
* Throws a `CheckpointException` (whether could get checkpoint id or not
depends on the exception's throw-point in this method)
So, I can not get the checkpoint id
[here](https://github.com/apache/flink/pull/8322/files#diff-a38ea0fa799bdaa0b354d80cd8368c60R442).
My thought is that we could inject the checkpoint id into the
`CheckpointException`(it seems the semantic looks strange?), if we can not
inject it, we can use a default value(-1). Then in `CheckpointFailureManager`,
if we can not get a normal checkpoint (we get `-1` which means the checkpoint
is not been generated in trigger phase), we would ignore this case. Actually,
it seems this case is not the scene which we want to tolerance.
What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #8366: [FLINK-12415][doc-zh]Translate HistoryServer page into Chinese
wuchong commented on a change in pull request #8366:
[FLINK-12415][doc-zh]Translate HistoryServer page into Chinese
URL: https://github.com/apache/flink/pull/8366#discussion_r283088100
##
File path: docs/monitoring/historyserver.zh.md
##
@@ -22,62 +22,62 @@ specific language governing permissions and limitations
under the License.
-->
-Flink has a history server that can be used to query the statistics of
completed jobs after the corresponding Flink cluster has been shut down.
+Flink 自带的 history server 可以在已执行完作业对应的 Flink 集群关闭之后查询该作业的统计信息。
-Furthermore, it exposes a REST API that accepts HTTP requests and responds
with JSON data.
+除此之外，它还提供了一个 REST API，可以通过 HTTP 以 JSON 格式发送和接收数据。
* This will be replaced by the TOC
{:toc}
-## Overview
+## 概览
-The HistoryServer allows you to query the status and statistics of completed
jobs that have been archived by a JobManager.
+History server 可以查询被 JobManager 存档的已执行完作业的状态和统计信息。
-After you have configured the HistoryServer *and* JobManager, you start and
stop the HistoryServer via its corresponding startup script:
+在配置完 History server 和 JobManager 之后，用户可以通过下面的启动脚本来开启和关停 History server：
{% highlight shell %}
-# Start or stop the HistoryServer
+# 开启或关停 History Server
bin/historyserver.sh (start|start-foreground|stop)
{% endhighlight %}
-By default, this server binds to `localhost` and listens at port `8082`.
+默认情况下，History server 绑定到本机 `localhost` 的 `8082` 端口。
-Currently, you can only run it as a standalone process.
+目前你可以把它当做单独的进程来运行。
-## Configuration
+## 配置
-The configuration keys `jobmanager.archive.fs.dir` and
`historyserver.archive.fs.refresh-interval` need to be adjusted for archiving
and displaying archived jobs.
+存档和展示已完成的作业 需要调配 `jobmanager.archive.fs.dir` 和
`historyserver.archive.fs.refresh-interval` 这俩配置项。
Review comment:
```suggestion
如果要存档和展示已完成的作业，需要配置 `jobmanager.archive.fs.dir` 和
`historyserver.archive.fs.refresh-interval` 这俩配置项。
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #8366: [FLINK-12415][doc-zh]Translate HistoryServer page into Chinese
wuchong commented on a change in pull request #8366:
[FLINK-12415][doc-zh]Translate HistoryServer page into Chinese
URL: https://github.com/apache/flink/pull/8366#discussion_r283088187
##
File path: docs/monitoring/historyserver.zh.md
##
@@ -22,62 +22,62 @@ specific language governing permissions and limitations
under the License.
-->
-Flink has a history server that can be used to query the statistics of
completed jobs after the corresponding Flink cluster has been shut down.
+Flink 自带的 history server 可以在已执行完作业对应的 Flink 集群关闭之后查询该作业的统计信息。
-Furthermore, it exposes a REST API that accepts HTTP requests and responds
with JSON data.
+除此之外，它还提供了一个 REST API，可以通过 HTTP 以 JSON 格式发送和接收数据。
* This will be replaced by the TOC
{:toc}
-## Overview
+## 概览
-The HistoryServer allows you to query the status and statistics of completed
jobs that have been archived by a JobManager.
+History server 可以查询被 JobManager 存档的已执行完作业的状态和统计信息。
-After you have configured the HistoryServer *and* JobManager, you start and
stop the HistoryServer via its corresponding startup script:
+在配置完 History server 和 JobManager 之后，用户可以通过下面的启动脚本来开启和关停 History server：
{% highlight shell %}
-# Start or stop the HistoryServer
+# 开启或关停 History Server
bin/historyserver.sh (start|start-foreground|stop)
{% endhighlight %}
-By default, this server binds to `localhost` and listens at port `8082`.
+默认情况下，History server 绑定到本机 `localhost` 的 `8082` 端口。
-Currently, you can only run it as a standalone process.
+目前你可以把它当做单独的进程来运行。
-## Configuration
+## 配置
-The configuration keys `jobmanager.archive.fs.dir` and
`historyserver.archive.fs.refresh-interval` need to be adjusted for archiving
and displaying archived jobs.
+存档和展示已完成的作业 需要调配 `jobmanager.archive.fs.dir` 和
`historyserver.archive.fs.refresh-interval` 这俩配置项。
**JobManager**
-The archiving of completed jobs happens on the JobManager, which uploads the
archived job information to a file system directory. You can configure the
directory to archive completed jobs in `flink-conf.yaml` by setting a directory
via `jobmanager.archive.fs.dir`.
+JobManager 会进行已完成作业的存档，把已存档作业的信息上传到一个文件系统目录上。用户可以通过设置
`jobmanager.archive.fs.dir` 来配置这个存档目录，将 `flink-conf.yaml` 中已完成的作业都存档下来。
{% highlight yaml %}
-# Directory to upload completed job information
+# 已完成作业信息的上传目录
jobmanager.archive.fs.dir: hdfs:///completed-jobs
{% endhighlight %}
**HistoryServer**
-The HistoryServer can be configured to monitor a comma-separated list of
directories in via `historyserver.archive.fs.dir`. The configured directories
are regularly polled for new archives; the polling interval can be configured
via `historyserver.archive.fs.refresh-interval`.
+History Server 可以监控 `historyserver.archive.fs.dir`
配置的用逗号分隔的文件目录列表。并且会为新存档定期轮询已配置的目录，轮询的间隔可以通过
`historyserver.archive.fs.refresh-interval` 来配置。
Review comment:
```suggestion
History Server 可以监控 `historyserver.archive.fs.dir` 配置的用逗号分隔的文件目录列表。History
Server 会定期轮询配置的目录以发现新存档，轮询的间隔可以通过 `historyserver.archive.fs.refresh-interval`
来配置。
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #8366: [FLINK-12415][doc-zh]Translate HistoryServer page into Chinese
wuchong commented on a change in pull request #8366:
[FLINK-12415][doc-zh]Translate HistoryServer page into Chinese
URL: https://github.com/apache/flink/pull/8366#discussion_r283088063
##
File path: docs/monitoring/historyserver.zh.md
##
@@ -22,62 +22,62 @@ specific language governing permissions and limitations
under the License.
-->
-Flink has a history server that can be used to query the statistics of
completed jobs after the corresponding Flink cluster has been shut down.
+Flink 自带的 history server 可以在已执行完作业对应的 Flink 集群关闭之后查询该作业的统计信息。
-Furthermore, it exposes a REST API that accepts HTTP requests and responds
with JSON data.
+除此之外，它还提供了一个 REST API，可以通过 HTTP 以 JSON 格式发送和接收数据。
* This will be replaced by the TOC
{:toc}
-## Overview
+## 概览
-The HistoryServer allows you to query the status and statistics of completed
jobs that have been archived by a JobManager.
+History server 可以查询被 JobManager 存档的已执行完作业的状态和统计信息。
-After you have configured the HistoryServer *and* JobManager, you start and
stop the HistoryServer via its corresponding startup script:
+在配置完 History server 和 JobManager 之后，用户可以通过下面的启动脚本来开启和关停 History server：
{% highlight shell %}
-# Start or stop the HistoryServer
+# 开启或关停 History Server
bin/historyserver.sh (start|start-foreground|stop)
{% endhighlight %}
-By default, this server binds to `localhost` and listens at port `8082`.
+默认情况下，History server 绑定到本机 `localhost` 的 `8082` 端口。
-Currently, you can only run it as a standalone process.
+目前你可以把它当做单独的进程来运行。
Review comment:
```suggestion
目前，只能当做单独的进程来运行。
```
We should translate the `only` word here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #8384: [FLINK-11610][docs-zh] Translate the "Examples" page into Chinese
wuchong commented on a change in pull request #8384: [FLINK-11610][docs-zh]
Translate the "Examples" page into Chinese
URL: https://github.com/apache/flink/pull/8384#discussion_r283087885
##
File path: docs/examples/index.zh.md
##
@@ -26,25 +26,23 @@ under the License.
-->
-## Bundled Examples
+## 附带示例
-The Flink sources include many examples for Flink's different APIs:
+在Flink源文件中，包含了许多Flink不同API接口的代码示例：
-* DataStream applications ({% gh_link
flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples
"Java" %} / {% gh_link
flink-examples/flink-examples-streaming/src/main/scala/org/apache/flink/streaming/scala/examples
"Scala" %})
-* DataSet applications ({% gh_link
flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java
"Java" %} / {% gh_link
flink-examples/flink-examples-batch/src/main/scala/org/apache/flink/examples/scala
"Scala" %})
-* Table API / SQL queries ({% gh_link
flink-examples/flink-examples-table/src/main/java/org/apache/flink/table/examples/java
"Java" %} / {% gh_link
flink-examples/flink-examples-table/src/main/scala/org/apache/flink/table/examples/scala
"Scala" %})
+* DataStream 应用 ({% gh_link
flink-examples/zh/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples
"Java" %} / {% gh_link
flink-examples/zh/flink-examples-streaming/src/main/scala/org/apache/flink/streaming/scala/examples
"Scala" %})
+* DataSet 应用 ({% gh_link
flink-examples/zh/flink-examples-batch/src/main/java/org/apache/flink/examples/java
"Java" %} / {% gh_link
flink-examples/zh/flink-examples-batch/src/main/scala/org/apache/flink/examples/scala
"Scala" %})
+* Table API / SQL 查询 ({% gh_link
flink-examples/zh/flink-examples-table/src/main/java/org/apache/flink/table/examples/java
"Java" %} / {% gh_link
flink-examples/zh/flink-examples-table/src/main/scala/org/apache/flink/table/examples/scala
"Scala" %})
Review comment:
```suggestion
* Table API / SQL 查询 ({% gh_link
flink-examples/flink-examples-table/src/main/java/org/apache/flink/table/examples/java
"Java" %} / {% gh_link
flink-examples/flink-examples-table/src/main/scala/org/apache/flink/table/examples/scala
"Scala" %})
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #8384: [FLINK-11610][docs-zh] Translate the "Examples" page into Chinese
wuchong commented on a change in pull request #8384: [FLINK-11610][docs-zh]
Translate the "Examples" page into Chinese
URL: https://github.com/apache/flink/pull/8384#discussion_r283087920
##
File path: docs/examples/index.zh.md
##
@@ -26,25 +26,23 @@ under the License.
-->
-## Bundled Examples
+## 附带示例
-The Flink sources include many examples for Flink's different APIs:
+在Flink源文件中，包含了许多Flink不同API接口的代码示例：
-* DataStream applications ({% gh_link
flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples
"Java" %} / {% gh_link
flink-examples/flink-examples-streaming/src/main/scala/org/apache/flink/streaming/scala/examples
"Scala" %})
-* DataSet applications ({% gh_link
flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java
"Java" %} / {% gh_link
flink-examples/flink-examples-batch/src/main/scala/org/apache/flink/examples/scala
"Scala" %})
-* Table API / SQL queries ({% gh_link
flink-examples/flink-examples-table/src/main/java/org/apache/flink/table/examples/java
"Java" %} / {% gh_link
flink-examples/flink-examples-table/src/main/scala/org/apache/flink/table/examples/scala
"Scala" %})
+* DataStream 应用 ({% gh_link
flink-examples/zh/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples
"Java" %} / {% gh_link
flink-examples/zh/flink-examples-streaming/src/main/scala/org/apache/flink/streaming/scala/examples
"Scala" %})
Review comment:
The code link doesn't provide Chinese version.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #8384: [FLINK-11610][docs-zh] Translate the "Examples" page into Chinese
wuchong commented on a change in pull request #8384: [FLINK-11610][docs-zh]
Translate the "Examples" page into Chinese
URL: https://github.com/apache/flink/pull/8384#discussion_r283087870
##
File path: docs/examples/index.zh.md
##
@@ -26,25 +26,23 @@ under the License.
-->
-## Bundled Examples
+## 附带示例
-The Flink sources include many examples for Flink's different APIs:
+在Flink源文件中，包含了许多Flink不同API接口的代码示例：
-* DataStream applications ({% gh_link
flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples
"Java" %} / {% gh_link
flink-examples/flink-examples-streaming/src/main/scala/org/apache/flink/streaming/scala/examples
"Scala" %})
-* DataSet applications ({% gh_link
flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java
"Java" %} / {% gh_link
flink-examples/flink-examples-batch/src/main/scala/org/apache/flink/examples/scala
"Scala" %})
-* Table API / SQL queries ({% gh_link
flink-examples/flink-examples-table/src/main/java/org/apache/flink/table/examples/java
"Java" %} / {% gh_link
flink-examples/flink-examples-table/src/main/scala/org/apache/flink/table/examples/scala
"Scala" %})
+* DataStream 应用 ({% gh_link
flink-examples/zh/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples
"Java" %} / {% gh_link
flink-examples/zh/flink-examples-streaming/src/main/scala/org/apache/flink/streaming/scala/examples
"Scala" %})
Review comment:
```suggestion
* DataStream 应用 ({% gh_link
flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples
"Java" %} / {% gh_link
flink-examples/flink-examples-streaming/src/main/scala/org/apache/flink/streaming/scala/examples
"Scala" %})
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wuchong commented on a change in pull request #8384: [FLINK-11610][docs-zh] Translate the "Examples" page into Chinese
wuchong commented on a change in pull request #8384: [FLINK-11610][docs-zh]
Translate the "Examples" page into Chinese
URL: https://github.com/apache/flink/pull/8384#discussion_r283087877
##
File path: docs/examples/index.zh.md
##
@@ -26,25 +26,23 @@ under the License.
-->
-## Bundled Examples
+## 附带示例
-The Flink sources include many examples for Flink's different APIs:
+在Flink源文件中，包含了许多Flink不同API接口的代码示例：
-* DataStream applications ({% gh_link
flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples
"Java" %} / {% gh_link
flink-examples/flink-examples-streaming/src/main/scala/org/apache/flink/streaming/scala/examples
"Scala" %})
-* DataSet applications ({% gh_link
flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java
"Java" %} / {% gh_link
flink-examples/flink-examples-batch/src/main/scala/org/apache/flink/examples/scala
"Scala" %})
-* Table API / SQL queries ({% gh_link
flink-examples/flink-examples-table/src/main/java/org/apache/flink/table/examples/java
"Java" %} / {% gh_link
flink-examples/flink-examples-table/src/main/scala/org/apache/flink/table/examples/scala
"Scala" %})
+* DataStream 应用 ({% gh_link
flink-examples/zh/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples
"Java" %} / {% gh_link
flink-examples/zh/flink-examples-streaming/src/main/scala/org/apache/flink/streaming/scala/examples
"Scala" %})
+* DataSet 应用 ({% gh_link
flink-examples/zh/flink-examples-batch/src/main/java/org/apache/flink/examples/java
"Java" %} / {% gh_link
flink-examples/zh/flink-examples-batch/src/main/scala/org/apache/flink/examples/scala
"Scala" %})
Review comment:
```suggestion
* DataSet 应用 ({% gh_link
flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java
"Java" %} / {% gh_link
flink-examples/flink-examples-batch/src/main/scala/org/apache/flink/examples/scala
"Scala" %})
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] eaglewatcherwb commented on a change in pull request #8309: [FLINK-12229] [runtime] Implement LazyFromSourcesScheduling Strategy
eaglewatcherwb commented on a change in pull request #8309: [FLINK-12229]
[runtime] Implement LazyFromSourcesScheduling Strategy
URL: https://github.com/apache/flink/pull/8309#discussion_r283083182
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/strategy/LazyFromSourcesSchedulingStrategy.java
##
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler.strategy;
+
+import org.apache.flink.api.common.InputDependencyConstraint;
+import org.apache.flink.runtime.execution.ExecutionState;
+import org.apache.flink.runtime.io.network.partition.ResultPartitionID;
+import org.apache.flink.runtime.jobgraph.IntermediateDataSet;
+import org.apache.flink.runtime.jobgraph.IntermediateDataSetID;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.scheduler.DeploymentOption;
+import org.apache.flink.runtime.scheduler.ExecutionVertexDeploymentOption;
+import org.apache.flink.runtime.scheduler.SchedulerOperations;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static
org.apache.flink.runtime.scheduler.strategy.SchedulingResultPartition.ResultPartitionState.DONE;
+import static
org.apache.flink.runtime.scheduler.strategy.SchedulingResultPartition.ResultPartitionState.PRODUCING;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * {@link SchedulingStrategy} instance for batch job which schedule vertices
when input data are ready.
+ */
+public class LazyFromSourcesSchedulingStrategy implements SchedulingStrategy {
+
+ private final SchedulerOperations schedulerOperations;
+
+ private final SchedulingTopology schedulingTopology;
+
+ private final Map
deploymentOptions;
+
+ private final Map
intermediateDataSets;
+
+ public LazyFromSourcesSchedulingStrategy(
+ SchedulerOperations schedulerOperations,
+ SchedulingTopology schedulingTopology) {
+ this.schedulerOperations = checkNotNull(schedulerOperations);
+ this.schedulingTopology = checkNotNull(schedulingTopology);
+ this.intermediateDataSets = new HashMap<>();
+ this.deploymentOptions = new HashMap<>();
+ }
+
+ @Override
+ public void startScheduling() {
+ List
executionVertexDeploymentOptions = new ArrayList<>();
+ DeploymentOption updateOption = new DeploymentOption(true);
+ DeploymentOption nonUpdateOption = new DeploymentOption(false);
+
+ for (SchedulingExecutionVertex schedulingVertex :
schedulingTopology.getVertices()) {
+ DeploymentOption option = nonUpdateOption;
+ for (SchedulingResultPartition srp :
schedulingVertex.getProducedResultPartitions()) {
+ SchedulingIntermediateDataSet sid =
intermediateDataSets.computeIfAbsent(srp.getResultId(),
+ (key) -> new
SchedulingIntermediateDataSet());
+ sid.addSchedulingResultPartition(srp);
+ if (srp.getPartitionType().isPipelined()) {
+ option = updateOption;
+ }
+ }
+ deploymentOptions.put(schedulingVertex.getId(), option);
+
+ if
(schedulingVertex.getConsumedResultPartitions().isEmpty()) {
+ // schedule vertices without consumed result
partition
+ executionVertexDeploymentOptions.add(
+ new
ExecutionVertexDeploymentOption(schedulingVertex.getId(), option));
+ }
+ }
+
+
schedulerOperations.allocateSlotsAndDeploy(executionVertexDeploymentOptions);
+ }
+
+ @Override
+ public void restartTasks(Set verticesToRestart) {
+ // increase counter of the dataset first
+ for (ExecutionVertexID
[GitHub] [flink] eaglewatcherwb commented on a change in pull request #8309: [FLINK-12229] [runtime] Implement LazyFromSourcesScheduling Strategy
eaglewatcherwb commented on a change in pull request #8309: [FLINK-12229]
[runtime] Implement LazyFromSourcesScheduling Strategy
URL: https://github.com/apache/flink/pull/8309#discussion_r283083182
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/strategy/LazyFromSourcesSchedulingStrategy.java
##
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler.strategy;
+
+import org.apache.flink.api.common.InputDependencyConstraint;
+import org.apache.flink.runtime.execution.ExecutionState;
+import org.apache.flink.runtime.io.network.partition.ResultPartitionID;
+import org.apache.flink.runtime.jobgraph.IntermediateDataSet;
+import org.apache.flink.runtime.jobgraph.IntermediateDataSetID;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.scheduler.DeploymentOption;
+import org.apache.flink.runtime.scheduler.ExecutionVertexDeploymentOption;
+import org.apache.flink.runtime.scheduler.SchedulerOperations;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static
org.apache.flink.runtime.scheduler.strategy.SchedulingResultPartition.ResultPartitionState.DONE;
+import static
org.apache.flink.runtime.scheduler.strategy.SchedulingResultPartition.ResultPartitionState.PRODUCING;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * {@link SchedulingStrategy} instance for batch job which schedule vertices
when input data are ready.
+ */
+public class LazyFromSourcesSchedulingStrategy implements SchedulingStrategy {
+
+ private final SchedulerOperations schedulerOperations;
+
+ private final SchedulingTopology schedulingTopology;
+
+ private final Map
deploymentOptions;
+
+ private final Map
intermediateDataSets;
+
+ public LazyFromSourcesSchedulingStrategy(
+ SchedulerOperations schedulerOperations,
+ SchedulingTopology schedulingTopology) {
+ this.schedulerOperations = checkNotNull(schedulerOperations);
+ this.schedulingTopology = checkNotNull(schedulingTopology);
+ this.intermediateDataSets = new HashMap<>();
+ this.deploymentOptions = new HashMap<>();
+ }
+
+ @Override
+ public void startScheduling() {
+ List
executionVertexDeploymentOptions = new ArrayList<>();
+ DeploymentOption updateOption = new DeploymentOption(true);
+ DeploymentOption nonUpdateOption = new DeploymentOption(false);
+
+ for (SchedulingExecutionVertex schedulingVertex :
schedulingTopology.getVertices()) {
+ DeploymentOption option = nonUpdateOption;
+ for (SchedulingResultPartition srp :
schedulingVertex.getProducedResultPartitions()) {
+ SchedulingIntermediateDataSet sid =
intermediateDataSets.computeIfAbsent(srp.getResultId(),
+ (key) -> new
SchedulingIntermediateDataSet());
+ sid.addSchedulingResultPartition(srp);
+ if (srp.getPartitionType().isPipelined()) {
+ option = updateOption;
+ }
+ }
+ deploymentOptions.put(schedulingVertex.getId(), option);
+
+ if
(schedulingVertex.getConsumedResultPartitions().isEmpty()) {
+ // schedule vertices without consumed result
partition
+ executionVertexDeploymentOptions.add(
+ new
ExecutionVertexDeploymentOption(schedulingVertex.getId(), option));
+ }
+ }
+
+
schedulerOperations.allocateSlotsAndDeploy(executionVertexDeploymentOptions);
+ }
+
+ @Override
+ public void restartTasks(Set verticesToRestart) {
+ // increase counter of the dataset first
+ for (ExecutionVertexID
[GitHub] [flink] jiasheng55 commented on a change in pull request #8396: [FLINK-12468][yarn] Unregister application from the YARN Resource Man…
jiasheng55 commented on a change in pull request #8396: [FLINK-12468][yarn]
Unregister application from the YARN Resource Man…
URL: https://github.com/apache/flink/pull/8396#discussion_r283084838
##
File path:
flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnConfigOptions.java
##
@@ -167,6 +167,15 @@
.defaultValue("")
.withDescription("A comma-separated list of tags to apply to
the Flink YARN application.");
+ /**
+* The address of Flink HistoryServer.
+* This configuration parameter allows setting the appTrackingUrl for
finished YARN applications.
+*/
+ public static final ConfigOption
APPLICATION_HISTORY_SERVER_ADDRESS =
Review comment:
@tillrohrmann Any suggestion? Should we stick to
`HistoryServer#HISTORY_SERVER_WEB_ADDRESS` ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (FLINK-12491) Incorrect documentation for directory path separators of CoreOptions.TMP_DIRS
Kezhu Wang created FLINK-12491:
--
Summary: Incorrect documentation for directory path separators of
CoreOptions.TMP_DIRS
Key: FLINK-12491
URL: https://issues.apache.org/jira/browse/FLINK-12491
Project: Flink
Issue Type: Improvement
Components: Runtime / Configuration
Affects Versions: 1.8.0, 1.7.2, 1.6.4, 1.9.0
Reporter: Kezhu Wang
Assignee: Kezhu Wang
{{CoreOptions.TMP_DIRS}} and {{ConfigConstants.TASK_MANAGER_TMP_DIR_KEY}} both
say that:
{quote}
The config parameter defining the directories for temporary files, separated by
* ",", "|", or the system's \{@link java.io.File#pathSeparator}.
{quote}
But the parsing phase uses {{String.split}} with argument {{",|" +
File.pathSeparator}} eventually. However, in fact the sole parameter of
{{String.split}} is a regular expression, so the directory path separators are
"," or {{java.io.File#pathSeparator}}. After digging into history, I found that
the documentation was introduced in commit
{{a7c407ace4f6cbfbde3e247071cee5a755ae66db}} and inherited by
{{76abcaa55d0d6ab704b7ab8164718e8e2dcae2c4}}. So, I think it is safe to drop
"|" from documentation.
{code:title=ConfigurationUtils.java}
public class ConfigurationUtils {
private static String[] splitPaths(@Nonnull String separatedPaths) {
return separatedPaths.length() > 0 ? separatedPaths.split(",|"
+ File.pathSeparator) : EMPTY;
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
