[jira] [Created] (FLINK-13849) The back-pressure monitoring tab in Web UI may cause errors
Xingcan Cui created FLINK-13849:
---
Summary: The back-pressure monitoring tab in Web UI may cause
errors
Key: FLINK-13849
URL: https://issues.apache.org/jira/browse/FLINK-13849
Project: Flink
Issue Type: Bug
Components: Runtime / Web Frontend
Affects Versions: 1.9.0
Reporter: Xingcan Cui
Clicking the back-pressure monitoring tab for a finished job in Web UI will
cause an internal server error. The exceptions are as follows.
{code:java}
2019-08-26 01:23:54,845 ERROR
org.apache.flink.runtime.rest.handler.job.JobVertexBackPressureHandler -
Unhandled exception.
org.apache.flink.runtime.messages.FlinkJobNotFoundException: Could not find
Flink job (09e107685e0b81b443b556062debb443)
at
org.apache.flink.runtime.dispatcher.Dispatcher.getJobMasterGatewayFuture(Dispatcher.java:825)
at
org.apache.flink.runtime.dispatcher.Dispatcher.requestOperatorBackPressureStats(Dispatcher.java:524)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:279)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:194)
at
org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
at akka.actor.ActorCell.invoke(ActorCell.scala:561)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
at akka.dispatch.Mailbox.run(Mailbox.scala:225)
at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #9517: [FLINK-13779][metrics] PrometheusPushGatewayReporter support push metrics with groupingKey
flinkbot edited a comment on issue #9517: [FLINK-13779][metrics] PrometheusPushGatewayReporter support push metrics with groupingKey URL: https://github.com/apache/flink/pull/9517#issuecomment-524187390 ## CI report: * cdbd732fd99f2d2b5ad3cac2d119ef201079aafa : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/124323759) * 19d2487dac6bcdd9308ab9948d610b3b3edf5b8f : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/124325244) * 63209029449bc869bfe957850809564f6682ed56 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/124347410) * 6f7573b875238677c7eab4e1f4d67075f5ad7888 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/124522987) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13836) Improve support of java.util.UUID for JDBCTypeUtil
[
https://issues.apache.org/jira/browse/FLINK-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915465#comment-16915465
]
Jark Wu commented on FLINK-13836:
-
Hi [~flacombe], I don't think UUID is a type, it should be a function. You can
call \{{INSERT INTO jdbc SELECT UUID() FROM T}} to insert a UUID to a JDBC
sink, and the UUID field is a VARCHAR type.
> Improve support of java.util.UUID for JDBCTypeUtil
> --
>
> Key: FLINK-13836
> URL: https://issues.apache.org/jira/browse/FLINK-13836
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / JDBC
>Affects Versions: 1.8.0
>Reporter: François Lacombe
>Priority: Major
>
> Currently, JDBCTypeUtil used by JDBCAppendTableSinkBuilder dones't support
> UUID types with java.util.UUID in Java.
> Could it be possible to handle that as to allow to write UUID directly to
> postgresql please?
>
> java.lang.IllegalArgumentException: Unsupported type:
> GenericType
> at
> org.apache.flink.api.java.io.jdbc.JDBCTypeUtil.typeInformationToSqlType(JDBCTypeUtil.java:91)
> at
> org.apache.flink.api.java.io.jdbc.JDBCAppendTableSinkBuilder.setParameterTypes(JDBCAppendTableSinkBuilder.java:103)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #9299: [FLINK-13405][docs-zh] Translate "Basic API Concepts" page into Chinese
flinkbot edited a comment on issue #9299: [FLINK-13405][docs-zh] Translate "Basic API Concepts" page into Chinese URL: https://github.com/apache/flink/pull/9299#issuecomment-516836559 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit cc0752472d590ec1c6e3c8a11d727e299c0d8ba6 (Mon Aug 26 04:07:47 UTC 2019) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] YngwieWang commented on a change in pull request #9299: [FLINK-13405][docs-zh] Translate "Basic API Concepts" page into Chinese
YngwieWang commented on a change in pull request #9299: [FLINK-13405][docs-zh]
Translate "Basic API Concepts" page into Chinese
URL: https://github.com/apache/flink/pull/9299#discussion_r317444646
##
File path: docs/dev/api_concepts.zh.md
##
@@ -95,31 +73,19 @@ createLocalEnvironment()
createRemoteEnvironment(String host, int port, String... jarFiles)
{% endhighlight %}
-Typically, you only need to use `getExecutionEnvironment()`, since this
-will do the right thing depending on the context: if you are executing
-your program inside an IDE or as a regular Java program it will create
-a local environment that will execute your program on your local machine. If
-you created a JAR file from your program, and invoke it through the
-[command line]({{ site.baseurl }}/ops/cli.html), the Flink cluster manager
-will execute your main method and `getExecutionEnvironment()` will return
-an execution environment for executing your program on a cluster.
+通常你只需要使用 `getExecutionEnvironment()`，因为它会根据上下文环境完成正确的工作，如果你在 IDE 中执行程序或者作为标准的
Java 程序来执行，它会创建你的本机执行环境。如果你将程序封装成 JAR 包，然后通过[命令行]({{ site.baseurl
}}/zh/ops/cli.html)调用，Flink 集群管理器会执行你的 main 方法并且 `getExecutionEnvironment()`
会返回在集群上执行程序的执行环境。
Review comment:
感谢，已修正。
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9299: [FLINK-13405][docs-zh] Translate "Basic API Concepts" page into Chinese
flinkbot edited a comment on issue #9299: [FLINK-13405][docs-zh] Translate "Basic API Concepts" page into Chinese URL: https://github.com/apache/flink/pull/9299#issuecomment-516836559 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit cc0752472d590ec1c6e3c8a11d727e299c0d8ba6 (Mon Aug 26 04:05:45 UTC 2019) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] YngwieWang commented on a change in pull request #9299: [FLINK-13405][docs-zh] Translate "Basic API Concepts" page into Chinese
YngwieWang commented on a change in pull request #9299: [FLINK-13405][docs-zh]
Translate "Basic API Concepts" page into Chinese
URL: https://github.com/apache/flink/pull/9299#discussion_r317444517
##
File path: docs/dev/api_concepts.zh.md
##
@@ -24,68 +24,46 @@ specific language governing permissions and limitations
under the License.
-->
-Flink programs are regular programs that implement transformations on
distributed collections
-(e.g., filtering, mapping, updating state, joining, grouping, defining
windows, aggregating).
-Collections are initially created from sources (e.g., by reading from files,
kafka topics, or from local, in-memory
-collections). Results are returned via sinks, which may for example write the
data to
-(distributed) files, or to standard output (for example, the command line
terminal).
-Flink programs run in a variety of contexts, standalone, or embedded in other
programs.
-The execution can happen in a local JVM, or on clusters of many machines.
-
-Depending on the type of data sources, i.e. bounded or unbounded sources, you
would either
-write a batch program or a streaming program where the DataSet API is used for
batch
-and the DataStream API is used for streaming. This guide will introduce the
basic concepts
-that are common to both APIs but please see our
-[Streaming Guide]({{ site.baseurl }}/dev/datastream_api.html) and
-[Batch Guide]({{ site.baseurl }}/dev/batch/index.html) for concrete
information about
-writing programs with each API.
-
-**NOTE:** When showing actual examples of how the APIs can be used we will use
-`StreamingExecutionEnvironment` and the `DataStream` API. The concepts are
exactly the same
-in the `DataSet` API, just replace by `ExecutionEnvironment` and `DataSet`.
+Flink
程序是实现了分布式集合转换（例如过滤、映射、更新状态、join、分组、定义窗口、聚合）的规范化程序。集合初始创建自数据源（例如读取文件、kafka
主题，或本地内存中的集合）。结果通过 sink 返回，例如，它可以将数据写入（分布式）文件，或标准输出（例如命令行终端）。Flink
程序可以在多种环境中运行，独立运行或嵌入到其他程序中。可以在本地 JVM 中执行，也可以在多台机器的集群上执行。
+
+针对有界和无界两种数据源类型，你可以使用 DataSet API 来编写批处理程序或使用 DataStream API
来编写流处理程序。本篇指南将介绍这两种 API 通用的基本概念，使用每种 API 编写程序的具体信息请查阅
+[流处理指南]({{ site.baseurl }}/zh/dev/datastream_api.html) 和
+[批处理指南]({{ site.baseurl }}/zh/dev/batch/index.html)。
+
+**请注意：** 当展示如何使用 API 的实际示例时我们使用 `StreamingExecutionEnvironment` 和 `DataStream
API`。对于批处理，将他们替换为 `ExecutionEnvironment` 和 `DataSet API` 即可，概念是完全相同的。
* This will be replaced by the TOC
{:toc}
-DataSet and DataStream
+DataSet 和 DataStream
--
-Flink has the special classes `DataSet` and `DataStream` to represent data in
a program. You
-can think of them as immutable collections of data that can contain
duplicates. In the case
-of `DataSet` the data is finite while for a `DataStream` the number of
elements can be unbounded.
+Flink 用特有的 `DataSet` 和 `DataStream` 类来表示程序中的数据。你可以将他们视为包含重复项的不可变数据集合。对于
`DataSet`，数据是有限的，而对于 `DataStream`，元素的数量可以是无限的。
-These collections differ from regular Java collections in some key ways.
First, they
-are immutable, meaning that once they are created you cannot add or remove
elements. You can also
-not simply inspect the elements inside.
+这些集合与标准的 Java 集合有一些关键的区别。首先它们是不可变的，也就是说它们一旦被创建你就不能添加或删除元素了。你也不能简单地检查它们内部的元素。
-A collection is initially created by adding a source in a Flink program and
new collections are
-derived from these by transforming them using API methods such as `map`,
`filter` and so on.
+在 Flink 程序中，集合最初通过添加数据源来创建，通过使用 API 的诸如 `map`、`filter` 等方法对数据源进行转换从而派生新的集合。
Review comment:
感谢，已修正。
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] YngwieWang commented on a change in pull request #9299: [FLINK-13405][docs-zh] Translate "Basic API Concepts" page into Chinese
YngwieWang commented on a change in pull request #9299: [FLINK-13405][docs-zh]
Translate "Basic API Concepts" page into Chinese
URL: https://github.com/apache/flink/pull/9299#discussion_r317443789
##
File path: docs/dev/api_concepts.zh.md
##
@@ -24,68 +24,46 @@ specific language governing permissions and limitations
under the License.
-->
-Flink programs are regular programs that implement transformations on
distributed collections
-(e.g., filtering, mapping, updating state, joining, grouping, defining
windows, aggregating).
-Collections are initially created from sources (e.g., by reading from files,
kafka topics, or from local, in-memory
-collections). Results are returned via sinks, which may for example write the
data to
-(distributed) files, or to standard output (for example, the command line
terminal).
-Flink programs run in a variety of contexts, standalone, or embedded in other
programs.
-The execution can happen in a local JVM, or on clusters of many machines.
-
-Depending on the type of data sources, i.e. bounded or unbounded sources, you
would either
-write a batch program or a streaming program where the DataSet API is used for
batch
-and the DataStream API is used for streaming. This guide will introduce the
basic concepts
-that are common to both APIs but please see our
-[Streaming Guide]({{ site.baseurl }}/dev/datastream_api.html) and
-[Batch Guide]({{ site.baseurl }}/dev/batch/index.html) for concrete
information about
-writing programs with each API.
-
-**NOTE:** When showing actual examples of how the APIs can be used we will use
-`StreamingExecutionEnvironment` and the `DataStream` API. The concepts are
exactly the same
-in the `DataSet` API, just replace by `ExecutionEnvironment` and `DataSet`.
+Flink
程序是实现了分布式集合转换（例如过滤、映射、更新状态、join、分组、定义窗口、聚合）的规范化程序。集合初始创建自数据源（例如读取文件、kafka
主题，或本地内存中的集合）。结果通过 sink 返回，例如，它可以将数据写入（分布式）文件，或标准输出（例如命令行终端）。Flink
程序可以在多种环境中运行，独立运行或嵌入到其他程序中。可以在本地 JVM 中执行，也可以在多台机器的集群上执行。
+
+针对有界和无界两种数据源类型，你可以使用 DataSet API 来编写批处理程序或使用 DataStream API
来编写流处理程序。本篇指南将介绍这两种 API 通用的基本概念，使用每种 API 编写程序的具体信息请查阅
+[流处理指南]({{ site.baseurl }}/zh/dev/datastream_api.html) 和
+[批处理指南]({{ site.baseurl }}/zh/dev/batch/index.html)。
+
+**请注意：** 当展示如何使用 API 的实际示例时我们使用 `StreamingExecutionEnvironment` 和 `DataStream
API`。对于批处理，将他们替换为 `ExecutionEnvironment` 和 `DataSet API` 即可，概念是完全相同的。
* This will be replaced by the TOC
{:toc}
-DataSet and DataStream
+DataSet 和 DataStream
--
-Flink has the special classes `DataSet` and `DataStream` to represent data in
a program. You
-can think of them as immutable collections of data that can contain
duplicates. In the case
-of `DataSet` the data is finite while for a `DataStream` the number of
elements can be unbounded.
+Flink 用特有的 `DataSet` 和 `DataStream` 类来表示程序中的数据。你可以将他们视为包含重复项的不可变数据集合。对于
`DataSet`，数据是有限的，而对于 `DataStream`，元素的数量可以是无限的。
Review comment:
同意，已修正。
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9299: [FLINK-13405][docs-zh] Translate "Basic API Concepts" page into Chinese
flinkbot edited a comment on issue #9299: [FLINK-13405][docs-zh] Translate "Basic API Concepts" page into Chinese URL: https://github.com/apache/flink/pull/9299#issuecomment-516836559 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit cc0752472d590ec1c6e3c8a11d727e299c0d8ba6 (Mon Aug 26 03:59:37 UTC 2019) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] YngwieWang commented on a change in pull request #9299: [FLINK-13405][docs-zh] Translate "Basic API Concepts" page into Chinese
YngwieWang commented on a change in pull request #9299: [FLINK-13405][docs-zh]
Translate "Basic API Concepts" page into Chinese
URL: https://github.com/apache/flink/pull/9299#discussion_r317443607
##
File path: docs/dev/api_concepts.zh.md
##
@@ -24,68 +24,46 @@ specific language governing permissions and limitations
under the License.
-->
-Flink programs are regular programs that implement transformations on
distributed collections
-(e.g., filtering, mapping, updating state, joining, grouping, defining
windows, aggregating).
-Collections are initially created from sources (e.g., by reading from files,
kafka topics, or from local, in-memory
-collections). Results are returned via sinks, which may for example write the
data to
-(distributed) files, or to standard output (for example, the command line
terminal).
-Flink programs run in a variety of contexts, standalone, or embedded in other
programs.
-The execution can happen in a local JVM, or on clusters of many machines.
-
-Depending on the type of data sources, i.e. bounded or unbounded sources, you
would either
-write a batch program or a streaming program where the DataSet API is used for
batch
-and the DataStream API is used for streaming. This guide will introduce the
basic concepts
-that are common to both APIs but please see our
-[Streaming Guide]({{ site.baseurl }}/dev/datastream_api.html) and
-[Batch Guide]({{ site.baseurl }}/dev/batch/index.html) for concrete
information about
-writing programs with each API.
-
-**NOTE:** When showing actual examples of how the APIs can be used we will use
-`StreamingExecutionEnvironment` and the `DataStream` API. The concepts are
exactly the same
-in the `DataSet` API, just replace by `ExecutionEnvironment` and `DataSet`.
+Flink
程序是实现了分布式集合转换（例如过滤、映射、更新状态、join、分组、定义窗口、聚合）的规范化程序。集合初始创建自数据源（例如读取文件、kafka
主题，或本地内存中的集合）。结果通过 sink 返回，例如，它可以将数据写入（分布式）文件，或标准输出（例如命令行终端）。Flink
程序可以在多种环境中运行，独立运行或嵌入到其他程序中。可以在本地 JVM 中执行，也可以在多台机器的集群上执行。
+
+针对有界和无界两种数据源类型，你可以使用 DataSet API 来编写批处理程序或使用 DataStream API
来编写流处理程序。本篇指南将介绍这两种 API 通用的基本概念，使用每种 API 编写程序的具体信息请查阅
Review comment:
官方中文文档[《Apache FLink
是什么》](https://flink.apache.org/zh/flink-architecture.html)
翻译为有界和无界，其实数据都是无限的，差异在于处理的时候是否设置边界，所以我认为有界 / 无界更合适。
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-13839) Support to set yarn node label for flink jobmanager and taskmanager container
[ https://issues.apache.org/jira/browse/FLINK-13839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915455#comment-16915455 ] Yang Wang commented on FLINK-13839: --- Hi [~yanghua], Thanks for your kind reminder. I think this ticket is not a duplicate of FLINK-7836. FLINK-7836 uses ApplicationSubmissionContext#setNodeLabelExpression() to set the node label for application. All the containers of this application will be started in the specified partition. We want to split it into two separate config options so that we could make the flink master container and taskmanager containers running on different partitions. The ResourceRequest#setNodeLabelExpression() will be used. It is very useful for cloud environment. Some nodes are stable and others are cheap but not stable(preemptible instances in alibaba cloud, etc.). If we could start the flink master on the stable nodes, the flink job will be more stable. > Support to set yarn node label for flink jobmanager and taskmanager container > - > > Key: FLINK-13839 > URL: https://issues.apache.org/jira/browse/FLINK-13839 > Project: Flink > Issue Type: New Feature > Components: Deployment / YARN >Reporter: Yang Wang >Priority: Major > > Yarn node label feature is introduced from 2.6. It is a way to group nodes > with similar characteristics and applications can specify where to run. In > the production or cloud environment, we want to the jobmanager running on > some more stable machines. The node label could help us to achieve that. > > However, the ResourceRequest.setNodeLabelExpression have not been supported > in the current hadoop version dependency(2.4.1). So we need to bump the > hadoop version to 2.6.5. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] kaibozhou edited a comment on issue #9517: [FLINK-13779][metrics] PrometheusPushGatewayReporter support push metrics with groupingKey
kaibozhou edited a comment on issue #9517: [FLINK-13779][metrics] PrometheusPushGatewayReporter support push metrics with groupingKey URL: https://github.com/apache/flink/pull/9517#issuecomment-524689138 Hi, @StephanEwen @zentol Thanks for your review. And if users do not specify groupingKey, the reporter will pass null which will be ignored by pushGateway. So it has no impact on the way of using the push gateway without grouping keys. The groupingKey is not just a global tag that applied to all metrics, but is also to specify the groups. From the doc of [pushgateway](https://github.com/prometheus/pushgateway#command-line): > "Pushed metrics are managed in groups, identified by a grouping key of any number of labels, of which the first must be the job label. The groups are easy to inspect via the web interface." It is a special kind of label for pushgateway. The following two pictures are the tests I made.   Thanks Kaibo This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9299: [FLINK-13405][docs-zh] Translate "Basic API Concepts" page into Chinese
flinkbot edited a comment on issue #9299: [FLINK-13405][docs-zh] Translate "Basic API Concepts" page into Chinese URL: https://github.com/apache/flink/pull/9299#issuecomment-516836559 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit cc0752472d590ec1c6e3c8a11d727e299c0d8ba6 (Mon Aug 26 03:46:23 UTC 2019) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] YngwieWang commented on a change in pull request #9299: [FLINK-13405][docs-zh] Translate "Basic API Concepts" page into Chinese
YngwieWang commented on a change in pull request #9299: [FLINK-13405][docs-zh]
Translate "Basic API Concepts" page into Chinese
URL: https://github.com/apache/flink/pull/9299#discussion_r317442089
##
File path: docs/dev/api_concepts.zh.md
##
@@ -24,68 +24,46 @@ specific language governing permissions and limitations
under the License.
-->
-Flink programs are regular programs that implement transformations on
distributed collections
-(e.g., filtering, mapping, updating state, joining, grouping, defining
windows, aggregating).
-Collections are initially created from sources (e.g., by reading from files,
kafka topics, or from local, in-memory
-collections). Results are returned via sinks, which may for example write the
data to
-(distributed) files, or to standard output (for example, the command line
terminal).
-Flink programs run in a variety of contexts, standalone, or embedded in other
programs.
-The execution can happen in a local JVM, or on clusters of many machines.
-
-Depending on the type of data sources, i.e. bounded or unbounded sources, you
would either
-write a batch program or a streaming program where the DataSet API is used for
batch
-and the DataStream API is used for streaming. This guide will introduce the
basic concepts
-that are common to both APIs but please see our
-[Streaming Guide]({{ site.baseurl }}/dev/datastream_api.html) and
-[Batch Guide]({{ site.baseurl }}/dev/batch/index.html) for concrete
information about
-writing programs with each API.
-
-**NOTE:** When showing actual examples of how the APIs can be used we will use
-`StreamingExecutionEnvironment` and the `DataStream` API. The concepts are
exactly the same
-in the `DataSet` API, just replace by `ExecutionEnvironment` and `DataSet`.
+Flink
程序是实现了分布式集合转换（例如过滤、映射、更新状态、join、分组、定义窗口、聚合）的规范化程序。集合初始创建自数据源（例如读取文件、kafka
主题，或本地内存中的集合）。结果通过 sink 返回，例如，它可以将数据写入（分布式）文件，或标准输出（例如命令行终端）。Flink
程序可以在多种环境中运行，独立运行或嵌入到其他程序中。可以在本地 JVM 中执行，也可以在多台机器的集群上执行。
Review comment:
感谢，这次讨论就是我发起的，结论是 Data Sink 不翻译，讨论结束后我复查了本文，本文中的 data source
全部是小写，不是专有名词的意思，所以我翻译为“数据源”，而全部的 sink 都没有翻译，二位的建议是否是全部的 source 无论大小写全部不予翻译呢？
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9517: [FLINK-13779][metrics] PrometheusPushGatewayReporter support push metrics with groupingKey
flinkbot edited a comment on issue #9517: [FLINK-13779][metrics] PrometheusPushGatewayReporter support push metrics with groupingKey URL: https://github.com/apache/flink/pull/9517#issuecomment-524187390 ## CI report: * cdbd732fd99f2d2b5ad3cac2d119ef201079aafa : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/124323759) * 19d2487dac6bcdd9308ab9948d610b3b3edf5b8f : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/124325244) * 63209029449bc869bfe957850809564f6682ed56 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/124347410) * 6f7573b875238677c7eab4e1f4d67075f5ad7888 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/124522987) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9502: [FLINK-13814][hive] HiveTableSink should strip quotes from partition …
flinkbot edited a comment on issue #9502: [FLINK-13814][hive] HiveTableSink should strip quotes from partition … URL: https://github.com/apache/flink/pull/9502#issuecomment-523777539 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 5ad14b3392b439f2b63b9fb079bf5546c874607d (Mon Aug 26 03:43:20 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] danny0405 commented on a change in pull request #9502: [FLINK-13814][hive] HiveTableSink should strip quotes from partition …
danny0405 commented on a change in pull request #9502: [FLINK-13814][hive]
HiveTableSink should strip quotes from partition …
URL: https://github.com/apache/flink/pull/9502#discussion_r317441644
##
File path:
flink-connectors/flink-connector-hive/src/test/java/org/apache/flink/connectors/hive/TableEnvHiveConnectorTest.java
##
@@ -171,6 +172,57 @@ public void testDecimal() throws Exception {
}
}
+ @Test
+ public void testStaticPartition() throws Exception {
+ hiveShell.execute("create database db1");
+ try {
+ hiveShell.execute("create table db1.src (x int)");
+ hiveShell.insertInto("db1",
"src").addRow(1).addRow(2).commit();
+ hiveShell.execute("create table db1.dest (x int)
partitioned by (p1 string, p2 double)");
+ TableEnvironment tableEnv =
getTableEnvWithHiveCatalog();
+ tableEnv.sqlUpdate("insert into db1.dest partition
(p1='1''1', p2=1.1) select x from db1.src");
Review comment:
Agree, a little confusing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] kaibozhou edited a comment on issue #9517: [FLINK-13779][metrics] PrometheusPushGatewayReporter support push metrics with groupingKey
kaibozhou edited a comment on issue #9517: [FLINK-13779][metrics] PrometheusPushGatewayReporter support push metrics with groupingKey URL: https://github.com/apache/flink/pull/9517#issuecomment-524689138 Hi, @StephanEwen @zentol Thanks for your review. And if users do not specify groupingKey, the reporter will pass null which will be ignored by pushGateway. So it has no impact on the way of using the push gateway without grouping keys. The groupingKey is not just a global tag that applied to all metrics, but is also to specify the groups. From the doc of [pushgateway](https://github.com/prometheus/pushgateway#command-line): > "Pushed metrics are managed in groups, identified by a grouping key of any number of labels, of which the first must be the job label. The groups are easy to inspect via the web interface." It is a special kind of label for pushgateway. Thanks Kaibo This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9517: [FLINK-13779][metrics] PrometheusPushGatewayReporter support push metrics with groupingKey
flinkbot edited a comment on issue #9517: [FLINK-13779][metrics] PrometheusPushGatewayReporter support push metrics with groupingKey URL: https://github.com/apache/flink/pull/9517#issuecomment-524187390 ## CI report: * cdbd732fd99f2d2b5ad3cac2d119ef201079aafa : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/124323759) * 19d2487dac6bcdd9308ab9948d610b3b3edf5b8f : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/124325244) * 63209029449bc869bfe957850809564f6682ed56 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/124347410) * 6f7573b875238677c7eab4e1f4d67075f5ad7888 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9517: [FLINK-13779][metrics] PrometheusPushGatewayReporter support push metrics with groupingKey
flinkbot edited a comment on issue #9517: [FLINK-13779][metrics] PrometheusPushGatewayReporter support push metrics with groupingKey URL: https://github.com/apache/flink/pull/9517#issuecomment-524185771 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 6f7573b875238677c7eab4e1f4d67075f5ad7888 (Mon Aug 26 03:26:01 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] kaibozhou commented on issue #9517: [FLINK-13779][metrics] PrometheusPushGatewayReporter support push metrics with groupingKey
kaibozhou commented on issue #9517: [FLINK-13779][metrics] PrometheusPushGatewayReporter support push metrics with groupingKey URL: https://github.com/apache/flink/pull/9517#issuecomment-524701214 All comments addressed, and I add a detailed doc description. Sorry for the misunderstood about the doc, I thought it would be generated automatically -) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-13848) Support “scheduleAtFixedRate/scheduleAtFixedDelay” in RpcEndpoint#MainThreadExecutor
Biao Liu created FLINK-13848:
Summary: Support “scheduleAtFixedRate/scheduleAtFixedDelay” in
RpcEndpoint#MainThreadExecutor
Key: FLINK-13848
URL: https://issues.apache.org/jira/browse/FLINK-13848
Project: Flink
Issue Type: Sub-task
Components: Runtime / Coordination
Reporter: Biao Liu
Fix For: 1.10.0
Currently the methods “scheduleAtFixedRate/scheduleAtFixedDelay" of
{{RpcEndpoint#MainThreadExecutor}} are not implemented. Because there was no
requirement on them before.
Now we are planning to implement these methods to support periodic checkpoint
triggering.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (FLINK-13828) Deprecate ConfigConstants.LOCAL_START_WEBSERVER
[
https://issues.apache.org/jira/browse/FLINK-13828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915434#comment-16915434
]
TisonKun commented on FLINK-13828:
--
Hi [~twalthr] could you assign this issue to me?
> Deprecate ConfigConstants.LOCAL_START_WEBSERVER
> ---
>
> Key: FLINK-13828
> URL: https://issues.apache.org/jira/browse/FLINK-13828
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
>Affects Versions: 1.10.0
>Reporter: TisonKun
>Priority: Major
> Fix For: 1.10.0
>
>
> Maybe backport to 1.9.1 because I heard this issue from a user of 1.9.0
> Exactly {{ConfigConstants.LOCAL_START_WEBSERVER}} has no power any more but
> we don't mark it as deprecated or give an investigation to revive it. Since
> it is part of public interface we cannot remove it between minor versions but
> deprecate it is necessary.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (FLINK-13827) shell variable should be escaped in start-scala-shell.sh
[
https://issues.apache.org/jira/browse/FLINK-13827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915431#comment-16915431
]
TisonKun commented on FLINK-13827:
--
Hi [~Zentol] what do you think? I volunteer to give it a fix.
> shell variable should be escaped in start-scala-shell.sh
>
>
> Key: FLINK-13827
> URL: https://issues.apache.org/jira/browse/FLINK-13827
> Project: Flink
> Issue Type: Bug
> Components: Scala Shell
>Affects Versions: 1.9.0, 1.10.0
>Reporter: TisonKun
>Priority: Major
> Fix For: 1.10.0, 1.9.1
>
>
> {code:java}
> diff --git a/flink-scala-shell/start-script/start-scala-shell.sh
> b/flink-scala-shell/start-script/start-scala-shell.sh
> index b6da81af72..65b9045584 100644
> --- a/flink-scala-shell/start-script/start-scala-shell.sh
> +++ b/flink-scala-shell/start-script/start-scala-shell.sh
> @@ -97,9 +97,9 @@ log_setting="-Dlog.file="$LOG"
> -Dlog4j.configuration=file:"$FLINK_CONF_DIR"/$LOG
>
> if ${EXTERNAL_LIB_FOUND}
> then
> -$JAVA_RUN -Dscala.color -cp "$FLINK_CLASSPATH" $log_setting
> org.apache.flink.api.scala.FlinkShell $@ --addclasspath "$EXT_CLASSPATH"
> +$JAVA_RUN -Dscala.color -cp "$FLINK_CLASSPATH" "$log_setting"
> org.apache.flink.api.scala.FlinkShell $@ --addclasspath "$EXT_CLASSPATH"
> else
> -$JAVA_RUN -Dscala.color -cp "$FLINK_CLASSPATH" $log_setting
> org.apache.flink.api.scala.FlinkShell $@
> +$JAVA_RUN -Dscala.color -cp "$FLINK_CLASSPATH" "$log_setting"
> org.apache.flink.api.scala.FlinkShell $@
> fi
>
> #restore echo
> {code}
> otherwise it is error prone when {{$log_setting}} contain arbitrary content.
> For example, if the parent dir contain whitespace, said {{flink-1.9.0 2}},
> then {{bin/start-scala-shell.sh local}} will fail with
> {{Error: Could not find or load main class
> 2.log.flink\-\*\-scala\-shell\-local\-\*.log}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (FLINK-13698) Rework threading model of CheckpointCoordinator
[
https://issues.apache.org/jira/browse/FLINK-13698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915430#comment-16915430
]
Biao Liu commented on FLINK-13698:
--
Sorry for a bit late response. I was trapped in some other things last week :(
Thanks for detailed reviewing [~pnowojski], [~till.rohrmann]. It helps a lot.
And I really appreciate that.
I will separate this proposal into series of subtasks.
> Rework threading model of CheckpointCoordinator
> ---
>
> Key: FLINK-13698
> URL: https://issues.apache.org/jira/browse/FLINK-13698
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
>Affects Versions: 1.10.0
>Reporter: Piotr Nowojski
>Assignee: Biao Liu
>Priority: Critical
>
> Currently {{CheckpointCoordinator}} and {{CheckpointFailureManager}} code is
> executed by multiple different threads (mostly {{ioExecutor}}, but not only).
> It's causing multiple concurrency issues, for example:
> https://issues.apache.org/jira/browse/FLINK-13497
> Proper fix would be to rethink threading model there. At first glance it
> doesn't seem that this code should be multi threaded, except of parts doing
> the actual IO operations, so it should be possible to run everything in one
> single ExecutionGraph's thread and just run asynchronously necessary IO
> operations with some feedback loop ("mailbox style").
> I would strongly recommend fixing this issue before adding new features in
> the \{{CheckpointCoordinator}} component.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (FLINK-13847) Update release scripts to also update docs/_config.yml
Tzu-Li (Gordon) Tai created FLINK-13847:
---
Summary: Update release scripts to also update docs/_config.yml
Key: FLINK-13847
URL: https://issues.apache.org/jira/browse/FLINK-13847
Project: Flink
Issue Type: Improvement
Components: Documentation, Release System
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
During the 1.9.0 release process, we missed quite a few configuration updates
in {{docs/_config.yml}} related to Flink versions. This should be able to be
done automatically in via the release scripts.
A list of settings in that file that needs to be touched on every major release
include:
* version
* version_title
* github_branch
* baseurl
* stable_baseurl
* javadocs_baseurl
* pythondocs_baseurl
* is_stable
* Add new link to previous_docs
This can probably be done via the {{tools/releasing/create_release_branch.sh}}
script, which is used for every major release.
We should also update the release guide in the project wiki to cover checking
that file as an item in checklists.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #9528: [FLINK-13829] [docs] Fix the base url of release 1.9 docs and mark 1.9 as stable
flinkbot edited a comment on issue #9528: [FLINK-13829] [docs] Fix the base url of release 1.9 docs and mark 1.9 as stable URL: https://github.com/apache/flink/pull/9528#issuecomment-524549337 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 358d971a69be296f264c555a14db8016b0f5b61a (Mon Aug 26 02:37:10 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (FLINK-13829) Incorrect version of 1.9 docs
[ https://issues.apache.org/jira/browse/FLINK-13829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tzu-Li (Gordon) Tai resolved FLINK-13829. - Resolution: Fixed > Incorrect version of 1.9 docs > - > > Key: FLINK-13829 > URL: https://issues.apache.org/jira/browse/FLINK-13829 > Project: Flink > Issue Type: Task > Components: Documentation >Affects Versions: 1.9.0 >Reporter: Paul Lin >Assignee: Zhu Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > The document of Flink 1.9 links to docs of master branch, which should be > 1.9-release branch. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] zhuzhurk closed pull request #9528: [FLINK-13829] [docs] Fix the base url of release 1.9 docs and mark 1.9 as stable
zhuzhurk closed pull request #9528: [FLINK-13829] [docs] Fix the base url of release 1.9 docs and mark 1.9 as stable URL: https://github.com/apache/flink/pull/9528 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhuzhurk commented on issue #9528: [FLINK-13829] [docs] Fix the base url of release 1.9 docs and mark 1.9 as stable
zhuzhurk commented on issue #9528: [FLINK-13829] [docs] Fix the base url of release 1.9 docs and mark 1.9 as stable URL: https://github.com/apache/flink/pull/9528#issuecomment-524693694 merged. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13829) Incorrect version of 1.9 docs
[ https://issues.apache.org/jira/browse/FLINK-13829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915426#comment-16915426 ] Tzu-Li (Gordon) Tai commented on FLINK-13829: - Merged to release-1.9: f2ab7dfe2350e5ee0d69a9c11e0d87a66ebd06b1 > Incorrect version of 1.9 docs > - > > Key: FLINK-13829 > URL: https://issues.apache.org/jira/browse/FLINK-13829 > Project: Flink > Issue Type: Task > Components: Documentation >Affects Versions: 1.9.0 >Reporter: Paul Lin >Assignee: Zhu Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The document of Flink 1.9 links to docs of master branch, which should be > 1.9-release branch. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #9528: [FLINK-13829] [docs] Fix the base url of release 1.9 docs and mark 1.9 as stable
flinkbot edited a comment on issue #9528: [FLINK-13829] [docs] Fix the base url of release 1.9 docs and mark 1.9 as stable URL: https://github.com/apache/flink/pull/9528#issuecomment-524549337 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 358d971a69be296f264c555a14db8016b0f5b61a (Mon Aug 26 02:32:04 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhuzhurk opened a new pull request #9528: [FLINK-13829] [docs] Fix the base url of release 1.9 docs and mark 1.9 as stable
zhuzhurk opened a new pull request #9528: [FLINK-13829] [docs] Fix the base url of release 1.9 docs and mark 1.9 as stable URL: https://github.com/apache/flink/pull/9528 ## What is the purpose of the change *The links in release 1.9 docs to pages of master docs, rather than pages of release-1.9 docs.* *Besides that, release 1.9 is not marked as stable in _config.yml, resulting in that the project templates in quickstart pages be incorrect.* ## Brief change log - *fix baseurl/javadocs_baseurl/pythondocs_baseurl in docs/_config.yml* - *set is_stable to true in docs/_config.yml* ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (FLINK-13829) Incorrect version of 1.9 docs
[ https://issues.apache.org/jira/browse/FLINK-13829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tzu-Li (Gordon) Tai reassigned FLINK-13829: --- Assignee: Zhu Zhu > Incorrect version of 1.9 docs > - > > Key: FLINK-13829 > URL: https://issues.apache.org/jira/browse/FLINK-13829 > Project: Flink > Issue Type: Task > Components: Documentation >Affects Versions: 1.9.0 >Reporter: Paul Lin >Assignee: Zhu Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The document of Flink 1.9 links to docs of master branch, which should be > 1.9-release branch. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (FLINK-13830) The Document about Cluster on yarn have some problems
[ https://issues.apache.org/jira/browse/FLINK-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tzu-Li (Gordon) Tai reassigned FLINK-13830: --- Assignee: Zhu Zhu > The Document about Cluster on yarn have some problems > - > > Key: FLINK-13830 > URL: https://issues.apache.org/jira/browse/FLINK-13830 > Project: Flink > Issue Type: Bug > Components: Documentation >Reporter: zhangmeng >Assignee: Zhu Zhu >Priority: Major > Labels: documentation, wish > Fix For: 1.9.1 > > Attachments: image-2019-08-23-17-48-22-123.png > > > Read the flink 1.9 documentation, YARN Setup section, there have some issues > with download the installation package. There is no > flink-1.10-SNAPSHOT-bin-hadoop2.tgz package in the > https://flink.apache.org/downloads.html download page. > !image-2019-08-23-17-48-22-123.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] zhuzhurk commented on issue #9528: [FLINK-13829] [docs] Fix the base url of release 1.9 docs and mark 1.9 as stable
zhuzhurk commented on issue #9528: [FLINK-13829] [docs] Fix the base url of release 1.9 docs and mark 1.9 as stable URL: https://github.com/apache/flink/pull/9528#issuecomment-524692949 > LGTM! Thanks for the fix, merging this .. Thanks Gordon for reviewing and merging this fix. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhuzhurk closed pull request #9528: [FLINK-13829] [docs] Fix the base url of release 1.9 docs and mark 1.9 as stable
zhuzhurk closed pull request #9528: [FLINK-13829] [docs] Fix the base url of release 1.9 docs and mark 1.9 as stable URL: https://github.com/apache/flink/pull/9528 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13830) The Document about Cluster on yarn have some problems
[ https://issues.apache.org/jira/browse/FLINK-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915424#comment-16915424 ] Tzu-Li (Gordon) Tai commented on FLINK-13830: - Fixed for release-1.9: f2ab7dfe2350e5ee0d69a9c11e0d87a66ebd06b1 > The Document about Cluster on yarn have some problems > - > > Key: FLINK-13830 > URL: https://issues.apache.org/jira/browse/FLINK-13830 > Project: Flink > Issue Type: Bug > Components: Documentation >Reporter: zhangmeng >Priority: Major > Labels: documentation, wish > Attachments: image-2019-08-23-17-48-22-123.png > > > Read the flink 1.9 documentation, YARN Setup section, there have some issues > with download the installation package. There is no > flink-1.10-SNAPSHOT-bin-hadoop2.tgz package in the > https://flink.apache.org/downloads.html download page. > !image-2019-08-23-17-48-22-123.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Closed] (FLINK-13830) The Document about Cluster on yarn have some problems
[ https://issues.apache.org/jira/browse/FLINK-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tzu-Li (Gordon) Tai closed FLINK-13830. --- Fix Version/s: 1.9.1 Resolution: Fixed > The Document about Cluster on yarn have some problems > - > > Key: FLINK-13830 > URL: https://issues.apache.org/jira/browse/FLINK-13830 > Project: Flink > Issue Type: Bug > Components: Documentation >Reporter: zhangmeng >Priority: Major > Labels: documentation, wish > Fix For: 1.9.1 > > Attachments: image-2019-08-23-17-48-22-123.png > > > Read the flink 1.9 documentation, YARN Setup section, there have some issues > with download the installation package. There is no > flink-1.10-SNAPSHOT-bin-hadoop2.tgz package in the > https://flink.apache.org/downloads.html download page. > !image-2019-08-23-17-48-22-123.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #9528: [FLINK-13829] [docs] Fix the base url of release 1.9 docs and mark 1.9 as stable
flinkbot edited a comment on issue #9528: [FLINK-13829] [docs] Fix the base url of release 1.9 docs and mark 1.9 as stable URL: https://github.com/apache/flink/pull/9528#issuecomment-524549337 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 358d971a69be296f264c555a14db8016b0f5b61a (Mon Aug 26 02:22:53 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-13829).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13830) The Document about Cluster on yarn have some problems
[ https://issues.apache.org/jira/browse/FLINK-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915423#comment-16915423 ] Zhu Zhu commented on FLINK-13830: - Actually all the cross page urls incorrectly link to the master docs. A fix is ongoing at [https://github.com/apache/flink/pull/9528|https://github.com/apache/flink/pull/9528/files]. > The Document about Cluster on yarn have some problems > - > > Key: FLINK-13830 > URL: https://issues.apache.org/jira/browse/FLINK-13830 > Project: Flink > Issue Type: Bug > Components: Documentation >Reporter: zhangmeng >Priority: Major > Labels: documentation, wish > Attachments: image-2019-08-23-17-48-22-123.png > > > Read the flink 1.9 documentation, YARN Setup section, there have some issues > with download the installation package. There is no > flink-1.10-SNAPSHOT-bin-hadoop2.tgz package in the > https://flink.apache.org/downloads.html download page. > !image-2019-08-23-17-48-22-123.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] tzulitai commented on issue #9528: [FLINK-13829] [docs] Fix the base url of release 1.9 docs and mark 1.9 as stable
tzulitai commented on issue #9528: [FLINK-13829] [docs] Fix the base url of release 1.9 docs and mark 1.9 as stable URL: https://github.com/apache/flink/pull/9528#issuecomment-524691510 LGTM! Thanks for the fix, merging this .. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9416: [FLINK-13646][build system] Add basic ARM CI job definition
flinkbot edited a comment on issue #9416: [FLINK-13646][build system] Add basic ARM CI job definition URL: https://github.com/apache/flink/pull/9416#issuecomment-520285134 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit df40969ada1c232fb00e6aea33f9a1c88b07855c (Mon Aug 26 02:18:48 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wangxiyuan commented on a change in pull request #9416: [FLINK-13646][build system] Add basic ARM CI job definition
wangxiyuan commented on a change in pull request #9416: [FLINK-13646][build system] Add basic ARM CI job definition URL: https://github.com/apache/flink/pull/9416#discussion_r317431504 ## File path: tools/openlab/build_core.yaml ## @@ -0,0 +1,43 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +- hosts: all + become: yes + roles: +- role: install-openjdk + java_version: '8' + with_scala: true + scala_version: '2.11.0' +- role: install-maven + maven_version: '3.2.5' + +- hosts: all + become: no + tasks: +- name: Build Flink + shell: +cmd: | + set -xe + + mvn clean install -nsu -Dflink.convergence.phase=install -Pcheck-convergence -Dflink.forkCount=2 \ + -Dflink.forkCountTestPackage=2 -Dmaven.javadoc.skip=true -B -DskipTests -Dhadoop.version=2.8.3 \ Review comment: > Thanks @wangxiyuan for your help, except from my comments, would you please share some documentation of the `theopenlab-ci`? Can we include the ARM built with our own account like https://travis-ci.org/ ? Sure, the openlab doc: https://docs.openlabtesting.org/ the CI service which openlab uses: https://zuul-ci.org/docs/zuul/ And zuul uses `ansible-playbook` for job execution tool, that's what this PR is done. > Can we include the ARM built with our own account like https://travis-ci.org/ ? We don't have the account concept in OpenLab, but we have added Flink into OpenLab already. It means that you can do anything you want by editing job related files(The file I created in this PR). For exampel, adding new kinds of jobs, remove old jobs, change job actions and so on. Or do you want any other request or privilege? Please let me know and I'll response to you if OpenLab supports or not. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] docete edited a comment on issue #8738: [FLINK-12845][sql-client] Execute multiple statements in command line…
docete edited a comment on issue #8738: [FLINK-12845][sql-client] Execute multiple statements in command line… URL: https://github.com/apache/flink/pull/8738#issuecomment-524689341 @wuchong @twalthr @JingsongLi IMO the client should have the ability to split input lines into commands( ";" as delimiter). And this should not depends on sql parser. There are several special commands expect SQL statement a SQL client should support. Take SET command(e.g. "SET key1=value1") as example, Should we extend the sql parser to support this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #8738: [FLINK-12845][sql-client] Execute multiple statements in command line…
flinkbot edited a comment on issue #8738: [FLINK-12845][sql-client] Execute multiple statements in command line… URL: https://github.com/apache/flink/pull/8738#issuecomment-502053854 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 88e0dacb1c12dedf338fa360af7a4813d8f634bd (Mon Aug 26 02:07:36 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] docete commented on issue #8738: [FLINK-12845][sql-client] Execute multiple statements in command line…
docete commented on issue #8738: [FLINK-12845][sql-client] Execute multiple statements in command line… URL: https://github.com/apache/flink/pull/8738#issuecomment-524689341 @wuchong @twalthr @JingsongLi IMO the client should have the ability to split input lines into commands( ";" as delimiter). And this should not depends on sql parser. There are several special commands expect SQL statement a SQL client should support. Take SET command(e.g. "SET key1=value1") as example, Should we expend the sql parser to support this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9517: [FLINK-13779][metrics] PrometheusPushGatewayReporter support push metrics with groupingKey
flinkbot edited a comment on issue #9517: [FLINK-13779][metrics] PrometheusPushGatewayReporter support push metrics with groupingKey URL: https://github.com/apache/flink/pull/9517#issuecomment-524185771 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 63209029449bc869bfe957850809564f6682ed56 (Mon Aug 26 02:06:34 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] kaibozhou commented on issue #9517: [FLINK-13779][metrics] PrometheusPushGatewayReporter support push metrics with groupingKey
kaibozhou commented on issue #9517: [FLINK-13779][metrics] PrometheusPushGatewayReporter support push metrics with groupingKey URL: https://github.com/apache/flink/pull/9517#issuecomment-524689138 Hi, @StephanEwen @zentol Thanks for your review. The groupingKey is just a global tag that applied to all metrics. And if users do not specify groupingKey, the reporter will pass null which will be ignored by pushGateway. So it has no impact on the way of using the push gateway without grouping keys. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13830) The Document about Cluster on yarn have some problems
[ https://issues.apache.org/jira/browse/FLINK-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915411#comment-16915411 ] zhangmeng commented on FLINK-13830: --- Is there a problem with the menu on the left? I clicked in the menu on the left side of the 1.9 document and automatically jumped to the master's document. > The Document about Cluster on yarn have some problems > - > > Key: FLINK-13830 > URL: https://issues.apache.org/jira/browse/FLINK-13830 > Project: Flink > Issue Type: Bug > Components: Documentation >Reporter: zhangmeng >Priority: Major > Labels: documentation, wish > Attachments: image-2019-08-23-17-48-22-123.png > > > Read the flink 1.9 documentation, YARN Setup section, there have some issues > with download the installation package. There is no > flink-1.10-SNAPSHOT-bin-hadoop2.tgz package in the > https://flink.apache.org/downloads.html download page. > !image-2019-08-23-17-48-22-123.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (FLINK-13830) The Document about Cluster on yarn have some problems
[ https://issues.apache.org/jira/browse/FLINK-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915411#comment-16915411 ] zhangmeng edited comment on FLINK-13830 at 8/26/19 1:57 AM: Is there have a problem with the menu on the left? I clicked in the menu on the left side of the 1.9 document and automatically jumped to the master's document. was (Author: zhangmeng0426): Is there a problem with the menu on the left? I clicked in the menu on the left side of the 1.9 document and automatically jumped to the master's document. > The Document about Cluster on yarn have some problems > - > > Key: FLINK-13830 > URL: https://issues.apache.org/jira/browse/FLINK-13830 > Project: Flink > Issue Type: Bug > Components: Documentation >Reporter: zhangmeng >Priority: Major > Labels: documentation, wish > Attachments: image-2019-08-23-17-48-22-123.png > > > Read the flink 1.9 documentation, YARN Setup section, there have some issues > with download the installation package. There is no > flink-1.10-SNAPSHOT-bin-hadoop2.tgz package in the > https://flink.apache.org/downloads.html download page. > !image-2019-08-23-17-48-22-123.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #9517: [FLINK-13779][metrics] PrometheusPushGatewayReporter support push metrics with groupingKey
flinkbot edited a comment on issue #9517: [FLINK-13779][metrics] PrometheusPushGatewayReporter support push metrics with groupingKey URL: https://github.com/apache/flink/pull/9517#issuecomment-524185771 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 63209029449bc869bfe957850809564f6682ed56 (Mon Aug 26 01:56:25 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] kaibozhou commented on a change in pull request #9517: [FLINK-13779][metrics] PrometheusPushGatewayReporter support push metrics with groupingKey
kaibozhou commented on a change in pull request #9517: [FLINK-13779][metrics]
PrometheusPushGatewayReporter support push metrics with groupingKey
URL: https://github.com/apache/flink/pull/9517#discussion_r317428987
##
File path:
flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusPushGatewayReporter.java
##
@@ -67,25 +73,44 @@ public void open(MetricConfig config) {
}
pushGateway = new PushGateway(host + ':' + port);
- log.info("Configured PrometheusPushGatewayReporter with
{host:{}, port:{}, jobName: {}, randomJobNameSuffix:{}, deleteOnShutdown:{}}",
host, port, jobName, randomSuffix, deleteOnShutdown);
+ log.info("Configured PrometheusPushGatewayReporter with
{host:{}, port:{}, jobName:{}, randomJobNameSuffix:{}, deleteOnShutdown:{},
groupingKey:{}}",
+ host, port, jobName, randomSuffix, deleteOnShutdown,
groupingKey);
+ }
+
+ @VisibleForTesting
+ static Map parserGroupingKey(final String
groupingKeyConfig) {
+ if (!groupingKeyConfig.isEmpty()) {
+ Map groupingKey = new HashMap<>();
+ String[] kvs = groupingKeyConfig.split(",");
+ for (String kv : kvs) {
+ int idx = kv.indexOf("=");
+ if (idx < 0) {
+ continue;
+ }
+ groupingKey.put(kv.substring(0, idx),
kv.substring(idx + 1));
+ }
+
+ return groupingKey;
+ }
+ return null;
}
@Override
public void report() {
try {
- pushGateway.push(CollectorRegistry.defaultRegistry,
jobName);
+ pushGateway.push(CollectorRegistry.defaultRegistry,
jobName, groupingKey);
Review comment:
I did some test and see the code of pushGateway SDK, both null and an empty
collection will be ignored.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9416: [FLINK-13646][build system] Add basic ARM CI job definition
flinkbot edited a comment on issue #9416: [FLINK-13646][build system] Add basic ARM CI job definition URL: https://github.com/apache/flink/pull/9416#issuecomment-520285134 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit df40969ada1c232fb00e6aea33f9a1c88b07855c (Mon Aug 26 01:51:20 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wangxiyuan commented on a change in pull request #9416: [FLINK-13646][build system] Add basic ARM CI job definition
wangxiyuan commented on a change in pull request #9416: [FLINK-13646][build system] Add basic ARM CI job definition URL: https://github.com/apache/flink/pull/9416#discussion_r317428424 ## File path: tools/openlab/build_core.yaml ## @@ -0,0 +1,43 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +- hosts: all + become: yes + roles: +- role: install-openjdk + java_version: '8' + with_scala: true + scala_version: '2.11.0' +- role: install-maven + maven_version: '3.2.5' + +- hosts: all + become: no + tasks: +- name: Build Flink + shell: +cmd: | + set -xe + + mvn clean install -nsu -Dflink.convergence.phase=install -Pcheck-convergence -Dflink.forkCount=2 \ + -Dflink.forkCountTestPackage=2 -Dmaven.javadoc.skip=true -B -DskipTests -Dhadoop.version=2.8.3 \ Review comment: This job name is not correct. I'll fix it. As I mentioned in the PR commit message. We'll take some steps to add ARM support for Flink. This PR is just the first step: 1. Add the basic build script to ensure the CI system and build job works as expect. The job should be marked as non-voting first, it means the CI test failure won't block Flink PR to be merged. It just includes build part. So that the test is skipped. We'll add the test part back in the next PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #8727: [FLINK-12828][sql-client] Support -f option with a sql script file as…
flinkbot edited a comment on issue #8727: [FLINK-12828][sql-client] Support -f option with a sql script file as… URL: https://github.com/apache/flink/pull/8727#issuecomment-501633487 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 94b5fa9694c53cd89229def77690088e396da8e5 (Mon Aug 26 01:44:13 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] docete commented on issue #8727: [FLINK-12828][sql-client] Support -f option with a sql script file as…
docete commented on issue #8727: [FLINK-12828][sql-client] Support -f option with a sql script file as… URL: https://github.com/apache/flink/pull/8727#issuecomment-524685762 @xuefuz @wuchong multi-statement support is introduced in https://issues.apache.org/jira/browse/FLINK-12845. After that, I will make minor changes to support multi-statement for sql scripts. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] cyq89051127 edited a comment on issue #9529: [FLINK-12979]Allowing set an empty line delimiter at end of an message using CsvRowSerializationSchema
cyq89051127 edited a comment on issue #9529: [FLINK-12979]Allowing set an empty line delimiter at end of an message using CsvRowSerializationSchema URL: https://github.com/apache/flink/pull/9529#issuecomment-524616401 I check the CI report. I can't see the relationship between the failure and my code. @rmetzger What can i do for now? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] cyq89051127 edited a comment on issue #9529: [FLINK-12979]Allowing set an empty line delimiter at end of an message using CsvRowSerializationSchema
cyq89051127 edited a comment on issue #9529: [FLINK-12979]Allowing set an empty line delimiter at end of an message using CsvRowSerializationSchema URL: https://github.com/apache/flink/pull/9529#issuecomment-524616401 I check the CI report. I can't see the relationship between the failure and my code. @rmetzger Could you do me a favor? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9530: [FLINK-13842][docs] Improve Documentation of the StreamingFileSink
flinkbot edited a comment on issue #9530: [FLINK-13842][docs] Improve Documentation of the StreamingFileSink URL: https://github.com/apache/flink/pull/9530#issuecomment-524624121 ## CI report: * f0ac794544411b47703a225715fb50f44d1472e0 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/124492706) * cf4df2f502dd5ec2aed06fc1625969f7f13a5e99 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/124501819) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9531: Merge pull request #1 from apache/master
flinkbot edited a comment on issue #9531: Merge pull request #1 from apache/master URL: https://github.com/apache/flink/pull/9531#issuecomment-524632904 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 1b68496d1632e8cbf87824b08e07ca7a0e319190 (Sun Aug 25 18:16:22 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **Invalid pull request title: No valid Jira ID provided** Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-13824) Code duplication in tools/travis_watchdog.sh to launch watchdog process
[
https://issues.apache.org/jira/browse/FLINK-13824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-13824.
Fix Version/s: 1.10.0
Resolution: Fixed
master: 1eb97e6b860029ea4981ee4318b809a007e01348
> Code duplication in tools/travis_watchdog.sh to launch watchdog process
> ---
>
> Key: FLINK-13824
> URL: https://issues.apache.org/jira/browse/FLINK-13824
> Project: Flink
> Issue Type: Improvement
> Components: Travis
>Reporter: Alex
>Assignee: Alex
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{travis_watchdog.sh}} scripts launches monitoring processes for Java
> processes (spawned by {{mvn install}} and {{mvn verify}}).
>
> The launch logic is copy-pasted twice and prints wrong pid in the second use
> case.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (FLINK-13824) Code duplication in tools/travis_watchdog.sh to launch watchdog process
[
https://issues.apache.org/jira/browse/FLINK-13824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-13824:
---
Labels: pull-request-available (was: )
> Code duplication in tools/travis_watchdog.sh to launch watchdog process
> ---
>
> Key: FLINK-13824
> URL: https://issues.apache.org/jira/browse/FLINK-13824
> Project: Flink
> Issue Type: Bug
> Components: Travis
>Reporter: Alex
>Priority: Minor
> Labels: pull-request-available
>
> {{travis_watchdog.sh}} scripts launches monitoring processes for Java
> processes (spawned by {{mvn install}} and {{mvn verify}}).
>
> The launch logic is copy-pasted twice and prints wrong pid in the second use
> case.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (FLINK-13824) Code duplication in tools/travis_watchdog.sh to launch watchdog process
[
https://issues.apache.org/jira/browse/FLINK-13824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-13824:
-
Issue Type: Improvement (was: Bug)
> Code duplication in tools/travis_watchdog.sh to launch watchdog process
> ---
>
> Key: FLINK-13824
> URL: https://issues.apache.org/jira/browse/FLINK-13824
> Project: Flink
> Issue Type: Improvement
> Components: Travis
>Reporter: Alex
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{travis_watchdog.sh}} scripts launches monitoring processes for Java
> processes (spawned by {{mvn install}} and {{mvn verify}}).
>
> The launch logic is copy-pasted twice and prints wrong pid in the second use
> case.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [flink] zentol closed pull request #9531: Merge pull request #1 from apache/master
zentol closed pull request #9531: Merge pull request #1 from apache/master URL: https://github.com/apache/flink/pull/9531 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (FLINK-13824) Code duplication in tools/travis_watchdog.sh to launch watchdog process
[
https://issues.apache.org/jira/browse/FLINK-13824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler reassigned FLINK-13824:
Assignee: Alex
> Code duplication in tools/travis_watchdog.sh to launch watchdog process
> ---
>
> Key: FLINK-13824
> URL: https://issues.apache.org/jira/browse/FLINK-13824
> Project: Flink
> Issue Type: Improvement
> Components: Travis
>Reporter: Alex
>Assignee: Alex
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{travis_watchdog.sh}} scripts launches monitoring processes for Java
> processes (spawned by {{mvn install}} and {{mvn verify}}).
>
> The launch logic is copy-pasted twice and prints wrong pid in the second use
> case.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [flink] zentol merged pull request #9507: [FLINK-13824][travis] Travis scripts refactoring
zentol merged pull request #9507: [FLINK-13824][travis] Travis scripts refactoring URL: https://github.com/apache/flink/pull/9507 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9530: [FLINK-13842][docs] Improve Documentation of the StreamingFileSink
flinkbot edited a comment on issue #9530: [FLINK-13842][docs] Improve Documentation of the StreamingFileSink URL: https://github.com/apache/flink/pull/9530#issuecomment-524624121 ## CI report: * f0ac794544411b47703a225715fb50f44d1472e0 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/124492706) * cf4df2f502dd5ec2aed06fc1625969f7f13a5e99 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/124501819) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9530: [FLINK-13842][docs] Improve Documentation of the StreamingFileSink
flinkbot edited a comment on issue #9530: [FLINK-13842][docs] Improve Documentation of the StreamingFileSink URL: https://github.com/apache/flink/pull/9530#issuecomment-524624121 ## CI report: * f0ac794544411b47703a225715fb50f44d1472e0 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/124492706) * cf4df2f502dd5ec2aed06fc1625969f7f13a5e99 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9530: [FLINK-13842][docs] Improve Documentation of the StreamingFileSink
flinkbot edited a comment on issue #9530: [FLINK-13842][docs] Improve Documentation of the StreamingFileSink URL: https://github.com/apache/flink/pull/9530#issuecomment-524623653 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit cf4df2f502dd5ec2aed06fc1625969f7f13a5e99 (Sun Aug 25 16:45:49 UTC 2019) **Warnings:** * Documentation files were touched, but no `.zh.md` files: Update Chinese documentation or file Jira ticket. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-13846) Implement benchmark case on MapState#isEmpty
Yun Tang created FLINK-13846:
Summary: Implement benchmark case on MapState#isEmpty
Key: FLINK-13846
URL: https://issues.apache.org/jira/browse/FLINK-13846
Project: Flink
Issue Type: Improvement
Reporter: Yun Tang
If FLINK-13034 merged, we need to implement benchmark case on
{{MapState#isEmpty} in https://github.com/dataArtisans/flink-benchmarks
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (FLINK-13845) Drop all the content of removed "Checkpointed" interface
Yun Tang created FLINK-13845:
Summary: Drop all the content of removed "Checkpointed" interface
Key: FLINK-13845
URL: https://issues.apache.org/jira/browse/FLINK-13845
Project: Flink
Issue Type: Improvement
Components: Documentation
Reporter: Yun Tang
Fix For: 1.10.0
>From [FLINK-7461|https://issues.apache.org/jira/browse/FLINK-7461], we have
>already removed the backward compatibility before Flink-1.1 and the deprecated
>{{Checkpointed}} interface has been totally removed. However, we still have
>many contents including java docs, documentation talked about this
>non-existing interface. I think it's time to remove these contents now.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #9531: Merge pull request #1 from apache/master
flinkbot edited a comment on issue #9531: Merge pull request #1 from apache/master URL: https://github.com/apache/flink/pull/9531#issuecomment-524633609 ## CI report: * 1b68496d1632e8cbf87824b08e07ca7a0e319190 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/124496819) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9416: [FLINK-13646][build system] Add basic ARM CI job definition
flinkbot edited a comment on issue #9416: [FLINK-13646][build system] Add basic ARM CI job definition URL: https://github.com/apache/flink/pull/9416#issuecomment-520285134 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit df40969ada1c232fb00e6aea33f9a1c88b07855c (Sun Aug 25 15:41:43 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] Myasuka commented on a change in pull request #9416: [FLINK-13646][build system] Add basic ARM CI job definition
Myasuka commented on a change in pull request #9416: [FLINK-13646][build system] Add basic ARM CI job definition URL: https://github.com/apache/flink/pull/9416#discussion_r317403756 ## File path: tools/openlab/build_core.yaml ## @@ -0,0 +1,43 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +- hosts: all + become: yes + roles: +- role: install-openjdk + java_version: '8' + with_scala: true + scala_version: '2.11.0' +- role: install-maven + maven_version: '3.2.5' + +- hosts: all + become: no + tasks: +- name: Build Flink + shell: +cmd: | + set -xe + + mvn clean install -nsu -Dflink.convergence.phase=install -Pcheck-convergence -Dflink.forkCount=2 \ + -Dflink.forkCountTestPackage=2 -Dmaven.javadoc.skip=true -B -DskipTests -Dhadoop.version=2.8.3 \ Review comment: Since you have described `Add the test script to run unit/intergration test` in this PR, why tests are all skipped by `-DskipTests`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] Myasuka commented on a change in pull request #9416: [FLINK-13646][build system] Add basic ARM CI job definition
Myasuka commented on a change in pull request #9416: [FLINK-13646][build system] Add basic ARM CI job definition URL: https://github.com/apache/flink/pull/9416#discussion_r317403768 ## File path: tools/openlab/build_test.yaml ## @@ -0,0 +1,39 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +- hosts: all + become: yes + roles: +- role: install-openjdk + java_version: '8' + with_scala: true + scala_version: '2.11.0' +- role: install-maven + maven_version: '3.2.5' + + tasks: +- name: Build Flink + shell: +cmd: | + set -xe + + mvn clean install -nsu -Dflink.convergence.phase=install -Pcheck-convergence -Dflink.forkCount=2 \ + -Dflink.forkCountTestPackage=2 -Dmaven.javadoc.skip=true -B -DskipTests -Dhadoop.version=2.8.3 \ Review comment: The same question as above. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9325: [FLINK-13542][metrics] Update datadog reporter to send metrics if the…
flinkbot edited a comment on issue #9325: [FLINK-13542][metrics] Update datadog reporter to send metrics if the… URL: https://github.com/apache/flink/pull/9325#issuecomment-517479994 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 166fa42707dd620d54756c1f28103f85649f7941 (Sun Aug 25 15:25:30 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] mans2singh closed pull request #9325: [FLINK-13542][metrics] Update datadog reporter to send metrics if the…
mans2singh closed pull request #9325: [FLINK-13542][metrics] Update datadog reporter to send metrics if the… URL: https://github.com/apache/flink/pull/9325 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] mans2singh commented on issue #9325: [FLINK-13542][metrics] Update datadog reporter to send metrics if the…
mans2singh commented on issue #9325: [FLINK-13542][metrics] Update datadog reporter to send metrics if the… URL: https://github.com/apache/flink/pull/9325#issuecomment-524638646 Closing request. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13844) AtOmXpLuS
[ https://issues.apache.org/jira/browse/FLINK-13844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915255#comment-16915255 ] Ken Krugler commented on FLINK-13844: - Hi [~AtOmXpLuS] - please add more details, as currently this looks like a bogus issue. And why is it classified as a major bug? Thanks, Ken > AtOmXpLuS > - > > Key: FLINK-13844 > URL: https://issues.apache.org/jira/browse/FLINK-13844 > Project: Flink > Issue Type: Bug > Components: Connectors / Google Cloud PubSub >Affects Versions: 1.9.0 >Reporter: KORAY DURGUT >Priority: Major > Fix For: shaded-8.0 > > > Everything in AtOmXpLuS.CoM -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #9531: Merge pull request #1 from apache/master
flinkbot edited a comment on issue #9531: Merge pull request #1 from apache/master URL: https://github.com/apache/flink/pull/9531#issuecomment-524633609 ## CI report: * 1b68496d1632e8cbf87824b08e07ca7a0e319190 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/124496819) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9531: Merge pull request #1 from apache/master
flinkbot commented on issue #9531: Merge pull request #1 from apache/master URL: https://github.com/apache/flink/pull/9531#issuecomment-524633609 ## CI report: * 1b68496d1632e8cbf87824b08e07ca7a0e319190 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9531: Merge pull request #1 from apache/master
flinkbot commented on issue #9531: Merge pull request #1 from apache/master URL: https://github.com/apache/flink/pull/9531#issuecomment-524632904 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 1b68496d1632e8cbf87824b08e07ca7a0e319190 (Sun Aug 25 14:02:49 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **Invalid pull request title: No valid Jira ID provided** Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zlzhang0122 commented on issue #9531: Merge pull request #1 from apache/master
zlzhang0122 commented on issue #9531: Merge pull request #1 from apache/master URL: https://github.com/apache/flink/pull/9531#issuecomment-524632788 m This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zlzhang0122 opened a new pull request #9531: Merge pull request #1 from apache/master
zlzhang0122 opened a new pull request #9531: Merge pull request #1 from apache/master URL: https://github.com/apache/flink/pull/9531 merge ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluser with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9530: [FLINK-13842][docs] Improve Documentation of the StreamingFileSink
flinkbot edited a comment on issue #9530: [FLINK-13842][docs] Improve Documentation of the StreamingFileSink URL: https://github.com/apache/flink/pull/9530#issuecomment-524624121 ## CI report: * f0ac794544411b47703a225715fb50f44d1472e0 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/124492706) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (FLINK-13843) Unify and clean up StreamingFileSink format builders
[
https://issues.apache.org/jira/browse/FLINK-13843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915232#comment-16915232
]
Kostas Kloudas edited comment on FLINK-13843 at 8/25/19 12:24 PM:
--
The problem is that the policy must have the same {{BucketID}} as the assigner
and the current set up is the only way to guarantee that. Essentially either
you change both or none (in the case of the row format builder).
The {{.withBucketAssigner()}} provides the option to change the assigner
*without* changing the {{BucketID}}.
was (Author: kkl0u):
The problem is that the policy must have the same {{BucketID}} as the assigner
and the current set up is the only way to guarantee that. Essentially either
you change both or none (in the case of the row format builder).
> Unify and clean up StreamingFileSink format builders
>
>
> Key: FLINK-13843
> URL: https://issues.apache.org/jira/browse/FLINK-13843
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream, Connectors / FileSystem
>Affects Versions: 1.10.0
>Reporter: Gyula Fora
>Priority: Major
>
> I think the StreamingFileSink contains some problems that will affect us in
> the long-run if we intend this sink to be the main exactly-once FS sink.
> *1. Code duplication*
> The StreamingFileSink currently has 2 builders for row and bulk formats:
> RowFormatBuilder, BulkFormatBuilder
> They both contain almost exactly the same config settings with a lot of code
> duplication that should be moved to a common superclass
> (StreamingFileSink.BucketsBuilder).
> *2. Inconsistent config options*
> I also noticed some strange/invalid configuration settings for the builders:
> - RowFormatBuilder#withBucketAssignerAndPolicy : feels like an internal
> method that is not used anywhere. It also overwrites the bucket factory
> - BulkFormatBuilder#withBucketAssigner : takes an extra type parameter
> compared to the row format for the bucket ID type
> - BulkFormatBuilder#withBucketCheckInterval : does not affect behavior as it
> always uses the OnCheckpointRollingPolicy
> This can probably solved by fixing the code duplication
> *3. Fragmented configuration*
> This is not a big problem but only affects the part file config options that
> were introduced recently. We have added 2 methods: withPartFilePrefix and
> withPartFileSuffix
> I think we should aim to group configs that belong together ->
> withPartFileConfig
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (FLINK-13843) Unify and clean up StreamingFileSink format builders
[
https://issues.apache.org/jira/browse/FLINK-13843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915232#comment-16915232
]
Kostas Kloudas commented on FLINK-13843:
The problem is that the policy must have the same {{BucketID}} as the assigner
and the current set up is the only way to guarantee that. Essentially either
you change both or none (in the case of the row format builder).
> Unify and clean up StreamingFileSink format builders
>
>
> Key: FLINK-13843
> URL: https://issues.apache.org/jira/browse/FLINK-13843
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream, Connectors / FileSystem
>Affects Versions: 1.10.0
>Reporter: Gyula Fora
>Priority: Major
>
> I think the StreamingFileSink contains some problems that will affect us in
> the long-run if we intend this sink to be the main exactly-once FS sink.
> *1. Code duplication*
> The StreamingFileSink currently has 2 builders for row and bulk formats:
> RowFormatBuilder, BulkFormatBuilder
> They both contain almost exactly the same config settings with a lot of code
> duplication that should be moved to a common superclass
> (StreamingFileSink.BucketsBuilder).
> *2. Inconsistent config options*
> I also noticed some strange/invalid configuration settings for the builders:
> - RowFormatBuilder#withBucketAssignerAndPolicy : feels like an internal
> method that is not used anywhere. It also overwrites the bucket factory
> - BulkFormatBuilder#withBucketAssigner : takes an extra type parameter
> compared to the row format for the bucket ID type
> - BulkFormatBuilder#withBucketCheckInterval : does not affect behavior as it
> always uses the OnCheckpointRollingPolicy
> This can probably solved by fixing the code duplication
> *3. Fragmented configuration*
> This is not a big problem but only affects the part file config options that
> were introduced recently. We have added 2 methods: withPartFilePrefix and
> withPartFileSuffix
> I think we should aim to group configs that belong together ->
> withPartFileConfig
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (FLINK-13844) AtOmXpLuS
KORAY DURGUT created FLINK-13844: - Summary: AtOmXpLuS Key: FLINK-13844 URL: https://issues.apache.org/jira/browse/FLINK-13844 Project: Flink Issue Type: Bug Components: Connectors / Google Cloud PubSub Affects Versions: 1.9.0 Reporter: KORAY DURGUT Fix For: shaded-8.0 Everything in AtOmXpLuS.CoM -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #9530: [FLINK-13842][docs] Improve Documentation of the StreamingFileSink
flinkbot edited a comment on issue #9530: [FLINK-13842][docs] Improve Documentation of the StreamingFileSink URL: https://github.com/apache/flink/pull/9530#issuecomment-524624121 ## CI report: * f0ac794544411b47703a225715fb50f44d1472e0 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/124492706) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13843) Unify and clean up StreamingFileSink format builders
[ https://issues.apache.org/jira/browse/FLINK-13843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915224#comment-16915224 ] Gyula Fora commented on FLINK-13843: In that case [~kkl0u] the RowFormatBuilder.withBucketAssigner should be changed to take the generic parameter. Just to make it consistent. > Unify and clean up StreamingFileSink format builders > > > Key: FLINK-13843 > URL: https://issues.apache.org/jira/browse/FLINK-13843 > Project: Flink > Issue Type: Improvement > Components: API / DataStream, Connectors / FileSystem >Affects Versions: 1.10.0 >Reporter: Gyula Fora >Priority: Major > > I think the StreamingFileSink contains some problems that will affect us in > the long-run if we intend this sink to be the main exactly-once FS sink. > *1. Code duplication* > The StreamingFileSink currently has 2 builders for row and bulk formats: > RowFormatBuilder, BulkFormatBuilder > They both contain almost exactly the same config settings with a lot of code > duplication that should be moved to a common superclass > (StreamingFileSink.BucketsBuilder). > *2. Inconsistent config options* > I also noticed some strange/invalid configuration settings for the builders: > - RowFormatBuilder#withBucketAssignerAndPolicy : feels like an internal > method that is not used anywhere. It also overwrites the bucket factory > - BulkFormatBuilder#withBucketAssigner : takes an extra type parameter > compared to the row format for the bucket ID type > - BulkFormatBuilder#withBucketCheckInterval : does not affect behavior as it > always uses the OnCheckpointRollingPolicy > This can probably solved by fixing the code duplication > *3. Fragmented configuration* > This is not a big problem but only affects the part file config options that > were introduced recently. We have added 2 methods: withPartFilePrefix and > withPartFileSuffix > I think we should aim to group configs that belong together -> > withPartFileConfig > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] flinkbot commented on issue #9530: [FLINK-13842][docs] Improve Documentation of the StreamingFileSink
flinkbot commented on issue #9530: [FLINK-13842][docs] Improve Documentation of the StreamingFileSink URL: https://github.com/apache/flink/pull/9530#issuecomment-524624121 ## CI report: * f0ac794544411b47703a225715fb50f44d1472e0 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13843) Unify and clean up StreamingFileSink format builders
[
https://issues.apache.org/jira/browse/FLINK-13843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915223#comment-16915223
]
Kostas Kloudas commented on FLINK-13843:
Just a comment for anyone working on this:
1) {{RowFormatBuilder.withBucketAssignerAndPolicy()}} may be renamed but it is
not an internal method as this is the only way to change the default type of
{{BucketID}} to {{ID}} (thus the generics)
2) {{BulkFormatBuilder.withBucketAssigner()}} has the {{ID}} parameter type
for the same reason as above. In essence it is aligned with the
{{RowFormatBuilder.withBucketAssignerAndPolicy()}} with the difference that now
the policy is fixed.
> Unify and clean up StreamingFileSink format builders
>
>
> Key: FLINK-13843
> URL: https://issues.apache.org/jira/browse/FLINK-13843
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream, Connectors / FileSystem
>Affects Versions: 1.10.0
>Reporter: Gyula Fora
>Priority: Major
>
> I think the StreamingFileSink contains some problems that will affect us in
> the long-run if we intend this sink to be the main exactly-once FS sink.
> *1. Code duplication*
> The StreamingFileSink currently has 2 builders for row and bulk formats:
> RowFormatBuilder, BulkFormatBuilder
> They both contain almost exactly the same config settings with a lot of code
> duplication that should be moved to a common superclass
> (StreamingFileSink.BucketsBuilder).
> *2. Inconsistent config options*
> I also noticed some strange/invalid configuration settings for the builders:
> - RowFormatBuilder#withBucketAssignerAndPolicy : feels like an internal
> method that is not used anywhere. It also overwrites the bucket factory
> - BulkFormatBuilder#withBucketAssigner : takes an extra type parameter
> compared to the row format for the bucket ID type
> - BulkFormatBuilder#withBucketCheckInterval : does not affect behavior as it
> always uses the OnCheckpointRollingPolicy
> This can probably solved by fixing the code duplication
> *3. Fragmented configuration*
> This is not a big problem but only affects the part file config options that
> were introduced recently. We have added 2 methods: withPartFilePrefix and
> withPartFileSuffix
> I think we should aim to group configs that belong together ->
> withPartFileConfig
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [flink] flinkbot commented on issue #9530: [FLINK-13842][docs] Improve Documentation of the StreamingFileSink
flinkbot commented on issue #9530: [FLINK-13842][docs] Improve Documentation of the StreamingFileSink URL: https://github.com/apache/flink/pull/9530#issuecomment-524623653 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit f0ac794544411b47703a225715fb50f44d1472e0 (Sun Aug 25 11:47:13 UTC 2019) **Warnings:** * Documentation files were touched, but no `.zh.md` files: Update Chinese documentation or file Jira ticket. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-13842) Improve Javadocs and web documentation of the StreamingFileSink
[ https://issues.apache.org/jira/browse/FLINK-13842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-13842: --- Labels: pull-request-available (was: ) > Improve Javadocs and web documentation of the StreamingFileSink > --- > > Key: FLINK-13842 > URL: https://issues.apache.org/jira/browse/FLINK-13842 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem, Documentation >Reporter: Gyula Fora >Assignee: Gyula Fora >Priority: Major > Labels: pull-request-available > > Both the javadocs and the web docs of the StreamingFileSink and associated > components should be improved for a smooth user experience with more detailed > explanations and examples. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] gyfora opened a new pull request #9530: [FLINK-13842][docs] Improve Documentation of the StreamingFileSink
gyfora opened a new pull request #9530: [FLINK-13842][docs] Improve Documentation of the StreamingFileSink URL: https://github.com/apache/flink/pull/9530 ## What is the purpose of the change Improve Documentation of the StreamingFileSink ## Brief change log - Reworked web docs - Improved javadocs ## Verifying this change Only docs, no runtime changes ## Does this pull request potentially affect one of the following parts: No This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13428) StreamingFileSink allow part file name to be configurable
[ https://issues.apache.org/jira/browse/FLINK-13428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915220#comment-16915220 ] Gyula Fora commented on FLINK-13428: I just checked the implementation and it contains virtually no javadocs or documentation on how to use the feature. Could you please add some docs for it [~eskabetxe] to help the users? :) > StreamingFileSink allow part file name to be configurable > - > > Key: FLINK-13428 > URL: https://issues.apache.org/jira/browse/FLINK-13428 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Reporter: Joao Boto >Assignee: Joao Boto >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Allow that part file name could be configurable: > * partPrefix and partSuffix can be passed > > the part prefix allow to set a better name to file > the part suffix (if used as extension) allow system like Athena or Presto to > automatic detect the type of file and the compression if applied -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (FLINK-13843) Unify and clean up StreamingFileSink format builders
Gyula Fora created FLINK-13843: -- Summary: Unify and clean up StreamingFileSink format builders Key: FLINK-13843 URL: https://issues.apache.org/jira/browse/FLINK-13843 Project: Flink Issue Type: Improvement Components: API / DataStream, Connectors / FileSystem Affects Versions: 1.10.0 Reporter: Gyula Fora I think the StreamingFileSink contains some problems that will affect us in the long-run if we intend this sink to be the main exactly-once FS sink. *1. Code duplication* The StreamingFileSink currently has 2 builders for row and bulk formats: RowFormatBuilder, BulkFormatBuilder They both contain almost exactly the same config settings with a lot of code duplication that should be moved to a common superclass (StreamingFileSink.BucketsBuilder). *2. Inconsistent config options* I also noticed some strange/invalid configuration settings for the builders: - RowFormatBuilder#withBucketAssignerAndPolicy : feels like an internal method that is not used anywhere. It also overwrites the bucket factory - BulkFormatBuilder#withBucketAssigner : takes an extra type parameter compared to the row format for the bucket ID type - BulkFormatBuilder#withBucketCheckInterval : does not affect behavior as it always uses the OnCheckpointRollingPolicy This can probably solved by fixing the code duplication *3. Fragmented configuration* This is not a big problem but only affects the part file config options that were introduced recently. We have added 2 methods: withPartFilePrefix and withPartFileSuffix I think we should aim to group configs that belong together -> withPartFileConfig -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #9383: [FLINK-13248] [runtime] Adding processing of downstream messages in AsyncWaitOperator's wait loops
flinkbot edited a comment on issue #9383: [FLINK-13248] [runtime] Adding processing of downstream messages in AsyncWaitOperator's wait loops URL: https://github.com/apache/flink/pull/9383#issuecomment-519130955 ## CI report: * 5d8448c4813f5b362f98f898998f1278f062d807 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122292142) * 4d628935e8899d6019566bfc93b5c688bc1835ec : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122941321) * d7c0bd5edc65110910d79ca7c7bf2139672f8c02 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/123229382) * b7a19fe5d83ee271e7560f90fbf07a7703937273 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/123235786) * 7650b3b19b05ed6a121566d7c19d5e7bc71489fa : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/123332630) * 2493723ebd2c307f47bbdfcf154a31ab97cda312 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/123335647) * f3f0fe6d16ef3bba35d06a797196f94f372701ff : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/123336279) * c6ee15104ee678c239367670773723920e34c26d : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/123371348) * d0e4fbf25a8ff9982171ed982868b51ad851aaf0 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/123472764) * 741386a495a5657bb654dcd0168f2d42873445e7 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/123476701) * 05e27c097851c65bd9a405b4aae376e2ef6c2b50 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/123645059) * 8fba78f22bc6c0d042cb1dde270c02af08d98bbd : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/123696981) * 681ac331e4c0b547e1d410b448bc34ff651dbc6a : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/123698096) * e18739460fbcf7c59be3c9121fc26fc279e0353a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/123703814) * 670affbc0ae2883e93bb2f6ca3c1300fb78f26c5 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/123726809) * 50a91cbc74c645576432d25d40e7a42190ac28e8 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/123758034) * 2fba64d8e9fa4939c5a6c3fb3d758d55ca344b6c : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/123842040) * 0507aa67d2f7183c3a7e4556fbf7732414647ac7 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/123855539) * 9a293ae331986d8fae16f619158f1e59572cd1e9 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/123856922) * 2bb1bc290a3fc8e4dc843be781063515e86509d8 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/123859379) * 9ff3bc6cdeaff18473a075076ae4931bbdae7173 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/124474961) * bfc5c6d643c502ec3a7ba181d4a80286af08ca50 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/124487989) * 0b54f92f49a56077b36856c4eda400938f2cda75 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/124488476) * 97b0b55c896ce68a8540758a38321ddbfac139c5 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/124489022) * fae34dfb176d977e452dd5a3e7af8838341e565f : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/124489295) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9529: [FLINK-12979]Allowing set an empty line delimiter at end of an message using CsvRowSerializationSchema
flinkbot edited a comment on issue #9529: [FLINK-12979]Allowing set an empty line delimiter at end of an message using CsvRowSerializationSchema URL: https://github.com/apache/flink/pull/9529#issuecomment-524607451 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 88ae31ca30e15c2af5fb6d396673bf07eedfc5c3 (Sun Aug 25 09:51:37 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
