[flink] annotated tag release-1.9.2 updated (c9d2c90 -> 17de1d4)
This is an automated email from the ASF dual-hosted git repository. hequn pushed a change to annotated tag release-1.9.2 in repository https://gitbox.apache.org/repos/asf/flink.git. *** WARNING: tag release-1.9.2 was modified! *** from c9d2c90 (commit) to 17de1d4 (tag) tagging aa45b145928bb2da588367fc886f1fcbfc30f422 (tag) length 991 bytes by hequn8128 on Thu Jan 30 14:53:14 2020 +0800 - Log - Apache Flink release 1.9.2 -BEGIN PGP SIGNATURE- iQIzBAABCAAdFiEE74hHTFZMemCKgi7sP/lqIFe2R2wFAl4yfWkACgkQP/lqIFe2 R2xOvg//cxX8Hk5fdzimNMQaj10vBvE1EhbrcZeu9sclspp+gO56108ElxXdK+dC HVHV9bXNd9tL1OiKrb8zPpFvCuOpV8lp27EHOnYduBqZlpIf7KQlrJdz5dRUN5ao Xrr++Nbh1DYNG/+ThoCnTNCdbuAt+SeJ+wKPqXext5h/BUjKFeRDXbn/pMiF5ZWd rsl8WiWfWDXmsflisWt9xOpQJhkBLGh0i8Mex7WHDgXBES2M13/Xp7Bv0OojfNDE eYViQCrZtagYf8FGYnO4ymjCVHaX3A5NCA2E8uXuL4HMioq1SYkIcnn8GFJiWIn8 MSqrXPGlUys/FwoT7DpZeulySXgTOuparsinCWW3oHxXIe7bOh8gW7rsI+4llpH2 9JvBfYDbLfw9AC49f/LqmPST2h5zQeHqtAuqNavJt98R5uaTETeJMKFUUKp+QQCj 7Kx59nlvpOCFTtKgT5RA01FT7LuIWev8qHQy+uIZ22x4Hbm7/AQQ8SwDFvHKh8oe J0jN+VKeaVRV3nPeywrJwCeyWvlAO+LXFabt2l0RTHZRhqOalrr3iZh2IEqmvLz4 NwPwfZ2RSVjmeOe4Pn/55unNNPojJt7FDQ5vHtLyvtOnZFGrF6eJOM9zpVXk7zKk BvqIioemyq8sKaCKG7a37s0Me4rcUeWhaZZxyg9YCqunVKfzrxQ= =Tm+r -END PGP SIGNATURE- --- No new revisions were added by this update. Summary of changes:
svn commit: r37793 - /release/flink/flink-1.9.1/
Author: jincheng Date: Thu Jan 30 06:36:39 2020 New Revision: 37793 Log: Remove flink-1.9.1 Removed: release/flink/flink-1.9.1/
svn commit: r37792 - /dev/flink/flink-1.9.2-rc1/ /release/flink/flink-1.9.2/
Author: jincheng Date: Thu Jan 30 06:26:44 2020 New Revision: 37792 Log: Release Flink 1.9.2 Added: release/flink/flink-1.9.2/ - copied from r37791, dev/flink/flink-1.9.2-rc1/ Removed: dev/flink/flink-1.9.2-rc1/
[flink-statefun] branch master updated: [hotfix] Add CoC to repository
This is an automated email from the ASF dual-hosted git repository. tzulitai pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink-statefun.git The following commit(s) were added to refs/heads/master by this push: new 87788bd [hotfix] Add CoC to repository 87788bd is described below commit 87788bd900fd9ff01e2a3767520bdec83c6f64cb Author: morsapaes AuthorDate: Wed Jan 29 12:46:25 2020 +0100 [hotfix] Add CoC to repository Adding a Code of Conduct (CoC) to the repository. --- CODE_OF_CONDUCT.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 000..8821b1e --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,3 @@ +# Code of Conduct + +Apache Flink and all its associated repositories follow the [Code of Conduct of the Apache Software Foundation](https://www.apache.org/foundation/policies/conduct).
[flink-web] branch asf-site updated (0c80e86 -> 3a953f8)
This is an automated email from the ASF dual-hosted git repository. fhueske pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/flink-web.git. from 0c80e86 Add Stateful Functions repository to website new fb51490 Add "State Unlocked" blog post new 3a953f8 Rebuild website The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: ...ocked-interacting-with-state-in-apache-flink.md | 253 +++ content/blog/feed.xml | 239 +++ content/blog/index.html| 36 +- content/blog/page10/index.html | 25 ++ content/blog/page2/index.html | 36 +- content/blog/page3/index.html | 38 +- content/blog/page4/index.html | 38 +- content/blog/page5/index.html | 38 +- content/blog/page6/index.html | 40 +- content/blog/page7/index.html | 40 +- content/blog/page8/index.html | 40 +- content/blog/page9/index.html | 40 +- ...state-in-flink-state-processor-api-visual-1.png | Bin 0 -> 52723 bytes ...state-in-flink-state-processor-api-visual-2.png | Bin 0 -> 46207 bytes .../managing-state-in-flink-visual-1.png | Bin 0 -> 419621 bytes .../managing-state-in-flink-visual-2.png | Bin 0 -> 408204 bytes content/index.html | 8 +- ...ked-interacting-with-state-in-apache-flink.html | 470 + content/zh/index.html | 8 +- ...state-in-flink-state-processor-api-visual-1.png | Bin 0 -> 52723 bytes ...state-in-flink-state-processor-api-visual-2.png | Bin 0 -> 46207 bytes .../managing-state-in-flink-visual-1.png | Bin 0 -> 419621 bytes .../managing-state-in-flink-visual-2.png | Bin 0 -> 408204 bytes 23 files changed, 1210 insertions(+), 139 deletions(-) create mode 100755 _posts/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink.md create mode 100755 content/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-1.png create mode 100755 content/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-2.png create mode 100755 content/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-1.png create mode 100755 content/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-2.png create mode 100644 content/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html create mode 100755 img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-1.png create mode 100755 img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-2.png create mode 100755 img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-1.png create mode 100755 img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-2.png
[flink-web] 02/02: Rebuild website
This is an automated email from the ASF dual-hosted git repository. fhueske pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/flink-web.git commit 3a953f8faf1ac137c55cd9a5d195f209f43eeab3 Author: Fabian Hueske AuthorDate: Wed Jan 29 18:34:00 2020 +0100 Rebuild website --- content/blog/feed.xml | 239 +++ content/blog/index.html| 36 +- content/blog/page10/index.html | 25 ++ content/blog/page2/index.html | 36 +- content/blog/page3/index.html | 38 +- content/blog/page4/index.html | 38 +- content/blog/page5/index.html | 38 +- content/blog/page6/index.html | 40 +- content/blog/page7/index.html | 40 +- content/blog/page8/index.html | 40 +- content/blog/page9/index.html | 40 +- ...state-in-flink-state-processor-api-visual-1.png | Bin 0 -> 52723 bytes ...state-in-flink-state-processor-api-visual-2.png | Bin 0 -> 46207 bytes .../managing-state-in-flink-visual-1.png | Bin 0 -> 419621 bytes .../managing-state-in-flink-visual-2.png | Bin 0 -> 408204 bytes content/index.html | 8 +- ...ked-interacting-with-state-in-apache-flink.html | 470 + content/zh/index.html | 8 +- 18 files changed, 957 insertions(+), 139 deletions(-) diff --git a/content/blog/feed.xml b/content/blog/feed.xml index 742368a..9a4424e 100644 --- a/content/blog/feed.xml +++ b/content/blog/feed.xml @@ -7,6 +7,245 @@ https://flink.apache.org/blog/feed.xml; rel="self" type="application/rss+xml" /> +State Unlocked: Interacting with State in Apache Flink +h1 id=introductionIntroduction/h1 + +pWith stateful stream-processing becoming the norm for complex event-driven applications and real-time analytics, a href=https://flink.apache.org/Apache Flink/a is often the backbone for running business logic and managing an organization’s most valuable asset — its data — as application state in Flink./p + +pIn order to provide a state-of-the-art experience to Flink developers, the Apache Flink community makes significant efforts to provide the safety and future-proof guarantees organizations need while managing state in Flink. In particular, Flink developers should have sufficient means to access and modify their state, as well as making bootstrapping state with existing data from external systems a piece-of-cake. These efforts span multiple Flink major releases and consist of the [...] + +ol + liEvolvable state schema in Apache Flink/li + liFlexibility in swapping state backends, and/li + liThe State processor API, an offline tool to read, write and modify state in Flink/li +/ol + +pThis post discusses the community’s efforts related to state management in Flink, provides some practical examples of how the different features and APIs can be utilized and covers some future ideas for new and improved ways of managing state in Apache Flink./p + +h1 id=stream-processing-what-is-stateStream processing: What is State?/h1 + +pTo set the tone for the remaining of the post, let us first try to explain the very definition of state in stream processing. When it comes to stateful stream processing, state comprises of the information that an application or stream processing engine will remember across events and streams as more realtime (unbounded) and/or offline (bounded) data flow through the system. Most trivial applications are inherently stateful; even the example of a simple COUNT operation, whereby [...] + +pTo better understand how Flink manages state, one can think of Flink like a three-layered state abstraction, as illustrated in the diagram below./p + +center +img src=/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-1.png width=600px alt=State in Apache Flink / +/center +pbr //p + +pOn the top layer, sits the Flink user code, for example, a codeKeyedProcessFunction/code that contains some value state. This is a simple variable whose value state annotations makes it automatically fault-tolerant, re-scalable and queryable by the runtime. These variables are backed by the configured state backend that sits either on-heap or on-disk (RocksDB State Backend) and provides data locality, proximity to the computation and speed when it comes to per-re [...] + +pA savepoint is a snapshot of the distributed, global state of an application at a logical point-in-time and is stored in an external distributed file system or blob storage such as HDFS, or S3. Upon upgrading an application or implementing a code change — such as adding a new operator or changing a field — the Flink job can restart by re-loading the application state from the savepoint into the
[flink-web] 01/02: Add "State Unlocked" blog post
This is an automated email from the ASF dual-hosted git repository. fhueske pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/flink-web.git commit fb51490a60d8084b61416607c6d9a8d9c2f91169 Author: Seth Wiesman AuthorDate: Tue Dec 10 12:30:47 2019 -0600 Add "State Unlocked" blog post This closes #288. --- ...ocked-interacting-with-state-in-apache-flink.md | 253 + ...state-in-flink-state-processor-api-visual-1.png | Bin 0 -> 52723 bytes ...state-in-flink-state-processor-api-visual-2.png | Bin 0 -> 46207 bytes .../managing-state-in-flink-visual-1.png | Bin 0 -> 419621 bytes .../managing-state-in-flink-visual-2.png | Bin 0 -> 408204 bytes 5 files changed, 253 insertions(+) diff --git a/_posts/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink.md b/_posts/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink.md new file mode 100755 index 000..ee6bb3c --- /dev/null +++ b/_posts/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink.md @@ -0,0 +1,253 @@ +--- +layout: post +title: "State Unlocked: Interacting with State in Apache Flink" +date: 2020-01-29 12:00:00 +categories: news +authors: +- seth: + name: "Seth Wiesman" + twitter: "sjwiesman" + + +excerpt: This post discusses the efforts of the Flink community as they relate to state management in Apache Flink. We showcase some practical examples of how the different features and APIs can be utilized and cover some future ideas for new and improved ways of managing state in Apache Flink. + +--- + +# Introduction + +With stateful stream-processing becoming the norm for complex event-driven applications and real-time analytics, [Apache Flink](https://flink.apache.org/) is often the backbone for running business logic and managing an organization’s most valuable asset — its data — as application state in Flink. + +In order to provide a state-of-the-art experience to Flink developers, the Apache Flink community makes significant efforts to provide the safety and future-proof guarantees organizations need while managing state in Flink. In particular, Flink developers should have sufficient means to access and modify their state, as well as making bootstrapping state with existing data from external systems a piece-of-cake. These efforts span multiple Flink major releases and consist of the following: + +1. Evolvable state schema in Apache Flink +2. Flexibility in swapping state backends, and +3. The State processor API, an offline tool to read, write and modify state in Flink + +This post discusses the community’s efforts related to state management in Flink, provides some practical examples of how the different features and APIs can be utilized and covers some future ideas for new and improved ways of managing state in Apache Flink. + + +# Stream processing: What is State? + +To set the tone for the remaining of the post, let us first try to explain the very definition of state in stream processing. When it comes to stateful stream processing, state comprises of the information that an application or stream processing engine will remember across events and streams as more realtime (unbounded) and/or offline (bounded) data flow through the system. Most trivial applications are inherently stateful; even the example of a simple COUNT operation, whereby when coun [...] + +To better understand how Flink manages state, one can think of Flink like a three-layered state abstraction, as illustrated in the diagram below. + + + + + + +On the top layer, sits the Flink user code, for example, a `KeyedProcessFunction` that contains some value state. This is a simple variable whose value state annotations makes it automatically fault-tolerant, re-scalable and queryable by the runtime. These variables are backed by the configured state backend that sits either on-heap or on-disk (RocksDB State Backend) and provides data locality, proximity to the computation and speed when it comes to per-record computations. Finally, when [...] + +A savepoint is a snapshot of the distributed, global state of an application at a logical point-in-time and is stored in an external distributed file system or blob storage such as HDFS, or S3. Upon upgrading an application or implementing a code change — such as adding a new operator or changing a field — the Flink job can restart by re-loading the application state from the savepoint into the state backend, making it local and available for the computation and continue processing as i [...] + + + + + + + + It is important to remember here that state is one of the most valuable components of a Flink application carrying all the information about both where you are now and where you are going. State is among the most long-lived components in a Flink service since it can be carried across jobs, operators, configurations, new features and bug fixes. + + +# Schema Evolution
[flink] branch master updated (fe284c8 -> 8699e03)
This is an automated email from the ASF dual-hosted git repository. chesnay pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from fe284c8 [FLINK-15487][table] Allow registering FLIP-65 functions in TableEnvironment add 8699e03 [FLINK-15738][build] Bump powermock to 2.0.4 No new revisions were added by this update. Summary of changes: pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[flink] branch master updated (631722d -> fe284c8)
This is an automated email from the ASF dual-hosted git repository. twalthr pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 631722d [FLINK-15743][docs] Add release notes for Flink 1.10 add fe284c8 [FLINK-15487][table] Allow registering FLIP-65 functions in TableEnvironment No new revisions were added by this update. Summary of changes: .../table/tests/test_environment_completeness.py | 8 +- flink-table/flink-table-api-java/pom.xml | 8 + .../apache/flink/table/api/TableEnvironment.java | 160 ++- .../table/api/internal/TableEnvironmentImpl.java | 74 +++- .../flink/table/catalog/FunctionCatalog.java | 387 + .../apache/flink/table/catalog/FunctionLookup.java | 23 +- .../expressions/resolver/ExpressionResolver.java | 7 +- .../resolver/rules/ResolveCallByArgumentsRule.java | 22 +- .../expressions/resolver/rules/ResolverRule.java | 4 +- .../table/functions/UserDefinedFunctionHelper.java | 206 - .../flink/table/catalog/FunctionCatalogTest.java | 470 + .../flink/table/catalog/UnresolvedIdentifier.java | 10 + .../table/functions/UserDefinedFunctionHelper.java | 360 .../types/extraction/TypeInferenceExtractor.java | 11 +- .../functions/UserDefinedFunctionHelperTest.java | 290 + .../catalog/FunctionCatalogOperatorTable.java | 102 +++-- .../flink/table/planner/delegation/ParserImpl.java | 2 +- .../table/planner/delegation/PlannerContext.java | 1 + .../operations/SqlToOperationConverter.java| 2 +- .../planner/runtime/stream/sql/FunctionITCase.java | 4 +- .../stream/table/TableAggregateITCase.scala| 20 +- .../utils/UserDefinedTableAggFunctions.scala | 5 +- .../catalog/FunctionCatalogOperatorTable.java | 8 +- .../org/apache/flink/table/planner/ParserImpl.java | 2 +- .../table/sqlexec/SqlToOperationConverter.java | 2 +- .../flink/table/api/internal/TableEnvImpl.scala| 65 ++- .../flink/table/catalog/PathResolutionTest.java| 2 +- .../table/runtime/stream/sql/FunctionITCase.java | 4 +- .../flink/table/runtime/batch/sql/CalcITCase.scala | 11 +- .../stream/table/TableAggregateITCase.scala| 24 +- .../flink/table/utils/MockTableEnvironment.scala | 31 +- 31 files changed, 1836 insertions(+), 489 deletions(-) delete mode 100644 flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/functions/UserDefinedFunctionHelper.java create mode 100644 flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/UserDefinedFunctionHelper.java create mode 100644 flink-table/flink-table-common/src/test/java/org/apache/flink/table/functions/UserDefinedFunctionHelperTest.java
[flink] branch master updated (5619307 -> 631722d)
This is an automated email from the ASF dual-hosted git repository. gary pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 5619307 [hotfix][runtime] Add missing space to exception message add 631722d [FLINK-15743][docs] Add release notes for Flink 1.10 No new revisions were added by this update. Summary of changes: docs/index.md | 1 + docs/release-notes/flink-1.10.md| 378 docs/release-notes/flink-1.10.zh.md | 378 3 files changed, 757 insertions(+) create mode 100644 docs/release-notes/flink-1.10.md create mode 100644 docs/release-notes/flink-1.10.zh.md
[flink] branch release-1.10 updated: [hotfix][docs] Fix divergence between flink-1.10.md and flink-1.10.zh.md
This is an automated email from the ASF dual-hosted git repository.
gary pushed a commit to branch release-1.10
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/release-1.10 by this push:
new e4e1bf9 [hotfix][docs] Fix divergence between flink-1.10.md and
flink-1.10.zh.md
e4e1bf9 is described below
commit e4e1bf9de554c7aa9d6cf66bf023155298829d3b
Author: Gary Yao
AuthorDate: Wed Jan 29 13:30:21 2020 +0100
[hotfix][docs] Fix divergence between flink-1.10.md and flink-1.10.zh.md
---
docs/release-notes/flink-1.10.zh.md | 81 +++--
1 file changed, 41 insertions(+), 40 deletions(-)
diff --git a/docs/release-notes/flink-1.10.zh.md
b/docs/release-notes/flink-1.10.zh.md
index 3cfa13b..cec6353 100644
--- a/docs/release-notes/flink-1.10.zh.md
+++ b/docs/release-notes/flink-1.10.zh.md
@@ -31,13 +31,11 @@ these notes carefully if you are planning to upgrade your
Flink version to 1.10.
### Clusters & Deployment
FileSystems should be loaded via Plugin Architecture
([FLINK-11956](https://issues.apache.org/jira/browse/FLINK-11956))
-In the s3-hadoop and s3-presto filesystems, classes from external
-dependencies, such as the AWS SDK, are no longer relocated. In the past, class
-relocation turned out to be problematic in combination with custom
-implementations of the `AWSCredentialsProvider` interface. As a consequence of
-removing class relocation, s3-hadoop and s3-presto filesystems can only be
-used as [plugins]({{ site.baseurl }}/ops/filesystems/#pluggable-file-systems).
-Other filesystems are strongly recommended to be only used as plugins.
+s3-hadoop and s3-presto filesystems do no longer use class relocations and need
+to be loaded through [plugins]({{ site.baseurl
}}/ops/filesystems/#pluggable-file-systems)
+but now seamlessly integrate with all credential providers. Other filesystems
+are strongly recommended to be only used as plugins as we will continue to
+remove relocations.
Flink Client respects Classloading Policy
([FLINK-13749](https://issues.apache.org/jira/browse/FLINK-13749))
The Flink client now also respects the configured classloading policy, i.e.,
@@ -48,14 +46,14 @@ which case they should configure the classloading policy
explicitly to use
`parent-first` classloading, which was the previous (hard-coded) behaviour.
Enable spreading out Tasks evenly across all TaskManagers
([FLINK-12122](https://issues.apache.org/jira/browse/FLINK-12122))
-When
[FLIP-6](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077)
-was rolled out with Flink 1.5.0, we changed how slots are allocated
-from TaskManagers (TMs). Instead of evenly allocating the slots from all
-registered TMs, we had the tendency to exhaust a TM before using another one.
-To use a scheduling strategy that is more similar to the pre-FLIP-6
-behaviour, where Flink tries to spread out the workload across all available
-TMs, one can set `cluster.evenly-spread-out-slots: true` in the
-`flink-conf.yaml`.
+When
+[FLIP-6](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077)
+was rolled out with Flink 1.5.0, we changed how slots are allocated from
+TaskManagers (TMs). Instead of evenly allocating the slots from all registered
+TMs, we had the tendency to exhaust a TM before using another one. To use a
+scheduling strategy that is more similar to the pre-FLIP-6 behaviour, where
+Flink tries to spread out the workload across all currently available TMs, one
+can set `cluster.evenly-spread-out-slots: true` in the `flink-conf.yaml`.
Directory Structure Change for highly available Artifacts
([FLINK-13633](https://issues.apache.org/jira/browse/FLINK-13633))
All highly available artifacts stored by Flink will now be stored under
@@ -102,8 +100,6 @@ If you try to reuse your previous Flink configuration
without any adjustments,
the new memory model can result in differently computed memory parameters for
the JVM and, thus, performance changes.
-Please check the user documentation for more
details.
-
# Deprecation and breaking changes
The following options have been removed and have no effect anymore:
@@ -126,7 +122,7 @@ The following options have been removed and have no effect
anymore:
taskmanager.memory.off-heap
- On-heap managed memory is no longer supported
+ Support for on-heap managed memory has been removed, leaving
off-heap managed memory as the only possibility
taskmanager.memory.preallocate
@@ -180,13 +176,18 @@ The container cut-off configuration options,
`containerized.heap-cutoff-ratio`
and `containerized.heap-cutoff-min`, have no effect for task executor processes
anymore but they still have the same semantics for the JobManager process.
- Fine Grained Operator Resource Management
([FLINK-14058](https://issues.apache.org/jira/browse/FLINK-14058))
-
+ Fine-grained Operator
[flink] branch release-1.10 updated (14a29a4 -> 5b8a3c2)
This is an automated email from the ASF dual-hosted git repository. gary pushed a change to branch release-1.10 in repository https://gitbox.apache.org/repos/asf/flink.git. from 14a29a4 [FLINK-15768] Consolidate executor-related classes in flink-client add 5b8a3c2 [FLINK-15743][docs] Add release notes for Flink 1.10 No new revisions were added by this update. Summary of changes: docs/index.md | 1 + docs/release-notes/flink-1.10.md| 378 docs/release-notes/flink-1.10.zh.md | 377 +++ 3 files changed, 756 insertions(+) create mode 100644 docs/release-notes/flink-1.10.md create mode 100644 docs/release-notes/flink-1.10.zh.md
[flink] branch master updated (236710c -> 5619307)
This is an automated email from the ASF dual-hosted git repository. chesnay pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 236710c [hotfix][javadocs] Fix OutputTag example add 5619307 [hotfix][runtime] Add missing space to exception message No new revisions were added by this update. Summary of changes: .../org/apache/flink/runtime/operators/shipping/OutputCollector.java| 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[flink] branch master updated (fcd2834 -> 4aae548)
This is an automated email from the ASF dual-hosted git repository. gary pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from fcd2834 [FLINK-15768] Consolidate executor-related classes in flink-client add 4aae548 [FLINK-15628][rest] Create webSumissionHandlers List with default capacity No new revisions were added by this update. Summary of changes: .../org/apache/flink/runtime/webmonitor/WebSubmissionExtension.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[flink] branch release-1.10 updated: [FLINK-15768] Consolidate executor-related classes in flink-client
This is an automated email from the ASF dual-hosted git repository.
kkloudas pushed a commit to branch release-1.10
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/release-1.10 by this push:
new 14a29a4 [FLINK-15768] Consolidate executor-related classes in
flink-client
14a29a4 is described below
commit 14a29a4621d8181c91855e5546dbeace1f70bc67
Author: hpeter
AuthorDate: Fri Jan 24 15:31:24 2020 -0800
[FLINK-15768] Consolidate executor-related classes in flink-client
---
.../deployment/{ => executors}/AbstractJobClusterExecutor.java | 6 +-
.../deployment/{ => executors}/AbstractSessionClusterExecutor.java | 5 -
.../flink/client/deployment/{ => executors}/ExecutorUtils.java | 2 +-
.../apache/flink/client/deployment/executors/RemoteExecutor.java| 1 -
.../kubernetes/executors/KubernetesSessionClusterExecutor.java | 2 +-
.../org/apache/flink/yarn/executors/YarnJobClusterExecutor.java | 2 +-
.../org/apache/flink/yarn/executors/YarnSessionClusterExecutor.java | 2 +-
7 files changed, 13 insertions(+), 7 deletions(-)
diff --git
a/flink-clients/src/main/java/org/apache/flink/client/deployment/AbstractJobClusterExecutor.java
b/flink-clients/src/main/java/org/apache/flink/client/deployment/executors/AbstractJobClusterExecutor.java
similarity index 90%
rename from
flink-clients/src/main/java/org/apache/flink/client/deployment/AbstractJobClusterExecutor.java
rename to
flink-clients/src/main/java/org/apache/flink/client/deployment/executors/AbstractJobClusterExecutor.java
index 16ea3e4..d8ccfc4 100644
---
a/flink-clients/src/main/java/org/apache/flink/client/deployment/AbstractJobClusterExecutor.java
+++
b/flink-clients/src/main/java/org/apache/flink/client/deployment/executors/AbstractJobClusterExecutor.java
@@ -16,11 +16,15 @@
* limitations under the License.
*/
-package org.apache.flink.client.deployment;
+package org.apache.flink.client.deployment.executors;
import org.apache.flink.annotation.Internal;
import org.apache.flink.api.dag.Pipeline;
import org.apache.flink.client.cli.ExecutionConfigAccessor;
+import org.apache.flink.client.deployment.ClusterClientFactory;
+import org.apache.flink.client.deployment.ClusterClientJobClientAdapter;
+import org.apache.flink.client.deployment.ClusterDescriptor;
+import org.apache.flink.client.deployment.ClusterSpecification;
import org.apache.flink.client.program.ClusterClientProvider;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.core.execution.JobClient;
diff --git
a/flink-clients/src/main/java/org/apache/flink/client/deployment/AbstractSessionClusterExecutor.java
b/flink-clients/src/main/java/org/apache/flink/client/deployment/executors/AbstractSessionClusterExecutor.java
similarity index 92%
rename from
flink-clients/src/main/java/org/apache/flink/client/deployment/AbstractSessionClusterExecutor.java
rename to
flink-clients/src/main/java/org/apache/flink/client/deployment/executors/AbstractSessionClusterExecutor.java
index bff56e4..cd07f92 100644
---
a/flink-clients/src/main/java/org/apache/flink/client/deployment/AbstractSessionClusterExecutor.java
+++
b/flink-clients/src/main/java/org/apache/flink/client/deployment/executors/AbstractSessionClusterExecutor.java
@@ -16,10 +16,13 @@
* limitations under the License.
*/
-package org.apache.flink.client.deployment;
+package org.apache.flink.client.deployment.executors;
import org.apache.flink.annotation.Internal;
import org.apache.flink.api.dag.Pipeline;
+import org.apache.flink.client.deployment.ClusterClientFactory;
+import org.apache.flink.client.deployment.ClusterClientJobClientAdapter;
+import org.apache.flink.client.deployment.ClusterDescriptor;
import org.apache.flink.client.program.ClusterClient;
import org.apache.flink.client.program.ClusterClientProvider;
import org.apache.flink.configuration.Configuration;
diff --git
a/flink-clients/src/main/java/org/apache/flink/client/deployment/ExecutorUtils.java
b/flink-clients/src/main/java/org/apache/flink/client/deployment/executors/ExecutorUtils.java
similarity index 97%
rename from
flink-clients/src/main/java/org/apache/flink/client/deployment/ExecutorUtils.java
rename to
flink-clients/src/main/java/org/apache/flink/client/deployment/executors/ExecutorUtils.java
index 4541b1c..4943d46 100644
---
a/flink-clients/src/main/java/org/apache/flink/client/deployment/ExecutorUtils.java
+++
b/flink-clients/src/main/java/org/apache/flink/client/deployment/executors/ExecutorUtils.java
@@ -16,7 +16,7 @@
* limitations under the License.
*/
-package org.apache.flink.client.deployment;
+package org.apache.flink.client.deployment.executors;
import org.apache.flink.api.dag.Pipeline;
import org.apache.flink.client.FlinkPipelineTranslationUtil;
diff --git
a/flink-clients/src/main/java/org/apache/flink/client/deployment/executors/RemoteExecutor.java
[flink] branch master updated (23ff230 -> fcd2834)
This is an automated email from the ASF dual-hosted git repository.
kkloudas pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git.
from 23ff230 [FLINK-15651][tests] Refactor JarHandlerTest
add fcd2834 [FLINK-15768] Consolidate executor-related classes in
flink-client
No new revisions were added by this update.
Summary of changes:
.../deployment/{ => executors}/AbstractJobClusterExecutor.java | 6 +-
.../deployment/{ => executors}/AbstractSessionClusterExecutor.java | 5 -
.../flink/client/deployment/{ => executors}/ExecutorUtils.java | 2 +-
.../apache/flink/client/deployment/executors/RemoteExecutor.java| 1 -
.../kubernetes/executors/KubernetesSessionClusterExecutor.java | 2 +-
.../org/apache/flink/yarn/executors/YarnJobClusterExecutor.java | 2 +-
.../org/apache/flink/yarn/executors/YarnSessionClusterExecutor.java | 2 +-
7 files changed, 13 insertions(+), 7 deletions(-)
rename flink-clients/src/main/java/org/apache/flink/client/deployment/{ =>
executors}/AbstractJobClusterExecutor.java (90%)
rename flink-clients/src/main/java/org/apache/flink/client/deployment/{ =>
executors}/AbstractSessionClusterExecutor.java (92%)
rename flink-clients/src/main/java/org/apache/flink/client/deployment/{ =>
executors}/ExecutorUtils.java (97%)
[flink] 02/02: [FLINK-15651][tests] Refactor JarHandlerTest
This is an automated email from the ASF dual-hosted git repository.
chesnay pushed a commit to branch release-1.10
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 0709bf1039078867c9165806a70be7e8429cd2c1
Author: Chesnay Schepler
AuthorDate: Thu Jan 23 15:33:25 2020 +0100
[FLINK-15651][tests] Refactor JarHandlerTest
---
flink-runtime-web/pom.xml | 19 +++
.../webmonitor/handlers/JarHandlerTest.java| 133 +
.../runtime/webmonitor/handlers/JarHandlers.java | 2 +-
.../handlers/utils/OutputTestProgram.java | 29 +
4 files changed, 82 insertions(+), 101 deletions(-)
diff --git a/flink-runtime-web/pom.xml b/flink-runtime-web/pom.xml
index b7fadbe..f152abd 100644
--- a/flink-runtime-web/pom.xml
+++ b/flink-runtime-web/pom.xml
@@ -214,6 +214,25 @@ under the License.
${test.ParameterProgramNoManifest.name}
+
+
+ test-output-program-jar
+
process-test-classes
+
+ jar
+
+
+
+
org/apache/flink/runtime/webmonitor/handlers/utils/OutputTestProgram.java
+
+
+
+
org.apache.flink.runtime.webmonitor.handlers.utils.OutputTestProgram
+
+
+
output-test-program
+
+
diff --git
a/flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/handlers/JarHandlerTest.java
b/flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/handlers/JarHandlerTest.java
index c565322..33ada39 100644
---
a/flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/handlers/JarHandlerTest.java
+++
b/flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/handlers/JarHandlerTest.java
@@ -18,33 +18,21 @@
package org.apache.flink.runtime.webmonitor.handlers;
-import org.apache.flink.api.common.time.Time;
-import org.apache.flink.configuration.Configuration;
-import org.apache.flink.configuration.RestOptions;
-import org.apache.flink.configuration.WebOptions;
-import org.apache.flink.runtime.rest.RestClient;
-import org.apache.flink.runtime.rest.RestClientConfiguration;
-import org.apache.flink.runtime.rest.messages.MessageHeaders;
-import org.apache.flink.runtime.rest.util.RestClientException;
-import org.apache.flink.runtime.testingUtils.TestingUtils;
-import org.apache.flink.runtime.testutils.MiniClusterResource;
-import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.client.program.ProgramInvocationException;
+import org.apache.flink.runtime.webmonitor.TestingDispatcherGateway;
import org.apache.flink.testutils.junit.category.AlsoRunWithLegacyScheduler;
import org.apache.flink.util.ExceptionUtils;
import org.apache.flink.util.TestLogger;
+import org.junit.Assert;
import org.junit.ClassRule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
import org.junit.rules.TemporaryFolder;
-import java.io.IOException;
-import java.net.URI;
-import java.nio.file.FileSystem;
-import java.nio.file.FileSystems;
import java.nio.file.Files;
import java.nio.file.Path;
-import java.util.HashMap;
+import java.nio.file.Paths;
import java.util.Optional;
import static org.hamcrest.CoreMatchers.containsString;
@@ -56,6 +44,8 @@ import static org.hamcrest.MatcherAssert.assertThat;
@Category(AlsoRunWithLegacyScheduler.class)
public class JarHandlerTest extends TestLogger {
+ private static final String JAR_NAME = "output-test-program.jar";
+
@ClassRule
public static final TemporaryFolder TMP = new TemporaryFolder();
@@ -75,96 +65,39 @@ public class JarHandlerTest extends TestLogger {
}
private static void runTest(Type type, String expectedCapturedStdOut,
String expectedCapturedStdErr) throws Exception {
- Path uploadDir = TMP.newFolder().toPath();
-
- Path actualUploadDir =
[flink] 01/02: [hotfix][tests] Make JarHandlers re-usable
This is an automated email from the ASF dual-hosted git repository.
chesnay pushed a commit to branch release-1.10
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 69028bdbba733c0a3308a303b864285453b5e31e
Author: Chesnay Schepler
AuthorDate: Thu Jan 23 15:33:06 2020 +0100
[hotfix][tests] Make JarHandlers re-usable
---
.../runtime/webmonitor/handlers/JarHandlers.java | 155 +
.../webmonitor/handlers/JarSubmissionITCase.java | 132 +-
2 files changed, 160 insertions(+), 127 deletions(-)
diff --git
a/flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/handlers/JarHandlers.java
b/flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/handlers/JarHandlers.java
new file mode 100644
index 000..f5405af
--- /dev/null
+++
b/flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/handlers/JarHandlers.java
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.webmonitor.handlers;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.dispatcher.DispatcherGateway;
+import org.apache.flink.runtime.rest.handler.HandlerRequest;
+import org.apache.flink.runtime.rest.messages.EmptyMessageParameters;
+import org.apache.flink.runtime.rest.messages.EmptyRequestBody;
+import org.apache.flink.runtime.rest.messages.JobPlanInfo;
+import org.apache.flink.runtime.testingUtils.TestingUtils;
+import org.apache.flink.runtime.webmonitor.RestfulGateway;
+import org.apache.flink.runtime.webmonitor.TestingDispatcherGateway;
+import org.apache.flink.runtime.webmonitor.retriever.GatewayRetriever;
+
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+
+/**
+ * TODO: Add javadoc.
+ */
+public class JarHandlers {
+
+ final JarUploadHandler uploadHandler;
+ final JarListHandler listHandler;
+ final JarPlanHandler planHandler;
+ final JarRunHandler runHandler;
+ final JarDeleteHandler deleteHandler;
+
+ JarHandlers(final Path jarDir, final TestingDispatcherGateway
restfulGateway) {
+ final GatewayRetriever
gatewayRetriever = () -> CompletableFuture.completedFuture(restfulGateway);
+ final Time timeout = Time.seconds(10);
+ final Map responseHeaders =
Collections.emptyMap();
+ final Executor executor = TestingUtils.defaultExecutor();
+
+ uploadHandler = new JarUploadHandler(
+ gatewayRetriever,
+ timeout,
+ responseHeaders,
+ JarUploadHeaders.getInstance(),
+ jarDir,
+ executor);
+
+ listHandler = new JarListHandler(
+ gatewayRetriever,
+ timeout,
+ responseHeaders,
+ JarListHeaders.getInstance(),
+
CompletableFuture.completedFuture("shazam://localhost:12345"),
+ jarDir.toFile(),
+ new Configuration(),
+ executor);
+
+ planHandler = new JarPlanHandler(
+ gatewayRetriever,
+ timeout,
+ responseHeaders,
+ JarPlanGetHeaders.getInstance(),
+ jarDir,
+ new Configuration(),
+ executor);
+
+ runHandler = new JarRunHandler(
+ gatewayRetriever,
+ timeout,
+ responseHeaders,
+ JarRunHeaders.getInstance(),
+ jarDir,
+ new Configuration(),
+ executor);
+
+ deleteHandler = new JarDeleteHandler(
+ gatewayRetriever,
+ timeout,
+ responseHeaders,
+ JarDeleteHeaders.getInstance(),
+
[flink] branch release-1.10 updated (fbe6e0d -> 0709bf1)
This is an automated email from the ASF dual-hosted git repository.
chesnay pushed a change to branch release-1.10
in repository https://gitbox.apache.org/repos/asf/flink.git.
from fbe6e0d [FLINK-15274][docs]Added separate plugin page.
new 69028bd [hotfix][tests] Make JarHandlers re-usable
new 0709bf1 [FLINK-15651][tests] Refactor JarHandlerTest
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
flink-runtime-web/pom.xml | 19 +++
.../webmonitor/handlers/JarHandlerTest.java| 133 +-
.../runtime/webmonitor/handlers/JarHandlers.java | 155 +
.../webmonitor/handlers/JarSubmissionITCase.java | 132 +-
.../{TestProgram.java => OutputTestProgram.java} | 10 +-
5 files changed, 216 insertions(+), 233 deletions(-)
create mode 100644
flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/handlers/JarHandlers.java
copy
flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/handlers/utils/{TestProgram.java
=> OutputTestProgram.java} (80%)
[flink] branch master updated (4615602 -> 23ff230)
This is an automated email from the ASF dual-hosted git repository.
chesnay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git.
from 4615602 [FLINK-15603][web] Expose checkpointStartDelayNanos metric in
the WebUI
add e7bfa88 [hotfix][tests] Make JarHandlers re-usable
add 23ff230 [FLINK-15651][tests] Refactor JarHandlerTest
No new revisions were added by this update.
Summary of changes:
flink-runtime-web/pom.xml | 19 +++
.../webmonitor/handlers/JarHandlerTest.java| 133 +-
.../runtime/webmonitor/handlers/JarHandlers.java | 155 +
.../webmonitor/handlers/JarSubmissionITCase.java | 132 +-
.../{TestProgram.java => OutputTestProgram.java} | 10 +-
5 files changed, 216 insertions(+), 233 deletions(-)
create mode 100644
flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/handlers/JarHandlers.java
copy
flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/handlers/utils/{TestProgram.java
=> OutputTestProgram.java} (80%)
