Re: [DISCUSS] FLIP-84: Improve & Refactor execute/sqlQuery/sqlUpdate APIS of TableEnvironment

[Jingsong Li]

Hi Godfrey,

Thanks for updating. +1 sketchy.

I have no idea to change "sqlQuery" to "fromQuery", I think "sqlQuery" is
OK, It's not that confusing with return values.

Can we change the "DmlBatch" to "Inserts"?  I don't see any other needs.
"Dml" seems a little weird.
It is better to support "Inserts addInsert" too. Users can
"inserts.addInsert().addInsert()"

I try to match the new interfaces with the old interfaces simply.
- "startInserts -> addInsert" replace old "sqlUpdate(insert)" and
"insertInto".
- "executeStatement" new one, execute all kinds of sqls immediately.
Including old "sqlUpdate(DDLs)".

Best,
Jingsong Lee

On Wed, Feb 12, 2020 at 11:10 AM godfreyhe  wrote:

> Hi everyone,
>
> I'd like to resume the discussion for FlIP-84 [0]. I had updated the
> document, the mainly changes are:
>
> 1. about "`void sqlUpdate(String sql)`" section
>   a) change "Optional executeSql(String sql) throws Exception"
> to "ResultTable executeStatement(String statement, String jobName) throws
> Exception". The reason is: "statement" is a more general concept than
> "sql",
> e.g. "show xx" is not a sql command (refer to [1]), but is a statement
> (just
> like JDBC). "insert" statement also has return value which is the affected
> row count, we can unify the return type to "ResultTable" instead of
> "Optional".
>   b) add two sub-interfaces for "ResultTable": "RowResultTable" is used for
> non-streaming select statement and will not contain change flag;
> "RowWithChangeFlagResultTable" is used for streaming select statement and
> will contain change flag.
>
> 2) about "Support batch sql execute and explain" section
> introduce "DmlBatch" to support both sql and Table API (which is borrowed
> from the ideas Dawid mentioned in the slack)
>
> interface TableEnvironment {
> DmlBatch startDmlBatch();
> }
>
> interface DmlBatch {
>   /**
>   * add insert statement to the batch
>   */
> void addInsert(String insert);
>
>  /**
>   * add Table with given sink name to the batch
>   */
> void addInsert(String sinkName, Table table);
>
>  /**
>   * execute the dml statements as a batch
>   */
>   ResultTable execute(String jobName) throws Exception
>
>   /**
>  * Returns the AST and the execution plan to compute the result of the
> batch
> dml statement.
>   */
>   String explain(boolean extended);
> }
>
> 3) about "Discuss a parse method for multiple statements execute in SQL
> CLI"
> section
> add the pros and cons for each solution
>
> 4) update the "Examples" section and "Summary" section based on the above
> changes
>
> Please refer the design doc[1] for more details and welcome any feedback.
>
> Bests,
> godfreyhe
>
>
> [0]
>
> https://docs.google.com/document/d/19-mdYJjKirh5aXCwq1fDajSaI09BJMMT95wy_YhtuZk/edit
> [1] https://www.geeksforgeeks.org/sql-ddl-dql-dml-dcl-tcl-commands/
>
>
>
> --
> Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
>


-- 
Best, Jingsong Lee

[jira] [Created] (FLINK-16031) Improve the description in the README file of PyFlink 1.9.x

[Wei Zhong (Jira)]

Wei Zhong created FLINK-16031:
-

 Summary:  Improve the description in the README file of PyFlink 
1.9.x
 Key: FLINK-16031
 URL: https://issues.apache.org/jira/browse/FLINK-16031
 Project: Flink
  Issue Type: Improvement
  Components: API / Python
Affects Versions: 1.9.1
Reporter: Wei Zhong
 Fix For: 1.9.3, 1.9.2


Currently, the description in the README file of PyFlink 1.9.x is not suitable 
for publishing in PyPI. It should be changed to be more user-friendly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16030) Add heartbeat between netty server and client to detect long connection alive

[begginghard (Jira)]

begginghard created FLINK-16030:
---

 Summary: Add heartbeat between netty server and client to detect 
long connection alive
 Key: FLINK-16030
 URL: https://issues.apache.org/jira/browse/FLINK-16030
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Network
Affects Versions: 1.10.0
Reporter: begginghard
 Fix For: 1.10.1


Network can fail in many ways, sometimes pretty subtle (e.g. high ratio packet 
loss).  
When the long tcp connection between netty client and server is lost, the 
server would failed to send response to the client, then shut down the channel. 
At the same time, the netty client does not know that the connection has been 
disconnected, so it has been waiting for two hours.

To detect the long tcp connection alive on netty client and server, we should 
have two ways: tcp keepalive and heartbeat.
 
The tcp keepalive is 2 hours by default. When the long tcp connection dead, you 
continue to wait for 2 hours, the netty client will trigger exception and enter 
failover recovery.

If you want to detect quickly, netty provides IdleStateHandler which it use 
ping-pang mechanism. If netty client sends continuously n ping message and 
receives no one pang message, then trigger exception.
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] Apache Flink Python API(PyFlink) 1.9.2 released

[Dian Fu]

Thanks for the great work, Jincheng.

Regards,
Dian

> 在 2020年2月13日，下午1:32，jincheng sun  写道：
> 
> Hi everyone,
> 
> The Apache Flink community is very happy to announce the release of Apache 
> Flink Python API(PyFlink) 1.9.2, which is the first release to PyPI for the 
> Apache Flink Python API 1.9 series.
>  
> Apache Flink® is an open-source stream processing framework for distributed, 
> high-performing, always-available, and accurate data streaming applications.
>  
> The release is available for download at:
> 
> https://pypi.org/project/apache-flink/1.9.2/#files 
> 
> 
> Or installed using pip command: 
> 
> pip install apache-flink==1.9.2
>  
> We would like to thank all contributors of the Apache Flink community who 
> helped to verify this release and made this release possible!
> 
> Best,
> Jincheng

[jira] [Created] (FLINK-16029) Remove register source and sink in test cases of planner

[Zhenghua Gao (Jira)]

Zhenghua Gao created FLINK-16029:


 Summary: Remove register source and sink in test cases of planner
 Key: FLINK-16029
 URL: https://issues.apache.org/jira/browse/FLINK-16029
 Project: Flink
  Issue Type: Sub-task
Reporter: Zhenghua Gao


Many test cases of planner use TableEnvironement.registerTableSource() and 
registerTableSink() which should be avoid。We want to refactor these cases via 
TableEnvironment.connect().



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[ANNOUNCE] Apache Flink Python API(PyFlink) 1.9.2 released

[jincheng sun]

Hi everyone,

The Apache Flink community is very happy to announce the release of Apache
Flink Python API(PyFlink) 1.9.2, which is the first release to PyPI for the
Apache Flink Python API 1.9 series.

Apache Flink® is an open-source stream processing framework for
distributed, high-performing, always-available, and accurate data streaming
applications.

The release is available for download at:

https://pypi.org/project/apache-flink/1.9.2/#files

Or installed using pip command:

pip install apache-flink==1.9.2

We would like to thank all contributors of the Apache Flink community who
helped to verify this release and made this release possible!

Best,
Jincheng

[jira] [Created] (FLINK-16028) hbase connector's 'connector.table-name' property should be optional rather than required

[Bowen Li (Jira)]

Bowen Li created FLINK-16028:


 Summary: hbase connector's 'connector.table-name' property should 
be optional rather than required
 Key: FLINK-16028
 URL: https://issues.apache.org/jira/browse/FLINK-16028
 Project: Flink
  Issue Type: Improvement
Reporter: Bowen Li


 

cc [~lzljs3620320] [~jark]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16027) kafka connector's 'connector.topic' property should be optional rather than required

[Bowen Li (Jira)]

Bowen Li created FLINK-16027:


 Summary: kafka connector's 'connector.topic' property should be 
optional rather than required
 Key: FLINK-16027
 URL: https://issues.apache.org/jira/browse/FLINK-16027
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Kafka
Reporter: Bowen Li
Assignee: Jingsong Lee
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16026) Travis failed due to python setup

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-16026:


 Summary: Travis failed due to python setup
 Key: FLINK-16026
 URL: https://issues.apache.org/jira/browse/FLINK-16026
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Reporter: Jingsong Lee
 Fix For: 1.11.0


[https://api.travis-ci.com/v3/job/286671652/log.txt]

[https://api.travis-ci.org/v3/job/649754603/log.txt]

[https://api.travis-ci.com/v3/job/286409130/log.txt]

Collecting avro-python3<2.0.0,>=1.8.1; python_version >= "3.0" (from 
apache-beam==2.19.0->apache-flink==1.11.dev0) Using cached 
https://files.pythonhosted.org/packages/31/21/d98e2515e5ca0337d7e747e8065227ee77faf5c817bbb74391899613178a/avro-python3-1.9.2.tar.gz
 Complete output from command python setup.py egg_info: Traceback (most recent 
call last): File "", line 1, in  File 
"/tmp/pip-install-d6uvsl_b/avro-python3/setup.py", line 41, in  import 
pycodestyle ModuleNotFoundError: No module named 'pycodestyle' 
 Command "python setup.py egg_info" 
failed with error code 1 in /tmp/pip-install-d6uvsl_b/avro-python3/ You are 
using pip version 10.0.1, however version 20.0.2 is available. You should 
consider upgrading via the 'pip install --upgrade pip' command.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Support scalar vectorized Python UDF in PyFlink

[Jingsong Li]

+1 (non-binding)
Thanks Dian for driving.

Best,
Jingsong Lee

On Thu, Feb 13, 2020 at 11:45 AM jincheng sun 
wrote:

> +1 (binding)
>
> Best,
> Jincheng
>
>
> Dian Fu  于2020年2月12日周三 下午1:31写道：
>
> > Hi all,
> >
> > I'd like to start the vote of FLIP-97[1] which is discussed and reached
> > consensus in the discussion thread[2].
> >
> > The vote will be open for at least 72 hours. Unless there is an
> objection,
> > I will try to close it by Feb 17, 2020 08:00 UTC if we have received
> > sufficient votes.
> >
> > Regards,
> > Dian
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-97%3A+Support+Scalar+Vectorized+Python+UDF+in+PyFlink
> > [2]
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-scalar-vectorized-Python-UDF-in-PyFlink-tt37264.html
>


-- 
Best, Jingsong Lee

Re: [VOTE] Support scalar vectorized Python UDF in PyFlink

[jincheng sun]

+1 (binding)

Best,
Jincheng


Dian Fu  于2020年2月12日周三 下午1:31写道：

> Hi all,
>
> I'd like to start the vote of FLIP-97[1] which is discussed and reached
> consensus in the discussion thread[2].
>
> The vote will be open for at least 72 hours. Unless there is an objection,
> I will try to close it by Feb 17, 2020 08:00 UTC if we have received
> sufficient votes.
>
> Regards,
> Dian
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-97%3A+Support+Scalar+Vectorized+Python+UDF+in+PyFlink
> [2]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-scalar-vectorized-Python-UDF-in-PyFlink-tt37264.html

Re: [ANNOUNCE] Apache Flink 1.10.0 released

[Zhijiang]

Really great work and thanks everyone involved, especially for the release 
managers!

Best,
Zhijiang
--
From:Kurt Young 
Send Time:2020 Feb. 13 (Thu.) 11:06
To:[None]
Cc:user ; dev 
Subject:Re: [ANNOUNCE] Apache Flink 1.10.0 released

Congratulations to everyone involved! 
Great thanks to Yu & Gary for being the release manager!

Best,
Kurt


On Thu, Feb 13, 2020 at 10:06 AM Hequn Cheng  wrote:

Great thanks to Yu & Gary for being the release manager! 
Also thanks to everyone who made this release possible!

Best, Hequn
On Thu, Feb 13, 2020 at 9:54 AM Rong Rong  wrote:
Congratulations, a big thanks to the release managers for all the hard works!!

--
Rong
On Wed, Feb 12, 2020 at 5:52 PM Yang Wang  wrote:
Excellent work. Thanks Gary & Yu for being the release manager.


Best,
Yang
Jeff Zhang  于2020年2月13日周四 上午9:36写道：
Congratulations! Really appreciated your hard work.

Yangze Guo  于2020年2月13日周四 上午9:29写道：
Thanks, Gary & Yu. Congrats to everyone involved!

 Best,
 Yangze Guo

 On Thu, Feb 13, 2020 at 9:23 AM Jingsong Li  wrote:
 >
 > Congratulations! Great work.
 >
 > Best,
 > Jingsong Lee
 >
 > On Wed, Feb 12, 2020 at 11:05 PM Leonard Xu  wrote:
 >>
 >> Great news！
 >> Thanks everyone involved !
 >> Thanks Gary and Yu for being the release manager !
 >>
 >> Best,
 >> Leonard Xu
 >>
 >> 在 2020年2月12日，23:02，Stephan Ewen  写道：
 >>
 >> Congrats to us all.
 >>
 >> A big piece of work, nicely done.
 >>
 >> Let's hope that this helps our users make their existing use cases easier 
 >> and also opens up new use cases.
 >>
 >> On Wed, Feb 12, 2020 at 3:31 PM 张光辉  wrote:
 >>>
 >>> Greet work.
 >>>
 >>> Congxian Qiu  于2020年2月12日周三 下午10:11写道：
 
  Great work.
  Thanks everyone involved.
  Thanks Gary and Yu for being the release manager
 
 
  Best,
  Congxian
 
 
  Jark Wu  于2020年2月12日周三 下午9:46写道：
 >
 > Congratulations to everyone involved!
 > Great thanks to Yu & Gary for being the release manager!
 >
 > Best,
 > Jark
 >
 > On Wed, 12 Feb 2020 at 21:42, Zhu Zhu  wrote:
 >>
 >> Cheers!
 >> Thanks Gary and Yu for the great job as release managers.
 >> And thanks to everyone whose contribution makes the release possible!
 >>
 >> Thanks,
 >> Zhu Zhu
 >>
 >> Wyatt Chun  于2020年2月12日周三 下午9:36写道：
 >>>
 >>> Sounds great. Congrats & Thanks!
 >>>
 >>> On Wed, Feb 12, 2020 at 9:31 PM Yu Li  wrote:
 
  The Apache Flink community is very happy to announce the release of 
  Apache Flink 1.10.0, which is the latest major release.
 
  Apache Flink(r) is an open-source stream processing framework for 
  distributed, high-performing, always-available, and accurate data 
  streaming applications.
 
  The release is available for download at:
  https://flink.apache.org/downloads.html
 
  Please check out the release blog post for an overview of the 
  improvements for this new major release:
  https://flink.apache.org/news/2020/02/11/release-1.10.0.html
 
  The full release notes are available in Jira:
  https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12345845
 
  We would like to thank all contributors of the Apache Flink community 
  who made this release possible!
 
  Cheers,
  Gary & Yu
 >>
 >>
 >
 >
 > --
 > Best, Jingsong Lee


-- 
Best Regards

Jeff Zhang

Re: [ANNOUNCE] Apache Flink 1.10.0 released

[Kurt Young]

Congratulations to everyone involved!
Great thanks to Yu & Gary for being the release manager!

Best,
Kurt


On Thu, Feb 13, 2020 at 10:06 AM Hequn Cheng  wrote:

> Great thanks to Yu & Gary for being the release manager!
> Also thanks to everyone who made this release possible!
>
> Best, Hequn
>
> On Thu, Feb 13, 2020 at 9:54 AM Rong Rong  wrote:
>
>> Congratulations, a big thanks to the release managers for all the hard
>> works!!
>>
>> --
>> Rong
>>
>> On Wed, Feb 12, 2020 at 5:52 PM Yang Wang  wrote:
>>
>>> Excellent work. Thanks Gary & Yu for being the release manager.
>>>
>>>
>>> Best,
>>> Yang
>>>
>>> Jeff Zhang  于2020年2月13日周四 上午9:36写道：
>>>
 Congratulations! Really appreciated your hard work.

 Yangze Guo  于2020年2月13日周四 上午9:29写道：

> Thanks, Gary & Yu. Congrats to everyone involved!
>
> Best,
> Yangze Guo
>
> On Thu, Feb 13, 2020 at 9:23 AM Jingsong Li 
> wrote:
> >
> > Congratulations! Great work.
> >
> > Best,
> > Jingsong Lee
> >
> > On Wed, Feb 12, 2020 at 11:05 PM Leonard Xu 
> wrote:
> >>
> >> Great news！
> >> Thanks everyone involved !
> >> Thanks Gary and Yu for being the release manager !
> >>
> >> Best,
> >> Leonard Xu
> >>
> >> 在 2020年2月12日，23:02，Stephan Ewen  写道：
> >>
> >> Congrats to us all.
> >>
> >> A big piece of work, nicely done.
> >>
> >> Let's hope that this helps our users make their existing use cases
> easier and also opens up new use cases.
> >>
> >> On Wed, Feb 12, 2020 at 3:31 PM 张光辉  wrote:
> >>>
> >>> Greet work.
> >>>
> >>> Congxian Qiu  于2020年2月12日周三 下午10:11写道：
> 
>  Great work.
>  Thanks everyone involved.
>  Thanks Gary and Yu for being the release manager
> 
> 
>  Best,
>  Congxian
> 
> 
>  Jark Wu  于2020年2月12日周三 下午9:46写道：
> >
> > Congratulations to everyone involved!
> > Great thanks to Yu & Gary for being the release manager!
> >
> > Best,
> > Jark
> >
> > On Wed, 12 Feb 2020 at 21:42, Zhu Zhu  wrote:
> >>
> >> Cheers!
> >> Thanks Gary and Yu for the great job as release managers.
> >> And thanks to everyone whose contribution makes the release
> possible!
> >>
> >> Thanks,
> >> Zhu Zhu
> >>
> >> Wyatt Chun  于2020年2月12日周三 下午9:36写道：
> >>>
> >>> Sounds great. Congrats & Thanks!
> >>>
> >>> On Wed, Feb 12, 2020 at 9:31 PM Yu Li 
> wrote:
> 
>  The Apache Flink community is very happy to announce the
> release of Apache Flink 1.10.0, which is the latest major release.
> 
>  Apache Flink® is an open-source stream processing framework
> for distributed, high-performing, always-available, and accurate data
> streaming applications.
> 
>  The release is available for download at:
>  https://flink.apache.org/downloads.html
> 
>  Please check out the release blog post for an overview of the
> improvements for this new major release:
>  https://flink.apache.org/news/2020/02/11/release-1.10.0.html
> 
>  The full release notes are available in Jira:
> 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12345845
> 
>  We would like to thank all contributors of the Apache Flink
> community who made this release possible!
> 
>  Cheers,
>  Gary & Yu
> >>
> >>
> >
> >
> > --
> > Best, Jingsong Lee
>


 --
 Best Regards

 Jeff Zhang

>>>

Re: [DISCUSS] Improve history server with log support

[SHI Xiaogang]

Hi Rong Rong,

Thanks for the proposal. We are also suffering from some pains brought by
history server. To address them, we propose a trace system, which is very
similar to the metric system, for historical information.

A trace is semi-structured information about events in Flink. Useful traces
include:
* job traces: which contain the job graph of submitted jobs.
* schedule traces: A schedule trace is typically composed of the
information of task slots. They are generated when a job finishes, fails,
or is canceled. As a job may restart mutliple times, a job typically has
multiple schedule traces.
* checkpoint traces: which are generated when a checkpoint completes or
fails.
* task manager traces: which are generated when a task manager terminates.
Users can access the link to aggregated logs intaskmanager traces.

Users can use TraceReport to collect traces in Flink and export them to
external storage (e.g., ElasticSearch). By retrieving traces when
exceptions happen, we can improve user experience in altering.

Regards,
Xiaogang

Rong Rong  于2020年2月13日周四 上午9:41写道：

> Hi All,
>
> Recently we have been experimenting using Flink’s history server as a
> centralized debugging service for completed streaming jobs.
>
> Specifically, we dynamically generate links to access log files on the YARN
> host; in the meantime, we use the Flink history server to show job graphs,
> exceptions and other info of the completed jobs[2].
>
> This causes some pain for our users, namely: It is inconvenient to go to
> YARN host to access logs; then go to Flink history server for the other
> information.
>
> Thus we would like to propose an improvement to the currently Flink history
> server:
>
>-
>
>To support dynamic links to residual log files from the host machine
>within the retention period [3];
>-
>
>To support dynamic links to aggregated log files provided by the
>cluster, if supported: such as Hadoop HistoryServer[1], or Kubernetes
>cluster level logging[4]?
>-
>
>   Similar integration with Hadoop HistoryServer was already proposed
>   before[5] with slightly different approach.
>
>
> Any feedback and suggestions are highly appreciated!
>
> --
>
> Rong
>
> [1]
>
> https://hadoop.apache.org/docs/r2.9.2/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/HistoryServerRest.html
>
> [2]
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.9/monitoring/historyserver.html
>
> [3]
>
> https://hadoop.apache.org/docs/r2.9.2/hadoop-yarn/hadoop-yarn-common/yarn-default.xml#yarn.nodemanager.log.retain-seconds
>
> [4]
>
> https://kubernetes.io/docs/concepts/cluster-administration/logging/#cluster-level-logging-architectures
> [5] https://issues.apache.org/jira/browse/FLINK-14317
>

Re: [DISCUSS] Support Python ML Pipeline API

[Becket Qin]

+1. I'd say this is almost a must-have for machine learning.

Thanks,

Jiangjie (Becket) Qin

On Thu, Feb 13, 2020 at 10:03 AM Rong Rong  wrote:

> Thanks for driving this initiative @Hequn Cheng .
>
> Moving towards python based ML is definitely a huge win consider how large
> the python-ML community is. a big +1 on my side!
> Regarding the doc, I only left a few comments on the specific APIs. overall
> the architecture looks very good!
>
> Looking forward to it!
> --
> Rong
>
> On Sun, Feb 9, 2020 at 10:28 PM Hequn Cheng  wrote:
>
> > Hi everyone,
> >
> > Thanks a lot for your feedback. I have created the FLIP[1].
> >
> > Best,
> > Hequn
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP+96%3A+Support+Python+ML+Pipeline+API
> >
> > On Mon, Feb 10, 2020 at 12:29 PM Dian Fu  wrote:
> >
> > > Hi Hequn,
> > >
> > > Thanks for bringing up the discussion. +1 to this feature. The design
> > LGTM.
> > > It's great that the Python ML users could use both the Java Pipeline
> > > Transformer/Estimator/Model classes and the Python
> > > Pipeline Transformer/Estimator/Model in the same job.
> > >
> > > Regards,
> > > Dian
> > >
> > > On Mon, Feb 10, 2020 at 11:08 AM jincheng sun <
> sunjincheng...@gmail.com>
> > > wrote:
> > >
> > > > Hi Hequn,
> > > >
> > > > Thanks for bring up this discussion.
> > > >
> > > > +1 for add Python ML Pipeline API, even though the Java pipeline API
> > may
> > > > change.
> > > >
> > > > I would like to suggest create a FLIP for this API changes. :)
> > > >
> > > > Best,
> > > > Jincheng
> > > >
> > > >
> > > > Hequn Cheng  于2020年2月5日周三 下午5:24写道：
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > FLIP-39[1] rebuilds the Flink ML pipeline on top of TableAPI and
> > > > introduces
> > > > > a new set of Java APIs. As Python is widely used in ML areas,
> > providing
> > > > > Python ML Pipeline APIs for Flink can not only make it easier to
> > write
> > > ML
> > > > > jobs for Python users but also broaden the adoption of Flink ML.
> > > > >
> > > > > Given this, Jincheng and I discussed offline about the support of
> > > Python
> > > > ML
> > > > > Pipeline API and drafted a design doc[2]. We'd like to achieve
> three
> > > > goals
> > > > > for supporting Python Pipeline API:
> > > > > - Add Python pipeline API according to Java pipeline API(we will
> > adapt
> > > > the
> > > > > Python pipeline API if Java pipeline API changes).
> > > > > - Support native Python Transformer/Estimator/Model, i.e., users
> can
> > > > write
> > > > > not only Python Transformer/Estimator/Model wrappers for calling
> Java
> > > > ones
> > > > > but also can write native Python Transformer/Estimator/Models.
> > > > > - Ease of use. Support keyword arguments when defining parameters.
> > > > >
> > > > > More details can be found in the design doc and we are looking
> > forward
> > > to
> > > > > your feedback.
> > > > >
> > > > > Best,
> > > > > Hequn
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-39+Flink+ML+pipeline+and+ML+libs
> > > > > [2]
> > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1fwSO5sRNWMoYuvNgfQJUV6N2n2q5UEVA4sezCljKcVQ/edit?usp=sharing
> > > > >
> > > >
> > >
> >
>

[RESULT] [VOTE] Release Flink Python API(PyFlink) 1.9.2 to PyPI, release candidate #1

[jincheng sun]

Hi everyone,

I'm happy to announce that we have unanimously approved this release.

There are 7 approving votes, 3 of which are binding:

* Wei Zhong
* Jincheng Sun(binding)
* Dian Fu
* Hequn Cheng
* Jark Wu (binding)
* Jiangjie (Becket) Qin (binding)
* Xingbo Huang

There are no disapproving votes.

Thanks, everyone!

Best,
Jincheng

Re: [DISCUSS] Support Python ML Pipeline API

[Rong Rong]

Thanks for driving this initiative @Hequn Cheng .

Moving towards python based ML is definitely a huge win consider how large
the python-ML community is. a big +1 on my side!
Regarding the doc, I only left a few comments on the specific APIs. overall
the architecture looks very good!

Looking forward to it!
--
Rong

On Sun, Feb 9, 2020 at 10:28 PM Hequn Cheng  wrote:

> Hi everyone,
>
> Thanks a lot for your feedback. I have created the FLIP[1].
>
> Best,
> Hequn
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP+96%3A+Support+Python+ML+Pipeline+API
>
> On Mon, Feb 10, 2020 at 12:29 PM Dian Fu  wrote:
>
> > Hi Hequn,
> >
> > Thanks for bringing up the discussion. +1 to this feature. The design
> LGTM.
> > It's great that the Python ML users could use both the Java Pipeline
> > Transformer/Estimator/Model classes and the Python
> > Pipeline Transformer/Estimator/Model in the same job.
> >
> > Regards,
> > Dian
> >
> > On Mon, Feb 10, 2020 at 11:08 AM jincheng sun 
> > wrote:
> >
> > > Hi Hequn,
> > >
> > > Thanks for bring up this discussion.
> > >
> > > +1 for add Python ML Pipeline API, even though the Java pipeline API
> may
> > > change.
> > >
> > > I would like to suggest create a FLIP for this API changes. :)
> > >
> > > Best,
> > > Jincheng
> > >
> > >
> > > Hequn Cheng  于2020年2月5日周三 下午5:24写道：
> > >
> > > > Hi everyone,
> > > >
> > > > FLIP-39[1] rebuilds the Flink ML pipeline on top of TableAPI and
> > > introduces
> > > > a new set of Java APIs. As Python is widely used in ML areas,
> providing
> > > > Python ML Pipeline APIs for Flink can not only make it easier to
> write
> > ML
> > > > jobs for Python users but also broaden the adoption of Flink ML.
> > > >
> > > > Given this, Jincheng and I discussed offline about the support of
> > Python
> > > ML
> > > > Pipeline API and drafted a design doc[2]. We'd like to achieve three
> > > goals
> > > > for supporting Python Pipeline API:
> > > > - Add Python pipeline API according to Java pipeline API(we will
> adapt
> > > the
> > > > Python pipeline API if Java pipeline API changes).
> > > > - Support native Python Transformer/Estimator/Model, i.e., users can
> > > write
> > > > not only Python Transformer/Estimator/Model wrappers for calling Java
> > > ones
> > > > but also can write native Python Transformer/Estimator/Models.
> > > > - Ease of use. Support keyword arguments when defining parameters.
> > > >
> > > > More details can be found in the design doc and we are looking
> forward
> > to
> > > > your feedback.
> > > >
> > > > Best,
> > > > Hequn
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-39+Flink+ML+pipeline+and+ML+libs
> > > > [2]
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1fwSO5sRNWMoYuvNgfQJUV6N2n2q5UEVA4sezCljKcVQ/edit?usp=sharing
> > > >
> > >
> >
>

[jira] [Created] (FLINK-16025) Service could expose different blob server port mismatched with JM Container

[Canbin Zheng (Jira)]

Canbin Zheng created FLINK-16025:


 Summary: Service could expose different blob server port 
mismatched with JM Container
 Key: FLINK-16025
 URL: https://issues.apache.org/jira/browse/FLINK-16025
 Project: Flink
  Issue Type: Bug
  Components: Deployment / Kubernetes
Affects Versions: 1.10.0
Reporter: Canbin Zheng
 Fix For: 1.10.1, 1.11.0


The Service would always expose 6124 port if it should expose that port, and 
while building ServicePort we do not explicitly specify a target port, so the 
target port would always be 6124 too.
{code:java}
// From ServiceDecorator.java

servicePorts.add(getServicePort(
 getPortName(BlobServerOptions.PORT.key()),
 Constants.BLOB_SERVER_PORT));

private ServicePort getServicePort(String name, int port) {
   return new ServicePortBuilder()
  .withName(name)
  .withPort(port)
  .build();
}


{code}
 meanwhile the Container of the JM would expose the blob server port which is 
configured in the Flink Configuration,
{code:java}
// From FlinkMasterDeploymentDecorator.java

final int blobServerPort = KubernetesUtils.parsePort(flinkConfig, 
BlobServerOptions.PORT);

...

final Container container = createJobManagerContainer(flinkConfig, mainClass, 
hasLogback, hasLog4j, blobServerPort);
{code}
 so there is a risk that the TM could not executing Task due to dependencies 
fetching failure if the Service exposes a blob server port which is different 
from the JM Container when one configures the blob server port with a value 
different from 6124.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Release Flink Python API(PyFlink) 1.9.2 to PyPI, release candidate #1

[jincheng sun]

Hi folks，

Thanks everyone for voting. I'm closing the vote now and will post the
result as a separate email.

Best,
Jincheng


Xingbo Huang  于2020年2月13日周四 上午9:28写道：

> +1 (non-binding)
>
> - Install the PyFlink by `pip install` [SUCCESS]
> - Run word_count.py [SUCCESS]
>
> Thanks,
> Xingbo
>
> Becket Qin  于2020年2月12日周三 下午2:28写道：
>
>> +1 (binding)
>>
>> - verified signature
>> - Ran word count example successfully.
>>
>> Thanks,
>>
>> Jiangjie (Becket) Qin
>>
>> On Wed, Feb 12, 2020 at 1:29 PM Jark Wu  wrote:
>>
>>> +1
>>>
>>> - checked/verified signatures and hashes
>>> - Pip installed the package successfully: pip install
>>> apache-flink-1.9.2.tar.gz
>>> - Run word count example successfully through the documentation [1].
>>>
>>> Best,
>>> Jark
>>>
>>> [1]:
>>>
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.9/tutorials/python_table_api.html
>>>
>>> On Tue, 11 Feb 2020 at 22:00, Hequn Cheng  wrote:
>>>
>>> > +1 (non-binding)
>>> >
>>> > - Check signature and checksum.
>>> > - Install package successfully with Pip under Python 3.7.4.
>>> > - Run wordcount example successfully under Python 3.7.4.
>>> >
>>> > Best, Hequn
>>> >
>>> > On Tue, Feb 11, 2020 at 12:17 PM Dian Fu 
>>> wrote:
>>> >
>>> > > +1 (non-binding)
>>> > >
>>> > > - Verified the signature and checksum
>>> > > - Pip installed the package successfully: pip install
>>> > > apache-flink-1.9.2.tar.gz
>>> > > - Run word count example successfully.
>>> > >
>>> > > Regards,
>>> > > Dian
>>> > >
>>> > > 在 2020年2月11日，上午11:44，jincheng sun  写道：
>>> > >
>>> > >
>>> > > +1 (binding)
>>> > >
>>> > > - Install the PyFlink by `pip install` [SUCCESS]
>>> > > - Run word_count in both command line and IDE [SUCCESS]
>>> > >
>>> > > Best,
>>> > > Jincheng
>>> > >
>>> > >
>>> > >
>>> > > Wei Zhong  于2020年2月11日周二 上午11:17写道：
>>> > >
>>> > >> Hi,
>>> > >>
>>> > >> Thanks for driving this, Jincheng.
>>> > >>
>>> > >> +1 (non-binding)
>>> > >>
>>> > >> - Verified signatures and checksums.
>>> > >> - Verified README.md and setup.py.
>>> > >> - Run `pip install apache-flink-1.9.2.tar.gz` in Python 2.7.15 and
>>> > Python
>>> > >> 3.7.5 successfully.
>>> > >> - Start local pyflink shell in Python 2.7.15 and Python 3.7.5 via
>>> > >> `pyflink-shell.sh local` and try the examples in the help message,
>>> run
>>> > well
>>> > >> and no exception.
>>> > >> - Try a word count example in IDE with Python 2.7.15 and Python
>>> 3.7.5,
>>> > >> run well and no exception.
>>> > >>
>>> > >> Best,
>>> > >> Wei
>>> > >>
>>> > >>
>>> > >> 在 2020年2月10日，19:12，jincheng sun  写道：
>>> > >>
>>> > >> Hi everyone,
>>> > >>
>>> > >> Please review and vote on the release candidate #1 for the PyFlink
>>> > >> version 1.9.2, as follows:
>>> > >>
>>> > >> [ ] +1, Approve the release
>>> > >> [ ] -1, Do not approve the release (please provide specific
>>> comments)
>>> > >>
>>> > >> The complete staging area is available for your review, which
>>> includes:
>>> > >>
>>> > >> * the official Apache binary convenience releases to be deployed to
>>> > >> dist.apache.org [1], which are signed with the key with fingerprint
>>> > >> 8FEA1EE9D0048C0CCC70B7573211B0703B79EA0E [2] and built from source
>>> code
>>> > [3].
>>> > >>
>>> > >> The vote will be open for at least 72 hours. It is adopted by
>>> majority
>>> > >> approval, with at least 3 PMC affirmative votes.
>>> > >>
>>> > >> Thanks,
>>> > >> Jincheng
>>> > >>
>>> > >> [1] https://dist.apache.org/repos/dist/dev/flink/flink-1.9.2-rc1/
>>> > >> [2] https://dist.apache.org/repos/dist/release/flink/KEYS
>>> > >> [3] https://github.com/apache/flink/tree/release-1.9.2
>>> > >>
>>> > >>
>>> > >
>>> >
>>>
>>

[DISCUSS] Improve history server with log support

[Rong Rong]

Hi All,

Recently we have been experimenting using Flink’s history server as a
centralized debugging service for completed streaming jobs.

Specifically, we dynamically generate links to access log files on the YARN
host; in the meantime, we use the Flink history server to show job graphs,
exceptions and other info of the completed jobs[2].

This causes some pain for our users, namely: It is inconvenient to go to
YARN host to access logs; then go to Flink history server for the other
information.

Thus we would like to propose an improvement to the currently Flink history
server:

   -

   To support dynamic links to residual log files from the host machine
   within the retention period [3];
   -

   To support dynamic links to aggregated log files provided by the
   cluster, if supported: such as Hadoop HistoryServer[1], or Kubernetes
   cluster level logging[4]?
   -

  Similar integration with Hadoop HistoryServer was already proposed
  before[5] with slightly different approach.


Any feedback and suggestions are highly appreciated!

--

Rong

[1]
https://hadoop.apache.org/docs/r2.9.2/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/HistoryServerRest.html

[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.9/monitoring/historyserver.html

[3]
https://hadoop.apache.org/docs/r2.9.2/hadoop-yarn/hadoop-yarn-common/yarn-default.xml#yarn.nodemanager.log.retain-seconds

[4]
https://kubernetes.io/docs/concepts/cluster-administration/logging/#cluster-level-logging-architectures
[5] https://issues.apache.org/jira/browse/FLINK-14317

Re: [ANNOUNCE] Apache Flink 1.10.0 released

[Jeff Zhang]

Congratulations! Really appreciated your hard work.

Yangze Guo  于2020年2月13日周四 上午9:29写道：

> Thanks, Gary & Yu. Congrats to everyone involved!
>
> Best,
> Yangze Guo
>
> On Thu, Feb 13, 2020 at 9:23 AM Jingsong Li 
> wrote:
> >
> > Congratulations! Great work.
> >
> > Best,
> > Jingsong Lee
> >
> > On Wed, Feb 12, 2020 at 11:05 PM Leonard Xu  wrote:
> >>
> >> Great news！
> >> Thanks everyone involved !
> >> Thanks Gary and Yu for being the release manager !
> >>
> >> Best,
> >> Leonard Xu
> >>
> >> 在 2020年2月12日，23:02，Stephan Ewen  写道：
> >>
> >> Congrats to us all.
> >>
> >> A big piece of work, nicely done.
> >>
> >> Let's hope that this helps our users make their existing use cases
> easier and also opens up new use cases.
> >>
> >> On Wed, Feb 12, 2020 at 3:31 PM 张光辉  wrote:
> >>>
> >>> Greet work.
> >>>
> >>> Congxian Qiu  于2020年2月12日周三 下午10:11写道：
> 
>  Great work.
>  Thanks everyone involved.
>  Thanks Gary and Yu for being the release manager
> 
> 
>  Best,
>  Congxian
> 
> 
>  Jark Wu  于2020年2月12日周三 下午9:46写道：
> >
> > Congratulations to everyone involved!
> > Great thanks to Yu & Gary for being the release manager!
> >
> > Best,
> > Jark
> >
> > On Wed, 12 Feb 2020 at 21:42, Zhu Zhu  wrote:
> >>
> >> Cheers!
> >> Thanks Gary and Yu for the great job as release managers.
> >> And thanks to everyone whose contribution makes the release
> possible!
> >>
> >> Thanks,
> >> Zhu Zhu
> >>
> >> Wyatt Chun  于2020年2月12日周三 下午9:36写道：
> >>>
> >>> Sounds great. Congrats & Thanks!
> >>>
> >>> On Wed, Feb 12, 2020 at 9:31 PM Yu Li  wrote:
> 
>  The Apache Flink community is very happy to announce the release
> of Apache Flink 1.10.0, which is the latest major release.
> 
>  Apache Flink® is an open-source stream processing framework for
> distributed, high-performing, always-available, and accurate data streaming
> applications.
> 
>  The release is available for download at:
>  https://flink.apache.org/downloads.html
> 
>  Please check out the release blog post for an overview of the
> improvements for this new major release:
>  https://flink.apache.org/news/2020/02/11/release-1.10.0.html
> 
>  The full release notes are available in Jira:
> 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12345845
> 
>  We would like to thank all contributors of the Apache Flink
> community who made this release possible!
> 
>  Cheers,
>  Gary & Yu
> >>
> >>
> >
> >
> > --
> > Best, Jingsong Lee
>


-- 
Best Regards

Jeff Zhang

Re: [ANNOUNCE] Apache Flink 1.10.0 released

[Yangze Guo]

Thanks, Gary & Yu. Congrats to everyone involved!

Best,
Yangze Guo

On Thu, Feb 13, 2020 at 9:23 AM Jingsong Li  wrote:
>
> Congratulations! Great work.
>
> Best,
> Jingsong Lee
>
> On Wed, Feb 12, 2020 at 11:05 PM Leonard Xu  wrote:
>>
>> Great news！
>> Thanks everyone involved !
>> Thanks Gary and Yu for being the release manager !
>>
>> Best,
>> Leonard Xu
>>
>> 在 2020年2月12日，23:02，Stephan Ewen  写道：
>>
>> Congrats to us all.
>>
>> A big piece of work, nicely done.
>>
>> Let's hope that this helps our users make their existing use cases easier 
>> and also opens up new use cases.
>>
>> On Wed, Feb 12, 2020 at 3:31 PM 张光辉  wrote:
>>>
>>> Greet work.
>>>
>>> Congxian Qiu  于2020年2月12日周三 下午10:11写道：

 Great work.
 Thanks everyone involved.
 Thanks Gary and Yu for being the release manager


 Best,
 Congxian


 Jark Wu  于2020年2月12日周三 下午9:46写道：
>
> Congratulations to everyone involved!
> Great thanks to Yu & Gary for being the release manager!
>
> Best,
> Jark
>
> On Wed, 12 Feb 2020 at 21:42, Zhu Zhu  wrote:
>>
>> Cheers!
>> Thanks Gary and Yu for the great job as release managers.
>> And thanks to everyone whose contribution makes the release possible!
>>
>> Thanks,
>> Zhu Zhu
>>
>> Wyatt Chun  于2020年2月12日周三 下午9:36写道：
>>>
>>> Sounds great. Congrats & Thanks!
>>>
>>> On Wed, Feb 12, 2020 at 9:31 PM Yu Li  wrote:

 The Apache Flink community is very happy to announce the release of 
 Apache Flink 1.10.0, which is the latest major release.

 Apache Flink® is an open-source stream processing framework for 
 distributed, high-performing, always-available, and accurate data 
 streaming applications.

 The release is available for download at:
 https://flink.apache.org/downloads.html

 Please check out the release blog post for an overview of the 
 improvements for this new major release:
 https://flink.apache.org/news/2020/02/11/release-1.10.0.html

 The full release notes are available in Jira:
 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12345845

 We would like to thank all contributors of the Apache Flink community 
 who made this release possible!

 Cheers,
 Gary & Yu
>>
>>
>
>
> --
> Best, Jingsong Lee

Re: [VOTE] Release Flink Python API(PyFlink) 1.9.2 to PyPI, release candidate #1

[Xingbo Huang]

+1 (non-binding)

- Install the PyFlink by `pip install` [SUCCESS]
- Run word_count.py [SUCCESS]

Thanks,
Xingbo

Becket Qin  于2020年2月12日周三 下午2:28写道：

> +1 (binding)
>
> - verified signature
> - Ran word count example successfully.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Wed, Feb 12, 2020 at 1:29 PM Jark Wu  wrote:
>
>> +1
>>
>> - checked/verified signatures and hashes
>> - Pip installed the package successfully: pip install
>> apache-flink-1.9.2.tar.gz
>> - Run word count example successfully through the documentation [1].
>>
>> Best,
>> Jark
>>
>> [1]:
>>
>> https://ci.apache.org/projects/flink/flink-docs-release-1.9/tutorials/python_table_api.html
>>
>> On Tue, 11 Feb 2020 at 22:00, Hequn Cheng  wrote:
>>
>> > +1 (non-binding)
>> >
>> > - Check signature and checksum.
>> > - Install package successfully with Pip under Python 3.7.4.
>> > - Run wordcount example successfully under Python 3.7.4.
>> >
>> > Best, Hequn
>> >
>> > On Tue, Feb 11, 2020 at 12:17 PM Dian Fu  wrote:
>> >
>> > > +1 (non-binding)
>> > >
>> > > - Verified the signature and checksum
>> > > - Pip installed the package successfully: pip install
>> > > apache-flink-1.9.2.tar.gz
>> > > - Run word count example successfully.
>> > >
>> > > Regards,
>> > > Dian
>> > >
>> > > 在 2020年2月11日，上午11:44，jincheng sun  写道：
>> > >
>> > >
>> > > +1 (binding)
>> > >
>> > > - Install the PyFlink by `pip install` [SUCCESS]
>> > > - Run word_count in both command line and IDE [SUCCESS]
>> > >
>> > > Best,
>> > > Jincheng
>> > >
>> > >
>> > >
>> > > Wei Zhong  于2020年2月11日周二 上午11:17写道：
>> > >
>> > >> Hi,
>> > >>
>> > >> Thanks for driving this, Jincheng.
>> > >>
>> > >> +1 (non-binding)
>> > >>
>> > >> - Verified signatures and checksums.
>> > >> - Verified README.md and setup.py.
>> > >> - Run `pip install apache-flink-1.9.2.tar.gz` in Python 2.7.15 and
>> > Python
>> > >> 3.7.5 successfully.
>> > >> - Start local pyflink shell in Python 2.7.15 and Python 3.7.5 via
>> > >> `pyflink-shell.sh local` and try the examples in the help message,
>> run
>> > well
>> > >> and no exception.
>> > >> - Try a word count example in IDE with Python 2.7.15 and Python
>> 3.7.5,
>> > >> run well and no exception.
>> > >>
>> > >> Best,
>> > >> Wei
>> > >>
>> > >>
>> > >> 在 2020年2月10日，19:12，jincheng sun  写道：
>> > >>
>> > >> Hi everyone,
>> > >>
>> > >> Please review and vote on the release candidate #1 for the PyFlink
>> > >> version 1.9.2, as follows:
>> > >>
>> > >> [ ] +1, Approve the release
>> > >> [ ] -1, Do not approve the release (please provide specific comments)
>> > >>
>> > >> The complete staging area is available for your review, which
>> includes:
>> > >>
>> > >> * the official Apache binary convenience releases to be deployed to
>> > >> dist.apache.org [1], which are signed with the key with fingerprint
>> > >> 8FEA1EE9D0048C0CCC70B7573211B0703B79EA0E [2] and built from source
>> code
>> > [3].
>> > >>
>> > >> The vote will be open for at least 72 hours. It is adopted by
>> majority
>> > >> approval, with at least 3 PMC affirmative votes.
>> > >>
>> > >> Thanks,
>> > >> Jincheng
>> > >>
>> > >> [1] https://dist.apache.org/repos/dist/dev/flink/flink-1.9.2-rc1/
>> > >> [2] https://dist.apache.org/repos/dist/release/flink/KEYS
>> > >> [3] https://github.com/apache/flink/tree/release-1.9.2
>> > >>
>> > >>
>> > >
>> >
>>
>

Re: [ANNOUNCE] Apache Flink 1.10.0 released

[Jingsong Li]

Congratulations! Great work.

Best,
Jingsong Lee

On Wed, Feb 12, 2020 at 11:05 PM Leonard Xu  wrote:

> Great news！
> Thanks everyone involved !
> Thanks Gary and Yu for being the release manager !
>
> Best,
> Leonard Xu
>
> 在 2020年2月12日，23:02，Stephan Ewen  写道：
>
> Congrats to us all.
>
> A big piece of work, nicely done.
>
> Let's hope that this helps our users make their existing use cases easier
> and also opens up new use cases.
>
> On Wed, Feb 12, 2020 at 3:31 PM 张光辉  wrote:
>
>> Greet work.
>>
>> Congxian Qiu  于2020年2月12日周三 下午10:11写道：
>>
>>> Great work.
>>> Thanks everyone involved.
>>> Thanks Gary and Yu for being the release manager
>>>
>>>
>>> Best,
>>> Congxian
>>>
>>>
>>> Jark Wu  于2020年2月12日周三 下午9:46写道：
>>>
 Congratulations to everyone involved!
 Great thanks to Yu & Gary for being the release manager!

 Best,
 Jark

 On Wed, 12 Feb 2020 at 21:42, Zhu Zhu  wrote:

> Cheers!
> Thanks Gary and Yu for the great job as release managers.
> And thanks to everyone whose contribution makes the release possible!
>
> Thanks,
> Zhu Zhu
>
> Wyatt Chun  于2020年2月12日周三 下午9:36写道：
>
>> Sounds great. Congrats & Thanks!
>>
>> On Wed, Feb 12, 2020 at 9:31 PM Yu Li  wrote:
>>
>>> The Apache Flink community is very happy to announce the release of
>>> Apache Flink 1.10.0, which is the latest major release.
>>>
>>> Apache Flink® is an open-source stream processing framework for
>>> distributed, high-performing, always-available, and accurate data 
>>> streaming
>>> applications.
>>>
>>> The release is available for download at:
>>> https://flink.apache.org/downloads.html
>>>
>>> Please check out the release blog post for an overview of the
>>> improvements for this new major release:
>>> https://flink.apache.org/news/2020/02/11/release-1.10.0.html
>>>
>>> The full release notes are available in Jira:
>>>
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12345845
>>>
>>> We would like to thank all contributors of the Apache Flink
>>> community who made this release possible!
>>>
>>> Cheers,
>>> Gary & Yu
>>>
>>
>

-- 
Best, Jingsong Lee

Re: [ANNOUNCE] Apache Flink 1.10.0 released

[Dian Fu]

Thanks Gary & Yu and everyone involved, Great work!

Regards,
Dian

> 在 2020年2月12日，下午11:57，Haibo Sun  写道：
> 
> Thanks Gary & Yu. Great work!
> 
> Best,
> Haibo
> 
> At 2020-02-12 21:31:00, "Yu Li"  wrote:
> >The Apache Flink community is very happy to announce the release of Apache
> >Flink 1.10.0, which is the latest major release.
> >
> >Apache Flink® is an open-source stream processing framework for
> >distributed, high-performing, always-available, and accurate data streaming
> >applications.
> >
> >The release is available for download at:
> >https://flink.apache.org/downloads.html
> >
> >Please check out the release blog post for an overview of the improvements
> >for this new major release:
> >https://flink.apache.org/news/2020/02/11/release-1.10.0.html
> >
> >The full release notes are available in Jira:
> >https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12345845
> >
> >We would like to thank all contributors of the Apache Flink community who
> >made this release possible!
> >
> >Cheers,
> >Gary & Yu

[jira] [Created] (FLINK-16024) support filter pushdown in jdbc connector

[Bowen Li (Jira)]

Bowen Li created FLINK-16024:


 Summary: support filter pushdown in jdbc connector
 Key: FLINK-16024
 URL: https://issues.apache.org/jira/browse/FLINK-16024
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / JDBC
Reporter: Bowen Li
Assignee: Jingsong Lee
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16023) jdbc connector's 'connector.table' property should be optional rather than required

[Bowen Li (Jira)]

Bowen Li created FLINK-16023:


 Summary: jdbc connector's 'connector.table' property should be 
optional rather than required
 Key: FLINK-16023
 URL: https://issues.apache.org/jira/browse/FLINK-16023
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / JDBC
Reporter: Bowen Li
Assignee: Jingsong Lee
 Fix For: 1.11.0


jdbc connector's 'connector.table' property should be optional rather than 
required.

 

connector should assume the table name in dbms is the same as that in Flink 
when this property is not present



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re:[ANNOUNCE] Apache Flink 1.10.0 released

[Haibo Sun]

Thanks Gary & Yu. Great work!


Best,
Haibo


At 2020-02-12 21:31:00, "Yu Li"  wrote:
>The Apache Flink community is very happy to announce the release of Apache
>Flink 1.10.0, which is the latest major release.
>
>Apache Flink® is an open-source stream processing framework for
>distributed, high-performing, always-available, and accurate data streaming
>applications.
>
>The release is available for download at:
>https://flink.apache.org/downloads.html
>
>Please check out the release blog post for an overview of the improvements
>for this new major release:
>https://flink.apache.org/news/2020/02/11/release-1.10.0.html
>
>The full release notes are available in Jira:
>https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12345845
>
>We would like to thank all contributors of the Apache Flink community who
>made this release possible!
>
>Cheers,
>Gary & Yu

[jira] [Created] (FLINK-16022) AbstractDownloadCache does not clean up download directory in case of failures

[Alexander Fedulov (Jira)]

Alexander Fedulov created FLINK-16022:
-

 Summary: AbstractDownloadCache does not clean up download 
directory in case of failures
 Key: FLINK-16022
 URL: https://issues.apache.org/jira/browse/FLINK-16022
 Project: Flink
  Issue Type: Improvement
  Components: Test Infrastructure
Affects Versions: 1.10.0
Reporter: Alexander Fedulov


The download directory is not cleaned up in the "unhappy" path. It's name is 
based on the hash of the downloaded file and serves as a condition to try 
unpacking the file instead of downloading. If something goes wrong during the 
downloading process, directory remains without the file and gets the 
DownloadCache into failing state.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] Apache Flink 1.10.0 released

[Fabian Hueske]

Congrats team and a big thank you to the release managers!

Am Mi., 12. Feb. 2020 um 16:33 Uhr schrieb Timo Walther :

> Congratualations everyone! Great stuff :-)
>
> Regards,
> Timo
>
>
> On 12.02.20 16:05, Leonard Xu wrote:
> > Great news！
> > Thanks everyone involved !
> > Thanks Gary and Yu for being the release manager !
> >
> > Best,
> > Leonard Xu
> >
> >> 在 2020年2月12日，23:02，Stephan Ewen  写道：
> >>
> >> Congrats to us all.
> >>
> >> A big piece of work, nicely done.
> >>
> >> Let's hope that this helps our users make their existing use cases
> easier and also opens up new use cases.
> >>
> >> On Wed, Feb 12, 2020 at 3:31 PM 张光辉  beggingh...@gmail.com>> wrote:
> >> Greet work.
> >>
> >> Congxian Qiu mailto:qcx978132...@gmail.com>>
> 于2020年2月12日周三 下午10:11写道：
> >> Great work.
> >> Thanks everyone involved.
> >> Thanks Gary and Yu for being the release manager
> >>
> >>
> >> Best,
> >> Congxian
> >>
> >>
> >> Jark Wu mailto:imj...@gmail.com>> 于2020年2月12日周三
> 下午9:46写道：
> >> Congratulations to everyone involved!
> >> Great thanks to Yu & Gary for being the release manager!
> >>
> >> Best,
> >> Jark
> >>
> >> On Wed, 12 Feb 2020 at 21:42, Zhu Zhu  reed...@gmail.com>> wrote:
> >> Cheers!
> >> Thanks Gary and Yu for the great job as release managers.
> >> And thanks to everyone whose contribution makes the release possible!
> >>
> >> Thanks,
> >> Zhu Zhu
> >>
> >> Wyatt Chun mailto:wyattc...@gmail.com>>
> 于2020年2月12日周三 下午9:36写道：
> >> Sounds great. Congrats & Thanks!
> >>
> >> On Wed, Feb 12, 2020 at 9:31 PM Yu Li  car...@gmail.com>> wrote:
> >> The Apache Flink community is very happy to announce the release of
> Apache Flink 1.10.0, which is the latest major release.
> >>
> >> Apache Flink® is an open-source stream processing framework for
> distributed, high-performing, always-available, and accurate data streaming
> applications.
> >>
> >> The release is available for download at:
> >> https://flink.apache.org/downloads.html <
> https://flink.apache.org/downloads.html>
> >>
> >> Please check out the release blog post for an overview of the
> improvements for this new major release:
> >> https://flink.apache.org/news/2020/02/11/release-1.10.0.html <
> https://flink.apache.org/news/2020/02/11/release-1.10.0.html>
> >>
> >> The full release notes are available in Jira:
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12345845
> <
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12345845
> >
> >>
> >> We would like to thank all contributors of the Apache Flink community
> who made this release possible!
> >>
> >> Cheers,
> >> Gary & Yu
> >
> >
>
>

Re: Add support for IAM Roles for Service Accounts in AWS EKS (Web Identity)

[Stephan Ewen]

This sounds like a good addition.

Can you comment on the jira issue, to have the discussion in one place.
Unless anyone raises concerns, I can assign you the issue then and we could
proceed with a PR.

On Tue, Feb 11, 2020 at 4:10 PM Rafi Aroch  wrote:

> Hi,
>
> IAM Roles for Service Accounts have many advantages when deploying Flink on
> AWS EKS.
>
> From AWS documentation:
>
> *With IAM roles for service accounts on Amazon EKS clusters, you can
> > associate an IAM role with a Kubernetes service account. This service
> > account can then provide AWS permissions to the containers in any pod
> that
> > uses that service account. With this feature, you no longer need to
> provide
> > extended permissions to the worker node IAM role so that pods on that
> node
> > can call AWS APIs.*
>
>
> As Kubernetes becomes the popular deployment method, I believe we should
> support this capability.
>
> In order for IAM Roles for Service Accounts to work, I see two necessary
> changes:
>
>- Bump the AWS SDK version to at least:  1.11.623.
>- Add dependency to AWS STS in order for the assume-role to work.
>
> This is relevant for S3 Filesystem & Kinesis modules.
>
> There is already an issue open:
> https://issues.apache.org/jira/browse/FLINK-14881
>
> Can I go ahead and create a pull request to add this?
>
> Thanks,
> Rafi
>

Re: [ANNOUNCE] Apache Flink 1.10.0 released

[Timo Walther]

Congratualations everyone! Great stuff :-)

Regards,
Timo


On 12.02.20 16:05, Leonard Xu wrote:

Great news！
Thanks everyone involved !
Thanks Gary and Yu for being the release manager !

Best,
Leonard Xu


在 2020年2月12日，23:02，Stephan Ewen  写道：

Congrats to us all.

A big piece of work, nicely done.

Let's hope that this helps our users make their existing use cases easier and 
also opens up new use cases.

On Wed, Feb 12, 2020 at 3:31 PM 张光辉 mailto:beggingh...@gmail.com>> wrote:
Greet work.

Congxian Qiu mailto:qcx978132...@gmail.com>> 
于2020年2月12日周三 下午10:11写道：
Great work.
Thanks everyone involved.
Thanks Gary and Yu for being the release manager


Best,
Congxian


Jark Wu mailto:imj...@gmail.com>> 于2020年2月12日周三 下午9:46写道：
Congratulations to everyone involved!
Great thanks to Yu & Gary for being the release manager!

Best,
Jark

On Wed, 12 Feb 2020 at 21:42, Zhu Zhu mailto:reed...@gmail.com>> wrote:
Cheers!
Thanks Gary and Yu for the great job as release managers.
And thanks to everyone whose contribution makes the release possible!

Thanks,
Zhu Zhu

Wyatt Chun mailto:wyattc...@gmail.com>> 于2020年2月12日周三 
下午9:36写道：
Sounds great. Congrats & Thanks!

On Wed, Feb 12, 2020 at 9:31 PM Yu Li mailto:car...@gmail.com>> wrote:
The Apache Flink community is very happy to announce the release of Apache 
Flink 1.10.0, which is the latest major release.

Apache Flink® is an open-source stream processing framework for distributed, 
high-performing, always-available, and accurate data streaming applications.

The release is available for download at:
https://flink.apache.org/downloads.html 


Please check out the release blog post for an overview of the improvements for 
this new major release:
https://flink.apache.org/news/2020/02/11/release-1.10.0.html 


The full release notes are available in Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12345845
 


We would like to thank all contributors of the Apache Flink community who made 
this release possible!

Cheers,
Gary & Yu

Re: [ANNOUNCE] Apache Flink 1.10.0 released

[Leonard Xu]

Great news！
Thanks everyone involved !
Thanks Gary and Yu for being the release manager !

Best,
Leonard Xu

> 在 2020年2月12日，23:02，Stephan Ewen  写道：
> 
> Congrats to us all.
> 
> A big piece of work, nicely done.
> 
> Let's hope that this helps our users make their existing use cases easier and 
> also opens up new use cases.
> 
> On Wed, Feb 12, 2020 at 3:31 PM 张光辉  > wrote:
> Greet work. 
> 
> Congxian Qiu mailto:qcx978132...@gmail.com>> 
> 于2020年2月12日周三 下午10:11写道：
> Great work.
> Thanks everyone involved.
> Thanks Gary and Yu for being the release manager
> 
> 
> Best,
> Congxian
> 
> 
> Jark Wu mailto:imj...@gmail.com>> 于2020年2月12日周三 下午9:46写道：
> Congratulations to everyone involved! 
> Great thanks to Yu & Gary for being the release manager!
> 
> Best,
> Jark
> 
> On Wed, 12 Feb 2020 at 21:42, Zhu Zhu  > wrote:
> Cheers!
> Thanks Gary and Yu for the great job as release managers. 
> And thanks to everyone whose contribution makes the release possible!
> 
> Thanks,
> Zhu Zhu
> 
> Wyatt Chun mailto:wyattc...@gmail.com>> 于2020年2月12日周三 
> 下午9:36写道：
> Sounds great. Congrats & Thanks!
> 
> On Wed, Feb 12, 2020 at 9:31 PM Yu Li  > wrote:
> The Apache Flink community is very happy to announce the release of Apache 
> Flink 1.10.0, which is the latest major release.
> 
> Apache Flink® is an open-source stream processing framework for distributed, 
> high-performing, always-available, and accurate data streaming applications.
> 
> The release is available for download at:
> https://flink.apache.org/downloads.html 
> 
> 
> Please check out the release blog post for an overview of the improvements 
> for this new major release:
> https://flink.apache.org/news/2020/02/11/release-1.10.0.html 
> 
> 
> The full release notes are available in Jira:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12345845
>  
> 
> 
> We would like to thank all contributors of the Apache Flink community who 
> made this release possible!
> 
> Cheers,
> Gary & Yu

Re: [ANNOUNCE] Apache Flink 1.10.0 released

[Stephan Ewen]

Congrats to us all.

A big piece of work, nicely done.

Let's hope that this helps our users make their existing use cases easier
and also opens up new use cases.

On Wed, Feb 12, 2020 at 3:31 PM 张光辉  wrote:

> Greet work.
>
> Congxian Qiu  于2020年2月12日周三 下午10:11写道：
>
>> Great work.
>> Thanks everyone involved.
>> Thanks Gary and Yu for being the release manager
>>
>>
>> Best,
>> Congxian
>>
>>
>> Jark Wu  于2020年2月12日周三 下午9:46写道：
>>
>>> Congratulations to everyone involved!
>>> Great thanks to Yu & Gary for being the release manager!
>>>
>>> Best,
>>> Jark
>>>
>>> On Wed, 12 Feb 2020 at 21:42, Zhu Zhu  wrote:
>>>
 Cheers!
 Thanks Gary and Yu for the great job as release managers.
 And thanks to everyone whose contribution makes the release possible!

 Thanks,
 Zhu Zhu

 Wyatt Chun  于2020年2月12日周三 下午9:36写道：

> Sounds great. Congrats & Thanks!
>
> On Wed, Feb 12, 2020 at 9:31 PM Yu Li  wrote:
>
>> The Apache Flink community is very happy to announce the release of
>> Apache Flink 1.10.0, which is the latest major release.
>>
>> Apache Flink® is an open-source stream processing framework for
>> distributed, high-performing, always-available, and accurate data 
>> streaming
>> applications.
>>
>> The release is available for download at:
>> https://flink.apache.org/downloads.html
>>
>> Please check out the release blog post for an overview of the
>> improvements for this new major release:
>> https://flink.apache.org/news/2020/02/11/release-1.10.0.html
>>
>> The full release notes are available in Jira:
>>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12345845
>>
>> We would like to thank all contributors of the Apache Flink community
>> who made this release possible!
>>
>> Cheers,
>> Gary & Yu
>>
>

[jira] [Created] (FLINK-16021) DescriptorProperties.putTableSchema does not include constraints

[Timo Walther (Jira)]

Timo Walther created FLINK-16021:


 Summary: DescriptorProperties.putTableSchema does not include 
constraints
 Key: FLINK-16021
 URL: https://issues.apache.org/jira/browse/FLINK-16021
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API
Reporter: Timo Walther


FLINK-14978 added primary keys as the first constraints but forgot about adding 
them to the property map as well.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[VOTE] Release flink-shaded 10.0, release candidate #3

[Chesnay Schepler]

Hi everyone,
Please review and vote on the release candidate #3 for the version 10.0, 
as follows:

[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to dist.apache.org 
[2], which are signed with the key with fingerprint 11D464BA [3],

* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "release-10.0-rc3 [5],
* website pull request listing the new release [6].

The vote will be open for at least 72 hours. It is adopted by majority 
approval, with at least 3 PMC affirmative votes.


Thanks,
Chesnay

[1] 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12346746

[2] https://dist.apache.org/repos/dist/dev/flink/flink-shaded-10.0-rc3/
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] https://repository.apache.org/content/repositories/orgapacheflink-1337
[5] 
https://gitbox.apache.org/repos/asf?p=flink-shaded.git;a=tag;h=refs/tags/release-10.0-rc3

[6] https://github.com/apache/flink-web/pull/304

[CANCELED][VOTE] Release flink-shaded 10.0, release candidate #2

[Chesnay Schepler]

Canceled because I forgot to do _the one thing_ that was required for 
rc2: cherry-picking the commit to the 10.0 release branch.


On 12/02/2020 12:52, Chesnay Schepler wrote:

Hi everyone,
Please review and vote on the release candidate #2 for the version 
10.0, as follows:

[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to dist.apache.org 
[2], which are signed with the key with fingerprint 11D464BA [3],

* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "release-10.0-rc2 [5],
* website pull request listing the new release [6].

The vote will be open for at least 72 hours. It is adopted by majority 
approval, with at least 3 PMC affirmative votes.


Thanks,
Chesnay

[1] 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12346746

[2] https://dist.apache.org/repos/dist/dev/flink/flink-shaded-10.0-rc2/
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] 
https://repository.apache.org/content/repositories/orgapacheflink-1336
[5] 
https://gitbox.apache.org/repos/asf?p=flink-shaded.git;a=tag;h=refs/tags/release-10.0-rc2

[6] https://github.com/apache/flink-web/pull/304

Re: [ANNOUNCE] Apache Flink 1.10.0 released

[Xintong Song]

Great to hear that!

Thanks for being the release managers, Gary & Yu. Great work!


Thank you~

Xintong Song



On Wed, Feb 12, 2020 at 9:31 PM Yu Li  wrote:

> The Apache Flink community is very happy to announce the release of Apache
> Flink 1.10.0, which is the latest major release.
>
> Apache Flink® is an open-source stream processing framework for
> distributed, high-performing, always-available, and accurate data streaming
> applications.
>
> The release is available for download at:
> https://flink.apache.org/downloads.html
>
> Please check out the release blog post for an overview of the improvements
> for this new major release:
> https://flink.apache.org/news/2020/02/11/release-1.10.0.html
>
> The full release notes are available in Jira:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12345845
>
> We would like to thank all contributors of the Apache Flink community who
> made this release possible!
>
> Cheers,
> Gary & Yu
>

[ANNOUNCE] Apache Flink 1.10.0 released

[Yu Li]

The Apache Flink community is very happy to announce the release of Apache
Flink 1.10.0, which is the latest major release.

Apache Flink® is an open-source stream processing framework for
distributed, high-performing, always-available, and accurate data streaming
applications.

The release is available for download at:
https://flink.apache.org/downloads.html

Please check out the release blog post for an overview of the improvements
for this new major release:
https://flink.apache.org/news/2020/02/11/release-1.10.0.html

The full release notes are available in Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12345845

We would like to thank all contributors of the Apache Flink community who
made this release possible!

Cheers,
Gary & Yu

[jira] [Created] (FLINK-16020) Use Flink-1.10 released version instead of the snapshot version

[Igal Shilman (Jira)]

Igal Shilman created FLINK-16020:


 Summary: Use Flink-1.10 released version instead of the snapshot 
version
 Key: FLINK-16020
 URL: https://issues.apache.org/jira/browse/FLINK-16020
 Project: Flink
  Issue Type: Task
  Components: Stateful Functions
Reporter: Igal Shilman


Since Flink 1.10 was released, we can stop using Flink 1.10-SNAPSHOT,
this include both maven and docker.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16019) ContinuousFileReaderOperatorBenchmark restarts indefinetly on failure

[Roman Khachatryan (Jira)]

Roman Khachatryan created FLINK-16019:
-

 Summary: ContinuousFileReaderOperatorBenchmark restarts 
indefinetly on failure
 Key: FLINK-16019
 URL: https://issues.apache.org/jira/browse/FLINK-16019
 Project: Flink
  Issue Type: Bug
  Components: Benchmarks
Reporter: Roman Khachatryan






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16018) Improve error reporting when submitting batch job (instead of AskTimeoutException)

[Robert Metzger (Jira)]

Robert Metzger created FLINK-16018:
--

 Summary: Improve error reporting when submitting batch job 
(instead of AskTimeoutException)
 Key: FLINK-16018
 URL: https://issues.apache.org/jira/browse/FLINK-16018
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Affects Versions: 1.11.0
Reporter: Robert Metzger


While debugging the {{Shaded Hadoop S3A end-to-end test (minio)}} pre-commit 
test, I noticed that the JobSubmission is not producing very helpful error 
messages.

Environment:
- A simple batch wordcount job 
- a unavailable minio s3 filesystem service

What happens from a user's perspective:
- The job submission fails after 10 seconds with a AskTimeoutException:
{code}
2020-02-07T11:38:27.1189393Z akka.pattern.AskTimeoutException: Ask timed out on 
[Actor[akka://flink/user/dispatcher#-939201095]] after [1 ms]. Message of 
type [org.apache.flink.runtime.rpc.messages.LocalFencedMessage]. A typical 
reason for `AskTimeoutException` is that the recipient actor didn't send a 
reply.
2020-02-07T11:38:27.1189538Zat 
akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635)
2020-02-07T11:38:27.1189616Zat 
akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635)
2020-02-07T11:38:27.1189713Zat 
akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:648)
2020-02-07T11:38:27.1189789Zat 
akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
2020-02-07T11:38:27.1189883Zat 
scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
2020-02-07T11:38:27.1189973Zat 
scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
2020-02-07T11:38:27.1190067Zat 
scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
2020-02-07T11:38:27.1190159Zat 
akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
2020-02-07T11:38:27.1190267Zat 
akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)
2020-02-07T11:38:27.1190358Zat 
akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)
2020-02-07T11:38:27.1190465Zat 
akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
2020-02-07T11:38:27.1190540Zat java.lang.Thread.run(Thread.java:748)
{code}

What a user would expect:
- An error message indicating why the job submission failed.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[VOTE] Release flink-shaded 10.0, release candidate #2

[Chesnay Schepler]

Hi everyone,
Please review and vote on the release candidate #2 for the version 10.0, 
as follows:

[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to dist.apache.org 
[2], which are signed with the key with fingerprint 11D464BA [3],

* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "release-10.0-rc2 [5],
* website pull request listing the new release [6].

The vote will be open for at least 72 hours. It is adopted by majority 
approval, with at least 3 PMC affirmative votes.


Thanks,
Chesnay

[1] 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12346746

[2] https://dist.apache.org/repos/dist/dev/flink/flink-shaded-10.0-rc2/
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] https://repository.apache.org/content/repositories/orgapacheflink-1336
[5] 
https://gitbox.apache.org/repos/asf?p=flink-shaded.git;a=tag;h=refs/tags/release-10.0-rc2

[6] https://github.com/apache/flink-web/pull/304

[jira] [Created] (FLINK-16017) Improve attachJobGraph Performance

[Jiayi Liao (Jira)]

Jiayi Liao created FLINK-16017:
--

 Summary: Improve attachJobGraph Performance
 Key: FLINK-16017
 URL: https://issues.apache.org/jira/browse/FLINK-16017
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Affects Versions: 1.10.0
Reporter: Jiayi Liao


Currently {{RegionPartitionReleaseStrategy}} and 
{{AdaptedRestartPipelinedRegionStrategyNG}} both need to compute distinct 
pipelined regions, which affects the performance due to the duplicate 
calculating.

The best idea that comes to my mind so far is calculating distinct pipelined 
regions in {{DefaultExecutionTopology}} so that we can remove the duplicate 
calculating.





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16016) The out-of-date warning doesn't show up in release-1.6 document

[Yu Li (Jira)]

Yu Li created FLINK-16016:
-

 Summary: The out-of-date warning doesn't show up in release-1.6 
document
 Key: FLINK-16016
 URL: https://issues.apache.org/jira/browse/FLINK-16016
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Reporter: Yu Li


We already have {{show_outdated_warning: true}} set in {{docs/_config.yml}} in 
release-1.6, and manually triggered a [new 
build|https://ci.apache.org/builders/flink-docs-release-1.6/builds/392] of the 
doc through build bot. However, the out-of-date warning still doesn't show, 
while the same process takes effect for {{release-1.8}}, so there must be 
something wrong for release-1.6.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16015) Refine fallback filesystems to only handle specific filesystems

[Arvid Heise (Jira)]

Arvid Heise created FLINK-16015:
---

 Summary: Refine fallback filesystems to only handle specific 
filesystems
 Key: FLINK-16015
 URL: https://issues.apache.org/jira/browse/FLINK-16015
 Project: Flink
  Issue Type: Improvement
  Components: FileSystems
Affects Versions: 1.10.1, 1.11.0
Reporter: Arvid Heise


Currently, if no s3 plugin is included, hadoop is used as a fallback, which 
introduces a wide variety of problems. We should probably only white list 
specific protocols that work well (e.g. hdfs).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16014) S3 plugin ClassNotFoundException SAXParser

[Arvid Heise (Jira)]

Arvid Heise created FLINK-16014:
---

 Summary: S3 plugin ClassNotFoundException SAXParser
 Key: FLINK-16014
 URL: https://issues.apache.org/jira/browse/FLINK-16014
 Project: Flink
  Issue Type: Bug
  Components: FileSystems
Affects Versions: 1.10.0, 1.11.0
Reporter: Arvid Heise


While stress-testing s3 plugin on EMR.

 
{noformat}
org.apache.flink.util.FlinkRuntimeException: Could not perform checkpoint 2 for 
operator Map (114/160).
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:839)
at 
org.apache.flink.streaming.runtime.io.CheckpointBarrierHandler.notifyCheckpoint(CheckpointBarrierHandler.java:104)
at 
org.apache.flink.streaming.runtime.io.CheckpointBarrierUnaligner.notifyBarrierReceived(CheckpointBarrierUnaligner.java:149)
at 
org.apache.flink.streaming.runtime.io.InputProcessorUtil$1.lambda$notifyBarrierReceived$0(InputProcessorUtil.java:80)
at 
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.run(StreamTaskActionExecutor.java:87)
at 
org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:78)
at 
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:255)
at 
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:186)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:508)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:492)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: 
org.apache.hadoop.fs.s3a.AWSClientIOException: getFileStatus on 
s3://emr-unaligned-checkpoints-testing-eu-central-1/inflight/9ae223e41008b17568d7f63c12360268_output/part-file.-1:
 com.amazonaws.SdkClientException: Couldn't initialize a SAX driver to create 
an XMLReader: Couldn't initialize a SAX driver to create an XMLReader
at 
org.apache.flink.runtime.io.network.BufferPersisterImpl$Writer.checkErroneousUnsafe(BufferPersisterImpl.java:262)
at 
org.apache.flink.runtime.io.network.BufferPersisterImpl$Writer.add(BufferPersisterImpl.java:137)
at 
org.apache.flink.runtime.io.network.BufferPersisterImpl.addBuffers(BufferPersisterImpl.java:66)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.prepareInflightDataSnapshot(StreamTask.java:935)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$5(StreamTask.java:898)
at 
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:870)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:826)
... 12 more
Caused by: org.apache.hadoop.fs.s3a.AWSClientIOException: getFileStatus on 
s3://emr-unaligned-checkpoints-testing-eu-central-1/inflight/9ae223e41008b17568d7f63c12360268_output/part-file.-1:
 com.amazonaws.SdkClientException: Couldn't initialize a SAX driver to create 
an XMLReader: Couldn't initialize a SAX driver to create an XMLReader
at 
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:177)
at 
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:145)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2251)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2149)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2088)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:2970)
at 
org.apache.flink.fs.s3hadoop.common.HadoopFileSystem.exists(HadoopFileSystem.java:152)
at 
org.apache.flink.core.fs.PluginFileSystemFactory$ClassLoaderFixingFileSystem.exists(PluginFileSystemFactory.java:143)
at 
org.apache.flink.core.fs.SafetyNetWrapperFileSystem.exists(SafetyNetWrapperFileSystem.java:102)
at 
org.apache.flink.runtime.io.network.BufferPersisterImpl$Writer.get(BufferPersisterImpl.java:213)
at 
org.apache.flink.runtime.io.network.BufferPersisterImpl$Writer.run(BufferPersisterImpl.java:167)
Caused by: com.amazonaws.SdkClientException: Couldn't initialize a SAX driver 
to create an XMLReader
at 
com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.(XmlResponsesSaxParser.java:118)
at 

[DISCUSS] What parts of Flink SQL should we improve for usability?

[Leonard Xu]

Hi, all

We have listened some user's compliant voice about Flink SQL usability from 
online and offline, especially about using Flink SQL to build ETL job, but it's 
not detailed. 
So, I'd like to start a discussion about Flink SQL usability, hope we can have 
more input from user side which will help improve usability more in next 
release. 


1. What's your(company’s) main business using Flink SQL to build ETL job ?

2. Which connectors are mainly used in your business ? (Kafka、JDBC、Hive、File 
System、Elasticsearch、HBase)

3. What features or improvements of Flink SQL do you think is still lacked 
according to your development experiences ?

4.  Which aspects of Flink SQL's Usability do you think are not good enough ?

5. Any other suggestions or complaints to Flink SQL ?


Any feedback is welcome.


Best,
Leonard Xu