Thanks Dian for driving this, definitely +1
Here's my 2 cents:
1. I would pay more attention on to_pandas than from_pandas. Because
to_pandas will be used more frequently I believe
2. I think ArrowTableSink may not be enough for to_pandas, because pandas
dataframe is on client side, it is not a
Hi Kurt,
It's excited to hear that the community aims to make Blink Planner default
in 1.11.
We have been using blink planner since 1.9 for streaming processing, it
works very well,
and covers many use cases in our company.
So +1 to make it default in 1.11 from our side.
Kurt Young
+1 to make blink planner as default planner.
We should give blink planner more exposure to encourage users trying out
new features and lead users to migrate to blink planner.
Glad to see blink planner is used in production since 1.9! @Benchao
Best,
Jark
On Wed, 1 Apr 2020 at 11:31, Benchao Li
Hi Jeff,
Thanks for your feedback.
ArrowTableSink is a Flink sink which is responsible for collecting the data of
the table. It will serialize the data of the table to Arrow format to make sure
that it could be deserialized to pandas dataframe efficiently. You are right
that pandas dataframe
Jingsong Lee created FLINK-16903:
Summary: Add sink.parallelism for file system factory
Key: FLINK-16903
URL: https://issues.apache.org/jira/browse/FLINK-16903
Project: Flink
Issue Type:
Hi everyone,
I'd like to start a discussion about supporting conversion between PyFlink
Table and Pandas DataFrame.
Pandas dataframe is the de-facto standard to work with tabular data in Python
community. PyFlink table is Flink’s representation of the tabular data in
Python language. It would
All issues FLINK-16891 to FLINK-16897 have been resolved, with Stephan's
PR, including the hard blocker on the missing jquery license.
I have some final licensing touches as well here (non of which are blockers
to the release, but nonetheless nice to fix if we're going have a new RC):
@Yu Li
I fixed the tagged JIRA versions while closing the issues.
On Wed, Apr 1, 2020 at 10:54 AM Tzu-Li (Gordon) Tai
wrote:
> All issues FLINK-16891 to FLINK-16897 have been resolved, with Stephan's
> PR, including the hard blocker on the missing jquery license.
>
> I have some final licensing
>> However, it seems the JobVertexID is derived from hashcode ...
You are right. JobVertexID is widely used and reworking it may affect the
public
interfaces, e.g. REST api. We can take it as a long term goal and exclude
it from this FLIP.
This same applies to IntermediateDataSetID, which can be
Hi everyone,
Thanks for the great feedbacks so far.
I updated the FLIP documentation according to the discussion. Changes
include:
- remove "version" key, and merge it into "connector"
- add "scan", "lookup", "sink" prefix to some property keys if they are
only used in that case.
- add a "New
Sounds good, I'll post a new link to this vote thread, which will have the
problem fixed in a new maven staging repository.
On Tue, Mar 31, 2020 at 2:51 PM Robert Metzger wrote:
> Thank you for looking into this.
>
> I'm fine with keeping this RC open, but re-vote on a new maven staging
>
Robert Metzger created FLINK-16878:
--
Summary: flink-table-planner contains unwanted dependency
org.apiguardian.api
Key: FLINK-16878
URL: https://issues.apache.org/jira/browse/FLINK-16878
Project:
Hi Kurt,
I also prefer "-" as version delimiter now. I didn't remove the "_"
proposal by mistake, that's why I sent another email last night :)
Regarding to "property-version", I also think we shouldn't let users to
learn about this. And ConfigOption provides a good ability
to support deprecated
Hi everyone,
I'd like to start the vote of FLIP-84[1] again, because we have some
feedbacks. The feedbacks are all about new introduced methods, here is the
discussion thread [2].
The vote will be open for at least 72 hours. Unless there is an objection,
I will try to close it by Apr 3, 2020
Thank you all for the feedback! Sorry for the belated reply.
@Till
I'm +1 for your two ideas and I'd like to move these two out of the
scope of this FLIP since the pipelined region scheduling is an ongoing
work now.
I also agree that we should not make the InstanceID in
TaskExecutorConnection
Found the culprit:
The Stateful Functions project uses the Apache POM as the parent POM, and
uses the `apache-release` build profile to build the staging jars.
The problem arises because the `apache-release` build profile itself
bundles a source release distribution to be released to Maven.
This
Jingsong Lee created FLINK-16877:
Summary: SingleDirectoryWriter should not produce file when no
input record
Key: FLINK-16877
URL: https://issues.apache.org/jira/browse/FLINK-16877
Project: Flink
While checking the release, I found a 77
MB statefun-parent-2.0.0-source-release.zip file in the maven staging repo:
https://repository.apache.org/content/repositories/orgapacheflink-1343/org/apache/flink/statefun-parent/2.0.0/
It seems that the file contains all ruby dependencies in docs/ from
Thank you for looking into this.
I'm fine with keeping this RC open, but re-vote on a new maven staging
repository.
On Tue, Mar 31, 2020 at 8:42 AM Tzu-Li (Gordon) Tai
wrote:
> Found the culprit:
>
> The Stateful Functions project uses the Apache POM as the parent POM, and
> uses the
Hi Robert,
I think you're right. There should be no tarballs / jars packaged for
statefun-parent actually, only the pom file since that's the parent module
which only has pom packaging.
I'm looking into it.
On Tue, Mar 31, 2020 at 2:23 PM Robert Metzger wrote:
> While checking the release, I
godfrey he created FLINK-16881:
--
Summary: use Catalog's total size info in planner
Key: FLINK-16881
URL: https://issues.apache.org/jira/browse/FLINK-16881
Project: Flink
Issue Type: New Feature
Hi, Zhu,
Thanks for the feedback.
> make JobVertexID a composition of JobID and a topology index
I think it is a good idea. However, it seems the JobVertexID is
derived from hashcode which used to identify them across submission.
I'm not familiar with that component though. I prefer to keep this
Hi all,
I would like to start the vote for FLIP-110 [1], which is discussed and
reached a consensus in the discussion thread [2].
The vote will be open until April 3rd (72h), unless there is an
objection or not enough votes.
Best,
Dawid
[1]
Hi Dawid,
Just two small questions:
- Can you explain more about "CONSTRAINTS, GENERATED, OPTIONS" in the FLIP?
I can image the meaning of "CONSTRAINTS, OPTIONS" in the example, but it is
hard to guess "GENERATED".
- Which category does partition keys belong to?
(I am sorry if I've disturbed the
-1
The current discussion has not completed. The last comments were sent
less than 24h ago.
Let's wait a bit longer to collect feedback from all stakeholders.
Thanks,
Timo
On 31.03.20 08:31, godfrey he wrote:
Hi everyone,
I'd like to start the vote of FLIP-84[1] again, because we have
Tzu-Li (Gordon) Tai created FLINK-16879:
---
Summary: Disable the source-release-assembly execution goal when
using the apache-release build profile
Key: FLINK-16879
URL:
Tzu-Li (Gordon) Tai created FLINK-16880:
---
Summary: Adapt Dockerfiles from flink-statefun to be added to
flink-statefun-docker
Key: FLINK-16880
URL: https://issues.apache.org/jira/browse/FLINK-16880
The container cut-off accounts for not only metaspace, but also native
memory footprint such as thread stack, code cache, compressed class space.
If you run streaming jobs with rocksdb state backend, it also accounts for
the rocksdb memory usage.
The consequence of less cut-off depends on your
=== NOTICE ===
For your testing, please continue to use this staging area for the Maven
artifacts:
https://repository.apache.org/content/repositories/orgapacheflink-1344/
The only difference between this staging repo and the original repo posted
in this thread (
Hi Jark,
Thanks for the proposal. I'm +1 since it's more simple and clear for sql
users.
I have a question about this: does this affect descriptors and related
validators?
*Best Regards,*
*Zhenghua Gao*
On Mon, Mar 30, 2020 at 2:02 PM Jark Wu wrote:
> Hi everyone,
>
> I want to start a
+1 this will reduce manual schema work a lot!
Thanks,
Timo
On 31.03.20 09:33, Dawid Wysakowicz wrote:
Hi all,
I would like to start the vote for FLIP-110 [1], which is discussed and
reached a consensus in the discussion thread [2].
The vote will be open until April 3rd (72h), unless there is
+1 I've been looking forward to it.
Best,
Jingsong Lee
On Tue, Mar 31, 2020 at 3:43 PM Timo Walther wrote:
> +1 this will reduce manual schema work a lot!
>
> Thanks,
> Timo
>
> On 31.03.20 09:33, Dawid Wysakowicz wrote:
> > Hi all,
> >
> > I would like to start the vote for FLIP-110 [1],
Hi,
Just wanted to notify the voters that after a comment from Jingsong I
introduced a new like-option in the FLIP. Because it happened very short
after the vote started I will not cancel the vote (only Timo voted
before the changed).
Feel free to change your votes if you disagree. Sorry for the
On Tue, Mar 31, 2020 at 5:57 PM Robert Metzger wrote:
> Thanks a lot Gordon!
>
> Checked:
> - files in the staging repository seem to be ok (no unexpected files,
> versions set correctly, quickstart archetype looks ok)
> - statefun-ridesharing-example-simulator-2.0.0.jar (and
>
>
Victor Wong created FLINK-16884:
---
Summary: The "rest.port" configuration is always "0" with
flink-yarn jobs
Key: FLINK-16884
URL: https://issues.apache.org/jira/browse/FLINK-16884
Project: Flink
+1 for this feature, although the WITH syntax breaks the SQL standard, but it’s
compatible with our CREATE TABLE syntax, seems well from my side.
Best,
Danny Chan
在 2020年3月31日 +0800 PM5:46，Dawid Wysakowicz ，写道：
> Hi,
>
> Just wanted to notify the voters that after a comment from Jingsong I
>
Thanks for the update, Jark. +1 for your proposal.
Some minor feedback from my side:
sink.bulk-flush -> sink.buffer-flush?
zookeeper.znode.parent -> zookeeper.znode-parent?
for consistency.
username -> secrect.username?
password -> secrect.password?
How about we prefix options that should
Thanks a lot Gordon!
Checked:
- files in the staging repository seem to be ok (no unexpected files,
versions set correctly, quickstart archetype looks ok)
- statefun-ridesharing-example-simulator-2.0.0.jar (and
For Kafka we traditionally exclude the NOTICE file since as far as we
can tell it is misleading anyway, see the flink-sql-connector-kafka modules.
@Robert for the Flink project the jquery license is in the source at
licenses/LICENSE.jquery
I'm a bit concerned just how many licensing issues
Hi,
As for the `connector.property-version`. I have doubts that having
deprecated keys mechanism is enough to fully support backwards
compatibility. I do think we are changing not just the key names, but
structure of the keys as well as the discovery mechanism and I think it
is really hard to
linweijiang created FLINK-16882:
---
Summary: NPE when reading hive table with RegexSerde
Key: FLINK-16882
URL: https://issues.apache.org/jira/browse/FLINK-16882
Project: Flink
Issue Type: Bug
Thanks for proposing this improvement Yangze. Big +1 for the overall
proposal. It can help a lot in troubleshooting.
Here are a few questions for it:
1. Shall we make JobVertexID a composition of JobID and a topology index?
This would help in the session cluster case, so that we can identify
Hi Jingsong,
I added a short description for the options:
* CONSTRAINTS: primary keys, unique key, does not include NOT NULL
constraint (in Flink it's part of the type)
* GENERATED: computed columns
* OPTIONS: connector properties in WITH (...) clause
I think partitions are a valid
Hi Timo,
I think your suggestion makes sense. I updated the document.
As there are no more comments I will start a vote for it.
Best,
Dawid
On 30/03/2020 16:40, Timo Walther wrote:
> Hi Dawid,
>
> thanks for updating the FLIP. One minor comment from my side, should
> we move the LIKE clause
Fabian Paul created FLINK-16883:
---
Summary: No support for log4j2 configuration formats besides
properties
Key: FLINK-16883
URL: https://issues.apache.org/jira/browse/FLINK-16883
Project: Flink
Hi community,
Now I am optimizing the flink 1.6 task memory configuration. I see the
source code, at first, the flink task config the cut-off memory, cut-off
memory = Math.max(600,containerized.heap-cutoff-ratio * TaskManager
Memory), containerized.heap-cutoff-ratio default value is 0.25. For
Igal Shilman created FLINK-16886:
Summary: Helm chart worker/master image name is using the wrong
template value
Key: FLINK-16886
URL: https://issues.apache.org/jira/browse/FLINK-16886
Project: Flink
Hi, Timo
So sorry about that, I'm in a little hurry. Let's wait for 24h.
Best,
Godfrey
Timo Walther 于2020年3月31日周二 下午5:26写道：
> -1
>
> The current discussion has not completed. The last comments were sent
> less than 24h ago.
>
> Let's wait a bit longer to collect feedback from all
Agree, an additional thorough licensing check would be good here.
I am doing one right now, will post the results soon...
On Tue, Mar 31, 2020 at 12:18 PM Chesnay Schepler
wrote:
> For Kafka we traditionally exclude the NOTICE file since as far as we
> can tell it is misleading anyway, see the
Hi everyone,
I would like to start a discussion thread on "Support Cython Optimizing
Python User Defined Function"
Scalar Python UDF FLIP-58[1] has already been supported in release 1.10 and
Python UDTF will be supported in the coming release of 1.11. In release
1.10, we focused on supporting
Chesnay Schepler created FLINK-16885:
Summary: SQL hive-connector wilcard excludes don't work on maven
3.1.X
Key: FLINK-16885
URL: https://issues.apache.org/jira/browse/FLINK-16885
Project: Flink
For the IntermediateDataSetID I was just thinking that it might actually be
interesting to know which job produced the result which, by using cluster
partitions, could be used across different jobs. Not saying that we have to
do it, though.
A small addition to Zhu Zhu's comment about TDD sizes:
Jark Wu created FLINK-16887:
---
Summary: Refactor retraction rules to support inferring
ChangelogMode
Key: FLINK-16887
URL: https://issues.apache.org/jira/browse/FLINK-16887
Project: Flink
Issue
Stephan Ewen created FLINK-16888:
Summary: Re-add jquery license file under "/licenses"
Key: FLINK-16888
URL: https://issues.apache.org/jira/browse/FLINK-16888
Project: Flink
Issue Type: Bug
Jark Wu created FLINK-16889:
---
Summary: Support converting BIGINT to TIMESTAMP for TO_TIMESTAMP
function
Key: FLINK-16889
URL: https://issues.apache.org/jira/browse/FLINK-16889
Project: Flink
The jquery license is in fact missing from the master/release-1.10
branches. https://issues.apache.org/jira/browse/FLINK-16888
On 31/03/2020 12:18, Chesnay Schepler wrote:
For Kafka we traditionally exclude the NOTICE file since as far as we
can tell it is misleading anyway, see the
Hi, Dawid
Regarding to `connector.property-version`,
I totally agree with you we should implicitly add a "property-version=1"
(without 'connector.' prefix) property for future evolving. So I updated
FLIP for this.
However, I still doubt to use property version to distinguish old/new
factory.
I have found a few things, am preparing a joint PR to fix them.
So far, only the missing jquery license would have been a release blocker.
On Tue, Mar 31, 2020 at 6:24 PM Chesnay Schepler wrote:
> The jquery license is in fact missing from the master/release-1.10
> branches.
+1 from my side. This will be a very useful feature.
Best,
Jark
> 2020年3月31日 18:15，Danny Chan 写道：
>
> +1 for this feature, although the WITH syntax breaks the SQL standard, but
> it’s compatible with our CREATE TABLE syntax, seems well from my side.
>
> Best,
> Danny Chan
> 在 2020年3月31日
Hi all,
I would like to start the vote for FLIP-119 [1], which is discussed and
reached a consensus in the discussion thread [2].
The vote will be open until April 3 (72h) unless there is an objection
or not enough votes.
Best,
Gary
[1]
Hi everyone,
In order to not postpone FLIP-95 further, I include the "removing
Factory#factoryVersion" in this FLIP.
I updated the "Proposed Changes" section to reflect the changes.
https://cwiki.apache.org/confluence/display/FLINK/FLIP-122%3A+New+Connector+Property+Keys+for+New+Factory
Please
Hi Godfrey,
Aljoscha, Dawid, Klou, and I had another discussion around FLIP-84. In
particular, we discussed how the current status of the FLIP and the
future requirements around multiline statements, async/sync, collect()
fit together.
We also updated the FLIP-84 Feedback Summary document
Stephan Ewen created FLINK-16892:
Summary: statefun-flink-distribution bundles many unwanted
dependencies
Key: FLINK-16892
URL: https://issues.apache.org/jira/browse/FLINK-16892
Project: Flink
Thank you Timo for the great summary! It covers (almost) all the topics.
Even though in the end we are not suggesting much changes to the current
state of FLIP I think it is important to lay out all possible use cases
so that we do not change the execution model every release.
There is one
Stephan Ewen created FLINK-16893:
Summary: Missing jquery license file in Stateful Functions
Key: FLINK-16893
URL: https://issues.apache.org/jira/browse/FLINK-16893
Project: Flink
Issue
Stephan Ewen created FLINK-16895:
Summary: Remove outdated and defunct build_docs.bat
Key: FLINK-16895
URL: https://issues.apache.org/jira/browse/FLINK-16895
Project: Flink
Issue Type:
Stephan Ewen created FLINK-16891:
Summary: Skip deployment of end-2-end test artifacts in StateFun
Key: FLINK-16891
URL: https://issues.apache.org/jira/browse/FLINK-16891
Project: Flink
Stephan Ewen created FLINK-16894:
Summary: Add copyright holder to MIT license header for
font-awesome
Key: FLINK-16894
URL: https://issues.apache.org/jira/browse/FLINK-16894
Project: Flink
Stephan Ewen created FLINK-16897:
Summary: Consolidate docs image files in "fig" folder with license
file
Key: FLINK-16897
URL: https://issues.apache.org/jira/browse/FLINK-16897
Project: Flink
Nico Kruber created FLINK-16890:
---
Summary: Add AvroGeneric benchmark
Key: FLINK-16890
URL: https://issues.apache.org/jira/browse/FLINK-16890
Project: Flink
Issue Type: New Feature
Stephan Ewen created FLINK-16896:
Summary: Remove unused components of docs system in StateFun
Key: FLINK-16896
URL: https://issues.apache.org/jira/browse/FLINK-16896
Project: Flink
Issue
Igal Shilman created FLINK-16898:
Summary: Helm chart worker deployment is using the wrong port for
lives checks
Key: FLINK-16898
URL: https://issues.apache.org/jira/browse/FLINK-16898
Project: Flink
I did a release check for license issues - all in all, we need a new RC.
The only blocker I found was the missing jquery license file.
Another somewhat critical thing is that "statefun-flink-distribution"
bundles many unwanted dependencies.
- Because the shading merges the notice files, this
Thanks for the reply, Dian, that make sense to me.
Dian Fu 于2020年4月1日周三 上午11:53写道：
> Hi Jeff,
>
> Thanks for your feedback.
>
> ArrowTableSink is a Flink sink which is responsible for collecting the
> data of the table. It will serialize the data of the table to Arrow format
> to make sure that
Jingsong Lee created FLINK-16904:
Summary: Migrate FileSystem descriptor to new table factory
Key: FLINK-16904
URL: https://issues.apache.org/jira/browse/FLINK-16904
Project: Flink
Issue
Igal Shilman created FLINK-16899:
Summary: Adjust module.yaml of python-k8s example to match
python-worker-service
Key: FLINK-16899
URL: https://issues.apache.org/jira/browse/FLINK-16899
Project:
Tzu-Li (Gordon) Tai created FLINK-16900:
---
Summary: Exclude NOTCE and LICENSE files of Kafka dependencies for
statefun-flink-distribution
Key: FLINK-16900
URL:
Hi Dev and User,
Blink planner for Table API & SQL is introduced in Flink 1.9 and already be
the default planner for
SQL client in Flink 1.10. And since we already decided not introducing any
new features to the
original Flink planner, it already lacked of so many great features that
the
Tzu-Li (Gordon) Tai created FLINK-16901:
---
Summary: Flink Kinesis connector NOTICE should have contents of
AWS KPL's THIRD_PARTY_NOTICES file manually merged in
Key: FLINK-16901
URL:
Hi Danny,
If I haven't missed anything, I don't know how to support the dynamic
options of format.
Connector don't know format information in TableFactory before obtains real
properties, so it can not list any format `supportedHintOptions`.
And I think it is important for support format dynamic
Tzu-Li (Gordon) Tai created FLINK-16902:
---
Summary: commons-codec dependency pulls in LICENSE.txt that should
be excluded
Key: FLINK-16902
URL: https://issues.apache.org/jira/browse/FLINK-16902
Hi Stephan,
Could you also share the method of license check, so more people could help
in future votes? And maybe adding some instructions into our wiki [1]? I
skipped the licensing check in my vote because not aware of a good way to
do it thoroughly, not sure whether I'm the only one having
82 matches
Mail list logo
