[jira] [Created] (FLINK-21173) "Streaming SQL end-to-end test (Old planner)" e2e test failed

2021-01-27 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-21173: - Summary: "Streaming SQL end-to-end test (Old planner)" e2e test failed Key: FLINK-21173 URL: https://issues.apache.org/jira/browse/FLINK-21173 Proj

[jira] [Created] (FLINK-21157) "Running HA per-job cluster (rocks, incremental) end-to-end test" gets stuck

2021-01-26 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-21157: - Summary: "Running HA per-job cluster (rocks, incremental) end-to-end test" gets stuck Key: FLINK-21157 URL: https://issues.apache.org/jira/browse/FLINK-21157

[jira] [Created] (FLINK-21156) org.apache.flink.connector.jdbc.catalog.PostgresCatalogITCase Fail

2021-01-26 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-21156: - Summary: org.apache.flink.connector.jdbc.catalog.PostgresCatalogITCase Fail Key: FLINK-21156 URL: https://issues.apache.org/jira/browse/FLINK-21156 Project: Flink

Re: [ANNOUNCE] Apache Flink 1.12.1 released

2021-01-19 Thread Guowei Ma
Thanks Xintong's effort! Best, Guowei On Tue, Jan 19, 2021 at 5:37 PM Yangze Guo wrote: > Thanks Xintong for the great work! > > Best, > Yangze Guo > > On Tue, Jan 19, 2021 at 4:47 PM Till Rohrmann > wrote: > > > > Thanks a lot for driving this release Xintong. This was indeed a release >

Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished

2021-01-18 Thread Guowei Ma
+1 non-binding Best, Guowei On Fri, Jan 15, 2021 at 10:56 PM Yun Gao wrote: > > Hi all, > > I would like to start the vote for FLIP-147[1], which propose to support > checkpoints after > tasks finished and is discussed in [2]. > > The vote will last at least 72 hours (Jan 20th due to weekend),

Re: [DISCUSS] Releasing Apache Flink 1.12.1

2020-12-17 Thread Guowei Ma
Thanks for driving this release Xintong. I think https://issues.apache.org/jira/browse/FLINK-20652 should be addressed. Best, Guowei On Fri, Dec 18, 2020 at 11:53 AM Jingsong Li wrote: > Thanks for volunteering as our release manager Xintong. +1 for releasing > Flink 1.12.1 soon. > > I think

[jira] [Created] (FLINK-20652) Improve the document for making the user could write a DataStream job that could be execute in the batch execution mode.

2020-12-17 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-20652: - Summary: Improve the document for making the user could write a DataStream job that could be execute in the batch execution mode. Key: FLINK-20652 URL: https://issues.apache.org/jira

Re: [ANNOUNCE] Apache Flink 1.12.0 released

2020-12-13 Thread Guowei Ma
Congratulations to everyone who contributed! Thanks for being our release managers Dian and Robert! Best, Guowei On Mon, Dec 14, 2020 at 11:41 AM Yang Wang wrote: > Congratulations! > Thanks Robert and Dian for being our release manager and everyone who > contributed. > > > Best, > Yang > >

Re: [VOTE] Release 1.12.0, release candidate #3

2020-12-06 Thread Guowei Ma
+1(non-binding) - build from source - build a docker image - start a session from local k8s cluster - submit a wordcount job in streaming mode. - submit a wordcount job in batch mode. Best, Guowei On Sat, Dec 5, 2020 at 3:13 PM Zhu Zhu wrote: > +1 (binding) > > - verified signature and

[jira] [Created] (FLINK-20339) `FileWriter` support to load StreamingFileSink's state.

2020-11-24 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-20339: - Summary: `FileWriter` support to load StreamingFileSink's state. Key: FLINK-20339 URL: https://issues.apache.org/jira/browse/FLINK-20339 Project: Flink Issue Type

[jira] [Created] (FLINK-20338) Make the SinkWriter load previous sink's state.

2020-11-24 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-20338: - Summary: Make the SinkWriter load previous sink's state. Key: FLINK-20338 URL: https://issues.apache.org/jira/browse/FLINK-20338 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-20337) Make migrate `StreamingFileSink` to `FileSink` possible

2020-11-24 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-20337: - Summary: Make migrate `StreamingFileSink` to `FileSink` possible Key: FLINK-20337 URL: https://issues.apache.org/jira/browse/FLINK-20337 Project: Flink Issue Type

[jira] [Created] (FLINK-20297) Make `SerializerTestBase::getTestData` return List

2020-11-23 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-20297: - Summary: Make `SerializerTestBase::getTestData` return List Key: FLINK-20297 URL: https://issues.apache.org/jira/browse/FLINK-20297 Project: Flink Issue Type

[jira] [Created] (FLINK-20225) Break link in the document

2020-11-18 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-20225: - Summary: Break link in the document Key: FLINK-20225 URL: https://issues.apache.org/jira/browse/FLINK-20225 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-20215) Keep the "Access the Flink UI " document same in “Kubernetes Setup” and "Native Kubernetes Setup Beta"

2020-11-18 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-20215: - Summary: Keep the "Access the Flink UI " document same in “Kubernetes Setup” and "Native Kubernetes Setup Beta" Key: FLINK-20215 URL: https://issues.apache.org/jir

[jira] [Created] (FLINK-20214) Unnecessary warning log when starting a k8s session cluster

2020-11-18 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-20214: - Summary: Unnecessary warning log when starting a k8s session cluster Key: FLINK-20214 URL: https://issues.apache.org/jira/browse/FLINK-20214 Project: Flink Issue

[jira] [Created] (FLINK-20211) Can not get the JobManager web ip according to the document

2020-11-18 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-20211: - Summary: Can not get the JobManager web ip according to the document Key: FLINK-20211 URL: https://issues.apache.org/jira/browse/FLINK-20211 Project: Flink Issue

[jira] [Created] (FLINK-20206) Failed to start the session, but there is no clear prompt.

2020-11-17 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-20206: - Summary: Failed to start the session, but there is no clear prompt. Key: FLINK-20206 URL: https://issues.apache.org/jira/browse/FLINK-20206 Project: Flink Issue

[jira] [Created] (FLINK-20203) Could not find any document about how to build a Flink image from local build.

2020-11-17 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-20203: - Summary: Could not find any document about how to build a Flink image from local build. Key: FLINK-20203 URL: https://issues.apache.org/jira/browse/FLINK-20203 Project

[jira] [Created] (FLINK-20031) Keep the uid of SinkWriter same as the SinkTransformation

2020-11-06 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-20031: - Summary: Keep the uid of SinkWriter same as the SinkTransformation Key: FLINK-20031 URL: https://issues.apache.org/jira/browse/FLINK-20031 Project: Flink Issue

[jira] [Created] (FLINK-20007) SinkTransformationTranslator fail to handle the PartitionTransformation

2020-11-05 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-20007: - Summary: SinkTransformationTranslator fail to handle the PartitionTransformation Key: FLINK-20007 URL: https://issues.apache.org/jira/browse/FLINK-20007 Project: Flink

[jira] [Created] (FLINK-19963) Let the `SinkWriter` support using the `TimerService`

2020-11-04 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-19963: - Summary: Let the `SinkWriter` support using the `TimerService` Key: FLINK-19963 URL: https://issues.apache.org/jira/browse/FLINK-19963 Project: Flink Issue Type

[jira] [Created] (FLINK-19958) Unified exception signature in Sink API

2020-11-03 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-19958: - Summary: Unified exception signature in Sink API Key: FLINK-19958 URL: https://issues.apache.org/jira/browse/FLINK-19958 Project: Flink Issue Type: Sub-task

Re: [ANNOUNCE] New Apache Flink Committer - Congxian Qiu

2020-11-03 Thread Guowei Ma
Congratulations! Congxian Best, Guowei On Wed, Nov 4, 2020 at 10:36 AM godfrey he wrote: > Congratulations! Congxian > > Best, > Godfrey > > Fabian Hueske 于2020年11月2日周一 下午7:00写道: > > > Congrats Congxian! > > > > Cheers, Fabian > > > > Am Mo., 2. Nov. 2020 um 10:33 Uhr schrieb Yang Wang < >

[jira] [Created] (FLINK-19936) Make SinkITCase more stable

2020-11-02 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-19936: - Summary: Make SinkITCase more stable Key: FLINK-19936 URL: https://issues.apache.org/jira/browse/FLINK-19936 Project: Flink Issue Type: Sub-task

Re: [VOTE] Remove flink-connector-filesystem module.

2020-10-30 Thread Guowei Ma
+1 Best, Guowei On Fri, Oct 30, 2020 at 6:02 PM Aljoscha Krettek wrote: > +1 > > Aljoscha > > On 29.10.20 09:18, Kostas Kloudas wrote: > > Hi all, > > > > Following the discussion in [1], I would like to start a vote on > > removing the flink-connector-filesystem module which includes the > >

[jira] [Created] (FLINK-19841) Rename GlobalStreamingCommitterOperator to StreamingGlobalCommitterOperator

2020-10-27 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-19841: - Summary: Rename GlobalStreamingCommitterOperator to StreamingGlobalCommitterOperator Key: FLINK-19841 URL: https://issues.apache.org/jira/browse/FLINK-19841 Project: Flink

[jira] [Created] (FLINK-19836) Serialize the committable by the serializer provided by the user during network shuffle

2020-10-27 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-19836: - Summary: Serialize the committable by the serializer provided by the user during network shuffle Key: FLINK-19836 URL: https://issues.apache.org/jira/browse/FLINK-19836

[jira] [Created] (FLINK-19834) Make the TestSink reusable in all the sink related tests.

2020-10-27 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-19834: - Summary: Make the TestSink reusable in all the sink related tests. Key: FLINK-19834 URL: https://issues.apache.org/jira/browse/FLINK-19834 Project: Flink Issue

[jira] [Created] (FLINK-19760) Make the `GlobalCommitter` a standalone interface that does not extend the `Committer`

2020-10-22 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-19760: - Summary: Make the `GlobalCommitter` a standalone interface that does not extend the `Committer` Key: FLINK-19760 URL: https://issues.apache.org/jira/browse/FLINK-19760

[jira] [Created] (FLINK-19738) Remove the `getCommitter` from the `AbstractStreamingCommitterOperator`

2020-10-20 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-19738: - Summary: Remove the `getCommitter` from the `AbstractStreamingCommitterOperator` Key: FLINK-19738 URL: https://issues.apache.org/jira/browse/FLINK-19738 Project: Flink

[jira] [Created] (FLINK-19736) Implement the `SinkTransformation`

2020-10-20 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-19736: - Summary: Implement the `SinkTransformation` Key: FLINK-19736 URL: https://issues.apache.org/jira/browse/FLINK-19736 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-19697) Make the streaming committer retry-able

2020-10-18 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-19697: - Summary: Make the streaming committer retry-able Key: FLINK-19697 URL: https://issues.apache.org/jira/browse/FLINK-19697 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-19586) Implement the stream committer operator

2020-10-12 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-19586: - Summary: Implement the stream committer operator Key: FLINK-19586 URL: https://issues.apache.org/jira/browse/FLINK-19586 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-19531) Implement the `WriterOperator`

2020-10-07 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-19531: - Summary: Implement the `WriterOperator` Key: FLINK-19531 URL: https://issues.apache.org/jira/browse/FLINK-19531 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-19512) Introduce the new sink api

2020-10-06 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-19512: - Summary: Introduce the new sink api Key: FLINK-19512 URL: https://issues.apache.org/jira/browse/FLINK-19512 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-19511) Rename the `SinkTransformation` to `LegacySinkTransformation`

2020-10-06 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-19511: - Summary: Rename the `SinkTransformation` to `LegacySinkTransformation` Key: FLINK-19511 URL: https://issues.apache.org/jira/browse/FLINK-19511 Project: Flink

[jira] [Created] (FLINK-19510) Introduce new unified sink api

2020-10-06 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-19510: - Summary: Introduce new unified sink api Key: FLINK-19510 URL: https://issues.apache.org/jira/browse/FLINK-19510 Project: Flink Issue Type: New Feature

Re: [VOTE] FLIP-143: Unified Sink API

2020-10-05 Thread Guowei Ma
, 2020 at 10:11 AM Guowei Ma wrote: > Hi all, > > The voting time for FLIP-143 [1] has passed. I'm closing the vote now. > > There were 5 votes, 4 of which are binding: > > - Aljoscha Krettek (binding) > - Steven Wu > - Kostas Kloudas (binding) > - Jingsong Li (bind

Re: Re: [ANNOUNCE] New PMC member: Zhu Zhu

2020-10-05 Thread Guowei Ma
Congratulations Zhu Zhu! Best, Guowei On Tue, Oct 6, 2020 at 5:47 AM Fabian Hueske wrote: > Congrats and welcome Zhu! > > Best, Fabian > > Am Mo., 5. Okt. 2020 um 14:33 Uhr schrieb Yun Gao > : > > > Congratulations Zhu! > > > > Best, > >

Re: [VOTE] FLIP-143: Unified Sink API

2020-09-29 Thread Guowei Ma
2020 at 7:56 PM Steven Wu > wrote: > > > > > > > > +1 (non-binding) > > > > > > > > Although I would love to continue the discussion for tweaking the > > > > CommitResult/GlobaCommitter interface maybe during the implementation > > > phase.

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-26 Thread Guowei Ma
items. we > don't > > want to commit them one by one. It is faster to commit the whole list as > > one batch. > > > > > 3. On the other hand, I think only when restoring IcebergSink needs a > > > collection of `GlobalCommT` and giving back another collection o

Re: [VOTE] FLIP-143: Unified Sink API

2020-09-25 Thread Guowei Ma
venience. [1] https://lists.apache.org/thread.html/rf09dfeeaf35da5ee98afe559b5a6e955c9f03ade0262727f6b5c4c1e%40%3Cdev.flink.apache.org%3E [2] https://cwiki.apache.org/confluence/x/KEJ4CQ Best, Guowei On Thu, Sep 24, 2020 at 8:13 PM Guowei Ma wrote: > Hi, all > > After the discussio

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-25 Thread Guowei Ma
] here. Sorry for the inconvenience. [1]https://cwiki.apache.org/confluence/x/KEJ4CQ Best, Guowei On Fri, Sep 25, 2020 at 3:26 PM Guowei Ma wrote: > Hi, Steven > > >>I also have a clarifying question regarding the WriterStateT. Since > >>IcebergWriter won't need to checkpoint any state,

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-25 Thread Guowei Ma
Yes I think you could do it. If you return Optional.empty() we would ignore all the state you return. Best, Guowei On Fri, Sep 25, 2020 at 3:14 PM Guowei Ma wrote: > Hi,Steven > > Thank you for reading the FLIP so carefully. > 1. The frame can not know which `GlobalCommT` to retry if

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-25 Thread Guowei Ma
ype? Since getWriterStateSerializer() returns Optional, that is clear and > we can return Optional.empty(). > > Thanks, > Steven > > On Wed, Sep 23, 2020 at 6:59 PM Guowei Ma wrote: > > > Thanks Aljoscha for your suggestion. I have updated FLIP. Any comments > are > > welc

[VOTE] FLIP-143: Unified Sink API

2020-09-24 Thread Guowei Ma
Hi, all After the discussion in [1], I would like to open a voting thread for FLIP-143 [2], which proposes a unified sink api. The vote will be open until September 29th (72h + weekend), unless there is an objection or not enough votes. [1]

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-23 Thread Guowei Ma
vote. > > Best, > Aljoscha > > On 23.09.20 06:13, Guowei Ma wrote: > > Hi, all > > > > Thank everyone very much for your ideas and suggestions. I would try to > > summarize again the consensus :). Correct me if I am wrong or > misunderstand > > yo

[jira] [Created] (FLINK-19375) The `kubernetes.secrets` and `kubernetes.env.secretKeyRef`'s document is missing

2020-09-23 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-19375: - Summary: The `kubernetes.secrets` and `kubernetes.env.secretKeyRef`'s document is missing Key: FLINK-19375 URL: https://issues.apache.org/jira/browse/FLINK-19375 Project

[jira] [Created] (FLINK-19374) update the `table.exec.state.ttl`'s documentation

2020-09-23 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-19374: - Summary: update the `table.exec.state.ttl`'s documentation Key: FLINK-19374 URL: https://issues.apache.org/jira/browse/FLINK-19374 Project: Flink Issue Type: Task

[jira] [Created] (FLINK-19373) `jmx.server.port`'s document is missing.

2020-09-23 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-19373: - Summary: `jmx.server.port`'s document is missing. Key: FLINK-19373 URL: https://issues.apache.org/jira/browse/FLINK-19373 Project: Flink Issue Type: Bug

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-22 Thread Guowei Ma
3, 2020 at 12:03 PM Guowei Ma wrote: > >> I think we should go with something like > > >> List filterRecoveredCommittables(List<>) > > >> to keep things simple. This should also be easy to do from the framework > >> side and then the sink doesn't

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-22 Thread Guowei Ma
successful so that the next checkpoint won't include the > >> committed GlobalCommT. > >> > >> Maybe GlobalCommitter can have an API like this? > >>> List snapshotState(); > >> > >> But then we still need the recover API if we don't let sink direct

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-22 Thread Guowei Ma
am not sure adding `isAvailable` is enough. Maybe it is not. But for the initial version I hope we could make the sink api sync because there is already a lot of stuff that has to finish. :--) What do you think? Best, Guowei On Tue, Sep 22, 2020 at 5:25 PM Aljoscha Krettek wrote: > On 22.09.20 1

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-22 Thread Guowei Ma
Thanks @aljoscha summary. I agree we should postpone the discussion of the sink topology first and focus on the normal file sink and IcebergSink in the Flink 1.12. I have three little questions: 1. I think maybe we could add a EOI interface to the `GlobalCommit`. So that we could make `write

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-20 Thread Guowei Ma
sformation createWriteTopology(); CommitFunction createCommitter(); } Best, Guowei On Sun, Sep 20, 2020 at 6:09 PM Guowei Ma wrote: > Hi, Stevn > I want to make a clarification first, the following reply only considers > the Iceberge sink, but does not consider other sinks. Before

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-20 Thread Guowei Ma
er (e.g. like sink coordinator) >2. help with checkpointing, bookkeeping, commit failure handling, >recovery > > > @Guowei Ma regarding the GlobalCommitter > interface, I have some clarifying questions. > > > void recoveredGlobalCommittables(List globalCommits) > >

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-19 Thread Guowei Ma
Hi, all >>Just to add to what Aljoscha said regarding the unique id. Iceberg sink >>checkpoints the unique id into state during snapshot. It also inserts the >>unique id into the Iceberg snapshot metadata during commit. When a job >>restores the state after failure, it needs to know if the

Re: Re: [ANNOUNCE] Apache Flink 1.11.2 released

2020-09-18 Thread Guowei Ma
Thanks Zhuzhu for driving the release!!! Best, Guowei On Fri, Sep 18, 2020 at 5:10 PM Yun Gao wrote: > Great! Very thanks @ZhuZhu for driving this and thanks for all contributed > to the release! > > Best, > Yun > > --Original Mail -- > *Sender:*Jingsong Li >

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-16 Thread Guowei Ma
osing, ...). If we > > make sinks non blocking (as is the network stack in the most part and as > > are the FLIP-27 sources), we will be able to snapshot state of the > operator > > immediately. For example, change from blocking to non blocking sources > was > > sp

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-16 Thread Guowei Ma
ill be able to snapshot state of the operator > immediately. For example, change from blocking to non blocking sources was > speeding up unaligned checkpoints from ~30seconds down to ~5seconds in > our benchmarks, but the difference can be even more profound (hours instead > of seconds/

[jira] [Created] (FLINK-19261) Update document according to RestOptions

2020-09-16 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-19261: - Summary: Update document according to RestOptions Key: FLINK-19261 URL: https://issues.apache.org/jira/browse/FLINK-19261 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-19260) Update documentation based on bin/flink output

2020-09-16 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-19260: - Summary: Update documentation based on bin/flink output Key: FLINK-19260 URL: https://issues.apache.org/jira/browse/FLINK-19260 Project: Flink Issue Type: Bug

Re: Re: [VOTE] FLIP-134: Batch execution for the DataStream API

2020-09-16 Thread Guowei Ma
+1 (non-binding) Best, Guowei On Wed, Sep 16, 2020 at 3:03 PM Yun Gao wrote: > +1 (non-binding) > > Very thanks for bring this up! And the FLIP is indeed necessary for stream > & batch unification. > > -- > Sender:Dawid

Re: [ANNOUNCE] New Apache Flink Committer - Niels Basjes

2020-09-15 Thread Guowei Ma
Congratulations :) Best, Guowei On Tue, Sep 15, 2020 at 6:14 PM Matthias Pohl wrote: > Congrats! > > Best, > Matthias > > On Tue, Sep 15, 2020 at 9:26 AM Dawid Wysakowicz > wrote: > > > Welcome, Niels! > > > > Best, > > > > Dawid > > > > On 14/09/2020 11:22, Matt Wang wrote: > > >

Re: [ANNOUNCE] New Apache Flink Committer - Arvid Heise

2020-09-15 Thread Guowei Ma
Congratulations :) Best, Guowei On Tue, Sep 15, 2020 at 6:41 PM 刘建刚 wrote: > Congratulations! > > Best > > Matthias Pohl 于2020年9月15日周二 下午6:07写道: > > > Congratulations! ;-) > > > > On Tue, Sep 15, 2020 at 11:47 AM Xingbo Huang > wrote: > > > > > Congratulations! > > > > > > Best, > > >

Re: [ANNOUNCE] New Apache Flink Committer - Yun Tang

2020-09-15 Thread Guowei Ma
Congratulations :) Best, Guowei On Wed, Sep 16, 2020 at 11:54 AM Zhijiang wrote: > Congratulations and welcome, Yun! > > > -- > From:Jark Wu > Send Time:2020年9月16日(星期三) 11:35 > To:dev > Cc:tangyun ; Yun Tang > Subject:Re:

Re: Re: [ANNOUNCE] New Apache Flink Committer - Igal Shilman

2020-09-15 Thread Guowei Ma
Congratulations :) Best, Guowei On Wed, Sep 16, 2020 at 11:54 AM Zhijiang wrote: > Congratulations and welcome, Igal! > > > -- > From:Yun Gao > Send Time:2020年9月16日(星期三) 10:59 > To:Stephan Ewen ; dev > Subject:Re: Re: [ANNOUNCE]

Re: [ANNOUNCE] New Apache Flink Committer - Godfrey He

2020-09-15 Thread Guowei Ma
Congratulations :) Best, Guowei On Wed, Sep 16, 2020 at 12:19 PM Jark Wu wrote: > Hi everyone, > > It's great seeing many new Flink committers recently, and on behalf of the > PMC, > I'd like to announce one more new committer: Godfrey He. > > Godfrey is a very long time contributor in the

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-15 Thread Guowei Ma
Wed, Sep 16, 2020 at 12:02 PM Guowei Ma wrote: > > Hi, Steven > Thanks you for your thoughtful ideas and concerns. > > >>I still like the concept of grouping data files per checkpoint for > streaming mode. it is cleaner and probably easier to manage and deal with > commit

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-15 Thread Guowei Ma
ot see the file-1-2 when ck1 happens in the > ExactlyOnce mode. > > @Guowei Ma I think you are right for exactly once > checkpoint semantics. what about "at least once"? I guess we can argue that > it is fine to commit file-1-2 for at least once mode. > > I still lik

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-15 Thread Guowei Ma
ting the > example. > > I've just briefly skimmed over the proposed interfaces. I would suggest one > addition to the Writer interface (as I understand this is the runtime > interface in this proposal?): add some availability method, to avoid, if > possible, blocking calls on the sink. W

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-15 Thread Guowei Ma
>> I would think that we only need flush() and the semantics are that it >> prepares for a commit, so on a physical level it would be called from >> "prepareSnapshotPreBarrier". Now that I'm thinking about it more I >> think flush() should be renamed to something like "prepareCommit()". >

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-14 Thread Guowei Ma
but I do not get why the second alternative would depend on `flink-runtime` or `flink-streaming-java`. The all the state api currently is in the flink-core. Could you give some further explanation? thanks :) Best, Guowei On Tue, Sep 15, 2020 at 12:05 PM Guowei Ma wrote: > ## Concurrent c

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-14 Thread Guowei Ma
gt; > > A alternative topology option for the IcebergSink might be : > DataFileWriter > -> Agg -> GlobalCommitter. One pro of this method is that we can let Agg > take care of the cleanup instead of coupling the cleanup logic to the > committer. > > @Guowei Ma I wou

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-14 Thread Guowei Ma
Hi all, Very thanks for the discussion and the valuable opinions! Currently there are several ongoing issues and we would like to show what we are thinking in the next few mails. It seems that the biggest issue now is about the topology of the sinks. Before deciding what the sink API would look

[DISCUSS] FLIP-143: Unified Sink API

2020-09-10 Thread Guowei Ma
Hi, devs & users As discussed in FLIP-131[1], Flink will deprecate the DataSet API in favor of DataStream API and Table API. Users should be able to use DataStream API to write jobs that support both bounded and unbounded execution modes. However, Flink does not provide a sink API to guarantee

Re: [VOTE] FLIP-131: Consolidate the user-facing Dataflow SDKs/APIs (and deprecate the DataSet API)

2020-09-03 Thread Guowei Ma
+1 Looking forward to having a unified datastream api. Best, Guowei On Thu, Sep 3, 2020 at 3:46 PM Dawid Wysakowicz wrote: > +1 > > I think it gives a clear idea why we should deprecate and eventually > remove the DataSet API. > > Best, > > Dawid > > On 03/09/2020 09:37, Yun Gao wrote: > >

Re: [ANNOUNCE] Apache Flink 1.10.2 released

2020-08-25 Thread Guowei Ma
Hi, Thanks a lot for being the release manager Zhu Zhu! Thanks everyone contributed to this! Best, Guowei On Wed, Aug 26, 2020 at 11:18 AM Yun Tang wrote: > Thanks for Zhu's work to manage this release and everyone who contributed > to this! > > Best, > Yun Tang >

Re: [DISCUSS] FLIP-134: DataStream Semantics for Bounded Input

2020-08-24 Thread Guowei Ma
Hi, Klou Thanks for your proposal. It's a very good idea. Just a little comment about the "Batch vs Streaming Scheduling". In the AUTOMATIC execution mode maybe we could not pick BATCH execution mode even if all sources are bounded. For example some applications would use the

Re: [ANNOUNCE] New PMC member: Piotr Nowojski

2020-07-07 Thread Guowei Ma
Congratulations! Best, Guowei Fabian Hueske 于2020年7月7日周二 下午4:34写道: > Congrats Piotr! > > Cheers, Fabian > > Am Di., 7. Juli 2020 um 10:04 Uhr schrieb Marta Paes Moreira < > ma...@ververica.com>: > > > Go Piotr! Congrats! > > > > On Tue, Jul 7, 2020 at 7:15 AM Biao Liu wrote: > > > > >

Re: [ANNOUNCE] Yu Li is now part of the Flink PMC

2020-06-17 Thread Guowei Ma
Congratulations , Yu! Best, Guowei Yang Wang 于2020年6月18日周四 上午10:36写道: > Congratulations , Yu! > > Best, > Yang > > Piotr Nowojski 于2020年6月17日周三 下午9:21写道: > > > Congratulations :) > > > > > On 17 Jun 2020, at 14:53, Yun Tang wrote: > > > > > > Congratulations , Yu! well deserved. > > > > > >

Re: NoSuchMethodError: org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.(Ljava/lang/String;Lorg/apache/flink/fs/s3presto/common/HadoopConfigLoader

2020-06-09 Thread Guowei Ma
On Tue, Jun 9, 2020 at 9:46 PM Guowei Ma wrote: > >> Hi, >> In 1.10 there is no >> 'Lorg/apache/flink/fs/s3presto/common/HadoopConfigLoader' . So I think >> there might be a legacy S3FileSystemFactory in your jar. You could check >> whether there is a 'org.ap

Re: [ANNOUNCE] New Apache Flink Committer - Xintong Song

2020-06-07 Thread Guowei Ma
Congratulations Best, Guowei Matt Wang 于2020年6月6日周六 上午9:56写道: > Congratulations! > > > --- > Best, > Matt Wang > > > On 06/5/2020 22:34,Andrey Zagrebin wrote: > Welcome to committers and congrats, Xintong! > > Cheers, > Andrey > > On Fri, Jun 5, 2020 at 4:22 PM Till Rohrmann wrote: > >

Re: [DISCUSS] FLINK-17989 - java.lang.NoClassDefFoundError org.apache.flink.fs.azure.common.hadoop.HadoopRecoverableWriter

2020-05-27 Thread Guowei Ma
Hi, I think the StreamingFileSink could not support Azure currently. You could find more detailed info from here[1]. [1] https://issues.apache.org/jira/browse/FLINK-17444 Best, Guowei Israel Ekpo 于2020年5月28日周四 上午6:04写道: > You can assign the task to me and I will like to collaborate with

Re: java.lang.NoSuchMethodError while writing to Kafka from Flink

2020-05-24 Thread Guowei Ma
Hi 1. You could check whether the 'org.apache.flink.api.java.clean' is in your classpath first. 2. Do you follow the doc[1] to deploy your local cluster and run some existed examples such as WordCount? [1]

Re: [ANNOUNCE] New Apache Flink PMC Member - Hequn Chen

2020-04-17 Thread Guowei Ma
Congratulations! Best, Guowei Xintong Song 于2020年4月17日周五 下午5:21写道: > Congratulations, Hequn~! > > Thank you~ > > Xintong Song > > > > On Fri, Apr 17, 2020 at 4:50 PM godfrey he wrote: > > > Congratulations, Hequn! > > > > Best, > > Godfrey > > > > Leonard Xu 于2020年4月17日周五 下午4:30写道: > > > >

Re: [ANNOUNCE] New Committers and PMC member

2020-04-07 Thread Guowei Ma
Congratulations! Best, Guowei Dawid Wysakowicz 于2020年4月7日周二 下午2:50写道: > Thank you all for the support! > > Best, > > Dawid > > On 02/04/2020 04:33, godfrey he wrote: > > Congratulations to all of you~ > > > > Best, > > Godfrey > > > > Ismaël Mejía 于2020年4月2日周四 上午6:42写道: > > > >> Congrats

[jira] [Created] (FLINK-17041) Migrate current TypeInformation creation to the TypeInformationExtractor framework

2020-04-07 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-17041: - Summary: Migrate current TypeInformation creation to the TypeInformationExtractor framework Key: FLINK-17041 URL: https://issues.apache.org/jira/browse/FLINK-17041 Project

[jira] [Created] (FLINK-17039) Introduce TypeInformationExtractor interface

2020-04-07 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-17039: - Summary: Introduce TypeInformationExtractor interface Key: FLINK-17039 URL: https://issues.apache.org/jira/browse/FLINK-17039 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-17038) Decouple resolving Type and creating TypeInformation process

2020-04-07 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-17038: - Summary: Decouple resolving Type and creating TypeInformation process Key: FLINK-17038 URL: https://issues.apache.org/jira/browse/FLINK-17038 Project: Flink

Re: [DISCUSS] FLIP-115: Filesystem connector in Table

2020-03-19 Thread Guowei Ma
Hi, I am very interested in the topic. I would like to join the offline discussion if possible. I think you guys already give many inputs and concerns. I would share some of my thoughts. Correct me if I misunderstand you. Flink is a unified engine. Since that Flink should provide the e2e

Re: [DISCUSS] Drop Bucketing Sink

2020-03-13 Thread Guowei Ma
+1 to drop it. To Jingsong : we are planning to implement the orc StreamingFileSink in 1.11. I think users also could reference the old BucktSink from the old version. Best, Guowei Jingsong Li 于2020年3月13日周五 上午10:07写道: > Hi Robert, > > +1 to drop it but maybe not 1.11. > > ORC has not been

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

2020-02-26 Thread Guowei Ma
Congratulations !! Best, Guowei Yun Tang 于2020年2月27日周四 上午2:11写道: > Congratulations and well deserved! > > > Best > Yun Tang > > From: Canbin Zheng > Sent: Monday, February 24, 2020 16:07 > To: dev > Subject: Re: [ANNOUNCE] Jingsong Lee becomes a Flink

Re: [VOTE] FLIP-27 - Refactor Source Interface

2020-02-03 Thread Guowei Ma
+1 (non-binding), thanks for driving. Best, Guowei Jingsong Li 于2020年2月4日周二 上午11:20写道: > +1 (non-binding), thanks for driving. > FLIP-27 is the basis of a lot of follow-up work. > > Best, > Jingsong Lee > > On Tue, Feb 4, 2020 at 10:26 AM Jark Wu wrote: > > > Thanks for driving this Becket!

[jira] [Created] (FLINK-15786) Load connector code with separate classloader

2020-01-28 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-15786: - Summary: Load connector code with separate classloader Key: FLINK-15786 URL: https://issues.apache.org/jira/browse/FLINK-15786 Project: Flink Issue Type

[jira] [Created] (FLINK-15784) Show the Watermark metrics of source node in the web UI.

2020-01-28 Thread Guowei Ma (Jira)
Guowei Ma created FLINK-15784: - Summary: Show the Watermark metrics of source node in the web UI. Key: FLINK-15784 URL: https://issues.apache.org/jira/browse/FLINK-15784 Project: Flink Issue

Re: Understanding watermark

2020-01-19 Thread Guowei Ma
e doesn't have input watermark show it show > "No Watermark" ? And we should have output watermark for source? > > And, yes we want to understand when we should expect to see watermarks for > our "combined" sources (bounded and un-bounded) for our pipeline? > > If

Re: Understanding watermark

2020-01-14 Thread Guowei Ma
Hi, Cam, I think you might want to know why the web page does not show the watermark of the source. Currently, the web only shows the "input" watermark. The source only outputs the watermark so the web shows you that there is "No Watermark". Actually Flink has "output" watermark metrics. I think

Re: [ANNOUNCE] Zhu Zhu becomes a Flink committer

2019-12-15 Thread Guowei Ma
Congrats Zhuzhu! Best, Guowei Zhenghua Gao 于2019年12月16日周一 上午10:47写道: > Congrats! > > *Best Regards,* > *Zhenghua Gao* > > > On Mon, Dec 16, 2019 at 10:36 AM Biao Liu wrote: > > > Congrats Zhu Zhu! > > > > Thanks, > > Biao /'bɪ.aʊ/ > > > > > > > > On Mon, 16 Dec 2019 at 10:23, Congxian Qiu >

<    1   2   3   4   >