Re: Re: [ANNOUNCE] Welcome Roman Khachatryan a new Apache Flink Committer

2021-02-10 Thread Kostas Kloudas
Congrats Roman! Kostas On Wed, Feb 10, 2021 at 2:08 PM Arvid Heise wrote: > > Congrats! Well deserved. > > On Wed, Feb 10, 2021 at 1:54 PM Yun Gao > wrote: > > > Congratulations Roman! > > > > Best, > > Yun > > > > > > --Original Mail -- > > Sender:Till

Re: [DISCUSS] FLIP-157 Migrate Flink Documentation from Jekyll to Hugo

2021-01-14 Thread Kostas Kloudas
+1 for moving to Hugo. Cheers, Kostas On Thu, Jan 14, 2021 at 1:27 PM Wei Zhong wrote: > > +1 for migrating to Hugo. > > Currently we have developed many plugins based on Jekyll because the native > features of Jekyll cannot meet our needs. It seems all of them can be > supported via Hugo

Re: [DISCUSS] Backport broadcast operations in BATCH mode to Flink

2021-01-07 Thread Kostas Kloudas
+1 on my side as it does not break anything and it can act as motivation for some people to upgrade. Cheers, Kostas On Thu, 7 Jan 2021, 12:39 Aljoscha Krettek, wrote: > 1.12.x > Reply-To: > > Hi, > > what do you think about backporting FLINK-20491 [1] to Flink 1.12.x? > > I (we, including

Re: [DISCUSS] Enforce common opinionated coding style using Spotless

2020-12-16 Thread Kostas Kloudas
t;> checkstyle > > > >>> rules serving. > > > >>> > > > >>> For automatic applying part, we suggest developing by IDEA and with > > > >>> Checkstyle Plugin on IDEA applying checkstyle suggestion is also > > > &g

[jira] [Created] (FLINK-20240) Add setter in the environment for the `execution.runtime-mode`

2020-11-19 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-20240: -- Summary: Add setter in the environment for the `execution.runtime-mode` Key: FLINK-20240 URL: https://issues.apache.org/jira/browse/FLINK-20240 Project: Flink

[jira] [Created] (FLINK-20141) Add documentation about the FileSink

2020-11-12 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-20141: -- Summary: Add documentation about the FileSink Key: FLINK-20141 URL: https://issues.apache.org/jira/browse/FLINK-20141 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-20020) Make UnsuccessfulExecutionException part of the JobClient.getJobExecutionResult() contract.

2020-11-06 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-20020: -- Summary: Make UnsuccessfulExecutionException part of the JobClient.getJobExecutionResult() contract. Key: FLINK-20020 URL: https://issues.apache.org/jira/browse/FLINK-20020

[jira] [Created] (FLINK-19993) Remove the flink-connector-filesystem module

2020-11-05 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-19993: -- Summary: Remove the flink-connector-filesystem module Key: FLINK-19993 URL: https://issues.apache.org/jira/browse/FLINK-19993 Project: Flink Issue Type

Re: [VOTE] Remove flink-connector-filesystem module.

2020-11-05 Thread Kostas Kloudas
at 5:05 PM Till Rohrmann wrote: > > +1 > > Cheers, > Till > > On Fri, Oct 30, 2020 at 11:54 AM Guowei Ma wrote: > > > +1 > > Best, > > Guowei > > > > > > On Fri, Oct 30, 2020 at 6:02 PM Aljoscha Krettek > > wrote: > >

[VOTE] Remove flink-connector-filesystem module.

2020-10-29 Thread Kostas Kloudas
Hi all, Following the discussion in [1], I would like to start a vote on removing the flink-connector-filesystem module which includes the BucketingSink. The vote will be open till November 3rd (72h, excluding the weekend) unless there is an objection or not enough votes. Cheers, Kostas [1]

Re: [DISCUSS] Remove flink-connector-filesystem module.

2020-10-28 Thread Kostas Kloudas
>> >> To be clear, I, personally, don't have a problem with removing it (we >> have removed other connectors in the past that did not have a migration >> plan), I just reject he argumentation. >> >> On 10/28/2020 12:21 PM, Kostas Kloudas wrote: >> > No, I do

Re: [DISCUSS] Remove flink-connector-filesystem module.

2020-10-28 Thread Kostas Kloudas
e users because there are better alternatives. > > On 10/28/2020 10:46 AM, Kostas Kloudas wrote: > > I think that the mailing lists is the best we can do and I would say > > that they seem to be working pretty well (e.g. the recent Mesos > > discussion). > > Of course they ar

Re: [DISCUSS] Remove flink-connector-filesystem module.

2020-10-28 Thread Kostas Kloudas
strict. On Wed, Oct 28, 2020 at 10:04 AM Chesnay Schepler wrote: > > If the conclusion is that we shouldn't remove it if _anyone_ is using > it, then we cannot remove it because the user ML obviously does not > reach all users. > > On 10/28/2020 9:28 AM, Kostas Kloudas wrote: > &

Re: [DISCUSS] Remove flink-connector-filesystem module.

2020-10-28 Thread Kostas Kloudas
gt; > > Seth > > > > https://github.com/apache/flink/blob/2ff3b771cbb091e1f43686dd8e176cea6d435501/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java#L170-L172 > > > > On Thu, Oct 15, 2020 at 2:57 PM Ko

[jira] [Created] (FLINK-19815) Disable CoFeedbackTransformation for BATCH mode

2020-10-26 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-19815: -- Summary: Disable CoFeedbackTransformation for BATCH mode Key: FLINK-19815 URL: https://issues.apache.org/jira/browse/FLINK-19815 Project: Flink Issue

[jira] [Created] (FLINK-19814) Disable FeedbackTransformation for BATCH mode

2020-10-26 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-19814: -- Summary: Disable FeedbackTransformation for BATCH mode Key: FLINK-19814 URL: https://issues.apache.org/jira/browse/FLINK-19814 Project: Flink Issue Type

Re: [DISCUSS] Releasing Apache Flink 1.11.3

2020-10-26 Thread Kostas Kloudas
+1 for releasing Flink 1.11.3 as it contains a number of important fixes and thanks Gordon and Xintong for volunteering. Cheers, Kostas On Mon, Oct 26, 2020 at 4:37 AM Yu Li wrote: > > +1 for releasing Flink 1.11.3, and thanks Gordon and Xintong for > volunteering as our release managers. > >

Re: [SURVEY] Remove Mesos support

2020-10-23 Thread Kostas Kloudas
Thanks Piyush for the message. After this, I revoke my +1. I agree with the previous opinions that we cannot drop code that is actively used by users, especially if it something that deep in the stack as support for cluster management framework. Cheers, Kostas On Fri, Oct 23, 2020 at 4:15 PM

Re: [SURVEY] Remove Mesos support

2020-10-23 Thread Kostas Kloudas
+1 for adding a warning about the removal of Mesos support and I would also propose to state explicitly in the warning the version that we are planning to actually remove it (e.g. 1.13 or even 1.14 if we feel it is too aggressive). This will help as a reminder to users and devs about the upcoming

[jira] [Created] (FLINK-19732) Disable Checkpointing when in BATCH mode

2020-10-20 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-19732: -- Summary: Disable Checkpointing when in BATCH mode Key: FLINK-19732 URL: https://issues.apache.org/jira/browse/FLINK-19732 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-19730) Create the CoFeedbackTransformation translator

2020-10-20 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-19730: -- Summary: Create the CoFeedbackTransformation translator Key: FLINK-19730 URL: https://issues.apache.org/jira/browse/FLINK-19730 Project: Flink Issue

[jira] [Created] (FLINK-19729) Create the FeedbackTransformation translator

2020-10-20 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-19729: -- Summary: Create the FeedbackTransformation translator Key: FLINK-19729 URL: https://issues.apache.org/jira/browse/FLINK-19729 Project: Flink Issue Type

Re: [DISCUSS] Remove flink-connector-filesystem module.

2020-10-15 Thread Kostas Kloudas
to remove it at this point. > > On Tue, Oct 13, 2020 at 2:33 PM Kostas Kloudas wrote: > > > @Chesnay Schepler Off the top of my head, I cannot find an easy way > > to migrate from the BucketingSink to the StreamingFileSink. It may be > > possible but it will require so

Re: [DISCUSS] Remove flink-connector-filesystem module.

2020-10-13 Thread Kostas Kloudas
@Chesnay Schepler Off the top of my head, I cannot find an easy way to migrate from the BucketingSink to the StreamingFileSink. It may be possible but it will require some effort because the logic would be "read the old state, commit it, and start fresh with the StreamingFileSink." On Tue, Oct

Re: [DISCUSS] Remove flink-connector-filesystem module.

2020-10-12 Thread Kostas Kloudas
rsions of the module compatible with 1.12+? > > On 10/12/2020 4:30 PM, Kostas Kloudas wrote: > > Hi all, > > > > As the title suggests, this thread is to discuss the removal of the > > flink-connector-filesystem module which contains (only) the deprecated > > Buc

[DISCUSS] Remove flink-connector-filesystem module.

2020-10-12 Thread Kostas Kloudas
Hi all, As the title suggests, this thread is to discuss the removal of the flink-connector-filesystem module which contains (only) the deprecated BucketingSink. The BucketingSin is deprecated since FLINK 1.9 [1] in favor of the relatively recently introduced StreamingFileSink. For the sake of a

[jira] [Created] (FLINK-19583) Expose the execution.runtime-mode to the users

2020-10-12 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-19583: -- Summary: Expose the execution.runtime-mode to the users Key: FLINK-19583 URL: https://issues.apache.org/jira/browse/FLINK-19583 Project: Flink Issue

[jira] [Created] (FLINK-19579) Create the SideOutputTransformation translator

2020-10-12 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-19579: -- Summary: Create the SideOutputTransformation translator Key: FLINK-19579 URL: https://issues.apache.org/jira/browse/FLINK-19579 Project: Flink Issue

[jira] [Created] (FLINK-19577) Create the UnionTransformation translator

2020-10-12 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-19577: -- Summary: Create the UnionTransformation translator Key: FLINK-19577 URL: https://issues.apache.org/jira/browse/FLINK-19577 Project: Flink Issue Type

[jira] [Created] (FLINK-19578) Create the PartitionTransformation translator

2020-10-12 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-19578: -- Summary: Create the PartitionTransformation translator Key: FLINK-19578 URL: https://issues.apache.org/jira/browse/FLINK-19578 Project: Flink Issue Type

[jira] [Created] (FLINK-19575) Create the LegacySourceTransformation translator

2020-10-12 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-19575: -- Summary: Create the LegacySourceTransformation translator Key: FLINK-19575 URL: https://issues.apache.org/jira/browse/FLINK-19575 Project: Flink Issue

[jira] [Created] (FLINK-19576) Create the SinkTransformation translator

2020-10-12 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-19576: -- Summary: Create the SinkTransformation translator Key: FLINK-19576 URL: https://issues.apache.org/jira/browse/FLINK-19576 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-19573) Create the MultipleInputTransformation translator

2020-10-12 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-19573: -- Summary: Create the MultipleInputTransformation translator Key: FLINK-19573 URL: https://issues.apache.org/jira/browse/FLINK-19573 Project: Flink Issue

[jira] [Created] (FLINK-19574) Create the SourceTransformation translator

2020-10-12 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-19574: -- Summary: Create the SourceTransformation translator Key: FLINK-19574 URL: https://issues.apache.org/jira/browse/FLINK-19574 Project: Flink Issue Type

[jira] [Created] (FLINK-19572) Create the TwoInputTransformation translator

2020-10-12 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-19572: -- Summary: Create the TwoInputTransformation translator Key: FLINK-19572 URL: https://issues.apache.org/jira/browse/FLINK-19572 Project: Flink Issue Type

[jira] [Created] (FLINK-19571) Port Transformation translation logic to StreamGraphTranslator framework

2020-10-12 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-19571: -- Summary: Port Transformation translation logic to StreamGraphTranslator framework Key: FLINK-19571 URL: https://issues.apache.org/jira/browse/FLINK-19571 Project

Re: [DISCUSS] Enforce common opinionated coding style using Spotless

2020-10-07 Thread Kostas Kloudas
Hi all, +1 for enforcing "a" codestyle using "a" tool. As the project grows both in terms of LOCs and contributors, this becomes more and more important as it eliminates some potential points of friction without any additional effort. >From the discussion, I am leaning towards having a single

[jira] [Created] (FLINK-19485) Consider runtime-mode when creating the StreamGraph

2020-10-01 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-19485: -- Summary: Consider runtime-mode when creating the StreamGraph Key: FLINK-19485 URL: https://issues.apache.org/jira/browse/FLINK-19485 Project: Flink

Re: [VOTE] FLIP-142: Disentangle StateBackends from Checkpointing

2020-09-29 Thread Kostas Kloudas
+1 (binding) Kostas On Tue, Sep 29, 2020 at 4:57 PM Yu Li wrote: > > +1 (binding) > > Thanks all for the patience of answering / addressing my questions in the > discussion thread. > > Best Regards, > Yu > > > On Thu, 17 Sep 2020 at 14:39, Dawid Wysakowicz > wrote: > > > +1 (binding) > > > >

Re: [VOTE] FLIP-143: Unified Sink API

2020-09-27 Thread Kostas Kloudas
+1 (binding) @Steven Wu I think there will be opportunities to fine tune the API during the implementation. Cheers, Kostas On Sun, Sep 27, 2020 at 7:56 PM Steven Wu wrote: > > +1 (non-binding) > > Although I would love to continue the discussion for tweaking the > CommitResult/GlobaCommitter

[jira] [Created] (FLINK-19392) Detect the execution mode based on the sources in the job.

2020-09-24 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-19392: -- Summary: Detect the execution mode based on the sources in the job. Key: FLINK-19392 URL: https://issues.apache.org/jira/browse/FLINK-19392 Project: Flink

Re: Kafka connection issues

2020-09-23 Thread Kostas Kloudas
pods. > > Thanks for your help. > > ~Ramya. > > On Tue, Sep 22, 2020 at 9:18 PM Kostas Kloudas wrote: >> >> Hi Ramya, >> >> Unfortunately your images are blocked. Could you upload them somewhere and >> post the links here? >> Also I think that the Ta

Re: Kafka connection issues

2020-09-22 Thread Kostas Kloudas
Hi Ramya, Unfortunately your images are blocked. Could you upload them somewhere and post the links here? Also I think that the TaskManager logs may be able to help a bit more. Could you please provide them here? Cheers, Kostas On Tue, Sep 22, 2020 at 8:58 AM Ramya Ramamurthy wrote: > Hi, > >

Re: Re: [VOTE] FLIP-134: Batch execution for the DataStream API

2020-09-18 Thread Kostas Kloudas
+1 My only suggestion (although by no means a blocker) would be to remove from the FLIP the `env.setRuntimeMode()` method. I say that because this is syntactic sugar over the `env.configure()` with the `execution.runtime-mode` option set to BATCH or STREAMING. These methods can be nice but they

Re: [DISCUSS] FLIP-134: DataStream Semantics for Bounded Input

2020-09-16 Thread Kostas Kloudas
Hi all, Thanks for keeping the discussion running while I was on holidays! I am catching up currently and I will post in the voting thread if I have any comments :) Cheers, Kostas On Wed, Sep 16, 2020 at 11:25 AM David Anderson wrote: > > Aljoscha, > > Thanks for the thorough response. I'm

Re: Re: [ANNOUNCE] New Apache Flink Committer - Igal Shilman

2020-09-16 Thread Kostas Kloudas
Congratulations Igal and welcome! Kostas On Wed, Sep 16, 2020 at 6:37 AM Guowei Ma wrote: > > Congratulations :) > Best, > Guowei > > > On Wed, Sep 16, 2020 at 11:54 AM Zhijiang > wrote: > > > Congratulations and welcome, Igal! > > > > > >

[VOTE] FLIP-134: DataStream Semantics for Bounded Input

2020-08-24 Thread Kostas Kloudas
Hi all, After the discussion in [1], I would like to open a voting thread for FLIP-134 [2] which discusses the semantics that the DataStream API will expose when applied on a bounded input. The vote will be open until 27th August (72h), unless there is an objection or not enough votes. Cheers,

Re: [DISCUSS] FLIP-134: DataStream Semantics for Bounded Input

2020-08-24 Thread Kostas Kloudas
Thanks a lot for the discussion! I will open a voting thread shortly! Kostas On Mon, Aug 24, 2020 at 9:46 AM Kostas Kloudas wrote: > > Hi Guowei, > > Thanks for the insightful comment! > > I agree that this can be a limitation of the current runtime, but I > think

Re: [DISCUSS] FLIP-134: DataStream Semantics for Bounded Input

2020-08-24 Thread Kostas Kloudas
lable in the BATCH mode in current > implementation. > So maybe we need more checks in the AUTOMATIC execution mode. > > Best, > Guowei > > > On Thu, Aug 20, 2020 at 10:27 PM Kostas Kloudas wrote: >> >> Hi all, >> >> Thanks for the comments! >> >&

Re: [DISCUSS] FLIP-134: DataStream Semantics for Bounded Input

2020-08-20 Thread Kostas Kloudas
timers at the end of > a job would be interesting, and would help in (at least some of) the cases I > have in mind. I don't have a better idea. > > David > > On Mon, Aug 17, 2020 at 8:24 PM Kostas Kloudas wrote: >> >> Hi Kurt and David, >> >> Thanks a lot for t

Re: [ANNOUNCE] New Flink Committer: David Anderson

2020-08-19 Thread Kostas Kloudas
Congratulations David! Kostas On Wed, Aug 19, 2020 at 2:33 PM Arvid Heise wrote: > > Congrats David! > > On Wed, Aug 19, 2020 at 11:17 AM Fabian Hueske wrote: > > > Congrats David, well deserved! > > > > Cheers, > > Fabian > > > > Am Mi., 19. Aug. 2020 um 11:05 Uhr schrieb Marta Paes Moreira <

Re: [DISCUSS] FLIP-134: DataStream Semantics for Bounded Input

2020-08-18 Thread Kostas Kloudas
ot;bounded >> > streaming" to be treated differently. If I've understood it correctly, the >> > section on scheduling allows me to choose STREAMING scheduling even if I >> > have bounded sources. I like that approach, because it recognizes that >> > even

Re: [DISCUSS] Removing deprecated methods from DataStream API

2020-08-17 Thread Kostas Kloudas
adFile,readFileStream(...),socketTextStream(...),socketTextStream(...) > (deprecated in 1.2) > > Looking forward to more opinions on the issue. > > Best, > > Dawid > > > On 17/08/2020 12:49, Kostas Kloudas wrote: > > Thanks a lot for starting this Dawi

Re: [DISCUSS] FLIP-134: DataStream Semantics for Bounded Input

2020-08-17 Thread Kostas Kloudas
ame job as in production, except with different sources and > sinks. While it might be a reasonable default, I'm not convinced that > switching a processing time streaming job to read from a bounded source > should always cause it to fail. > > David > > On Wed, Aug 12, 2020 a

Re: [DISCUSS] Removing deprecated methods from DataStream API

2020-08-17 Thread Kostas Kloudas
Thanks a lot for starting this Dawid, Big +1 for the proposed clean-up, and I would also add the deprecated methods of the StreamExecutionEnvironment like: enableCheckpointing(long interval, CheckpointingMode mode, boolean force) enableCheckpointing() isForceCheckpointing()

[DISCUSS] FLIP-134: DataStream Semantics for Bounded Input

2020-08-12 Thread Kostas Kloudas
Hi all, As described in FLIP-131 [1], we are aiming at deprecating the DataSet API in favour of the DataStream API and the Table API. After this work is done, the user will be able to write a program using the DataStream API and this will execute efficiently on both bounded and unbounded data.

Re: [DISCUSS] Planning Flink 1.12

2020-08-05 Thread Kostas Kloudas
+1 for end of October from me as well. Cheers, Kostas On Wed, Aug 5, 2020 at 9:59 AM Till Rohrmann wrote: > +1 for end of October from my side as well. > > Cheers, > Till > > On Tue, Aug 4, 2020 at 9:46 PM Stephan Ewen wrote: > > > The end of October sounds good from my side, unless it

[jira] [Created] (FLINK-18519) Propagate exception to client when execution fails for REST submission

2020-07-07 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-18519: -- Summary: Propagate exception to client when execution fails for REST submission Key: FLINK-18519 URL: https://issues.apache.org/jira/browse/FLINK-18519 Project

Re: NullPointer Exception while trying to access or read ReadOnly ctx in processElement method in KeyedBroadCastProcessFunction in Apache Flink

2020-07-03 Thread Kostas Kloudas
Hi all, Just as an addition to what Dawid asked, I would also like to ask: 1) which Flink version are you using? because the stack trace line numbers do not match the current master. 2) as a clarification (although maybe not relevant here), there is no guarantee on the order of the elements, so

Re: [VOTE] Release 1.11.0, release candidate #4

2020-07-02 Thread Kostas Kloudas
Hi all, As far as the issue that Chesnay mentioned that leads to a "Caused by: org.apache.flink.api.common.InvalidProgramException:" for DataSet examples with print() collect() or count() as sink, this was a semi-intensional side-effect of the application mode. Before, in these cases, the output

[jira] [Created] (FLINK-18469) Add Application Mode to release notes.

2020-07-02 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-18469: -- Summary: Add Application Mode to release notes. Key: FLINK-18469 URL: https://issues.apache.org/jira/browse/FLINK-18469 Project: Flink Issue Type: Sub

Re: [Reminder] Prefer {% link %} tag in documentation

2020-06-09 Thread Kostas Kloudas
Thanks for the heads up Seth! Kostas On Tue, Jun 9, 2020 at 7:27 PM Seth Wiesman wrote: > > The tag is new to Jekyll 4.0 which we only recently updated to. > > There are a lot of existing tags that would need to be updated first :) > I opened a ticket to track that work and then yes that would

[jira] [Created] (FLINK-18084) Create documentation for the Application Mode

2020-06-03 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-18084: -- Summary: Create documentation for the Application Mode Key: FLINK-18084 URL: https://issues.apache.org/jira/browse/FLINK-18084 Project: Flink Issue Type

[DISCUSS] Remove dependency shipping through nested jars during job submission.

2020-05-20 Thread Kostas Kloudas
Hi all, I would like to bring the discussion in https://issues.apache.org/jira/browse/FLINK-17745 to the dev mailing list, just to hear the opinions of the community. In a nutshell, in the early days of Flink, users could submit their jobs as fat-jars that had a specific structure. More

[jira] [Created] (FLINK-17729) Make mandatory to have lib/, plugin/ and dist in yarn.provided.lib.dirs

2020-05-15 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-17729: -- Summary: Make mandatory to have lib/, plugin/ and dist in yarn.provided.lib.dirs Key: FLINK-17729 URL: https://issues.apache.org/jira/browse/FLINK-17729 Project

[jira] [Created] (FLINK-17639) Document which FileSystems are supported by the StreamingFileSink

2020-05-12 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-17639: -- Summary: Document which FileSystems are supported by the StreamingFileSink Key: FLINK-17639 URL: https://issues.apache.org/jira/browse/FLINK-17639 Project: Flink

[jira] [Created] (FLINK-17620) Rename StandaloneJobClusterEntryPoint to StandaloneApplicationClusterEntryPoint

2020-05-11 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-17620: -- Summary: Rename StandaloneJobClusterEntryPoint to StandaloneApplicationClusterEntryPoint Key: FLINK-17620 URL: https://issues.apache.org/jira/browse/FLINK-17620

[jira] [Created] (FLINK-17515) Move file uploading functionality to a separate class

2020-05-05 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-17515: -- Summary: Move file uploading functionality to a separate class Key: FLINK-17515 URL: https://issues.apache.org/jira/browse/FLINK-17515 Project: Flink

Re: [DISCUSS] FLIP-126: Unify (and separate) Watermark Assigners

2020-04-27 Thread Kostas Kloudas
Hi Aljoscha, Thanks for opening the discussion! I have two comments on the FLIP: 1) we could add lifecycle methods to the Generator, i.e. open()/ close(), probably with a Context as argument: I have not fully thought this through but I think that this is more aligned with the rest of our rich

[jira] [Created] (FLINK-16661) Introduce the YarnApplicationClusterEntrypoint

2020-03-18 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-16661: -- Summary: Introduce the YarnApplicationClusterEntrypoint Key: FLINK-16661 URL: https://issues.apache.org/jira/browse/FLINK-16661 Project: Flink Issue

[jira] [Created] (FLINK-16660) Introduce the ApplicationDispatcherBootstrap

2020-03-18 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-16660: -- Summary: Introduce the ApplicationDispatcherBootstrap Key: FLINK-16660 URL: https://issues.apache.org/jira/browse/FLINK-16660 Project: Flink Issue Type

[jira] [Created] (FLINK-16658) Wire the EmbeddedExecutor to the StandaloneClusterEntrypoint.

2020-03-18 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-16658: -- Summary: Wire the EmbeddedExecutor to the StandaloneClusterEntrypoint. Key: FLINK-16658 URL: https://issues.apache.org/jira/browse/FLINK-16658 Project: Flink

[jira] [Created] (FLINK-16657) Wire the EmbeddedExecutor to the Web Submission logic.

2020-03-18 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-16657: -- Summary: Wire the EmbeddedExecutor to the Web Submission logic. Key: FLINK-16657 URL: https://issues.apache.org/jira/browse/FLINK-16657 Project: Flink

[jira] [Created] (FLINK-16656) Introduce the DispatcherBootstrap

2020-03-18 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-16656: -- Summary: Introduce the DispatcherBootstrap Key: FLINK-16656 URL: https://issues.apache.org/jira/browse/FLINK-16656 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-16655) Introduce the EmbeddedExecutor

2020-03-18 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-16655: -- Summary: Introduce the EmbeddedExecutor Key: FLINK-16655 URL: https://issues.apache.org/jira/browse/FLINK-16655 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-16654) Implement Application Mode according to FLIP-85

2020-03-18 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-16654: -- Summary: Implement Application Mode according to FLIP-85 Key: FLINK-16654 URL: https://issues.apache.org/jira/browse/FLINK-16654 Project: Flink Issue

Re: [DISCUSS] FLIP-115: Filesystem connector in Table

2020-03-18 Thread Kostas Kloudas
Hi all, I also agree with Stephan on this! It has been more than a year now that most of our efforts have had the "unify" / "unification"/ etc either on their title or in their core and this has been the focus of all our resources. By deviating from this now, we only put more stress on other

Re: [DISCUSS] Drop Bucketing Sink

2020-03-17 Thread Kostas Kloudas
t; > > >> David > > > > >> > > > > >> [1] > > > > >> > > > > >> > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-46%3A+Graceful+Shutdown+Handling+by+UDFs > > >

Re: [Discussion] Job generation / submission hooks & Atlas integration

2020-03-13 Thread Kostas Kloudas
that could be registered in the StreamExecutionEnvironment. This meant > > that > > > we could work on the StreamGraph, register metadata in the JobGraph, and > > > execute the actual atlas registration logic in the JobMaster when the job > > > starts. > > >

Re: [Discussion] Job generation / submission hooks & Atlas integration

2020-03-12 Thread Kostas Kloudas
ed to fit the pattern :) For not always being able to return the > pipeline, this might be expected depending on how the JobClient, so we need > to handle it some way. > > > On Thu, Mar 12, 2020 at 8:30 PM Kostas Kloudas wrote: > > > Hi again, > > > > Just to clarify, I a

Re: [Discussion] Job generation / submission hooks & Atlas integration

2020-03-12 Thread Kostas Kloudas
Hi again, Just to clarify, I am not against exposing the Pipeline if this will lead to a "clean" solution. And, I. forgot to say that the last solution, if adopted, would have to work on the JobGraph, which may not be that desirable. Kostas On Thu, Mar 12, 2020 at 8:26 PM Kostas Klou

Re: [Discussion] Job generation / submission hooks & Atlas integration

2020-03-12 Thread Kostas Kloudas
Hi all, I do not have a strong opinion on the topic yet, but I would like to share my thoughts on this. In the solution proposing a wrapping AtlasExecutor around the Flink Executors, if we allow the user to use the CLI to submit jobs, then this means that the CLI code may have to change so that

Re: [DISCUSS] Drop Bucketing Sink

2020-03-12 Thread Kostas Kloudas
Hi Robert, +1 for dropping the BucketingSink. In any case, it has not been maintained for quite some time. Cheers, Kostas On Thu, Mar 12, 2020 at 3:41 PM Robert Metzger wrote: > > Hi all, > > I'm currently investigating a failing end to end test for the bucketing > sink [1]. > The bucketing

Re: [VOTE] [FLIP-85] Flink Application Mode

2020-03-12 Thread Kostas Kloudas
Hi all, Thanks for the votes. We have 3 binding +1 votes: Tison, Rong Rong, and Aljoscha and 0 negative ones. I will move the FLIP to accepted and will start opening JIRAs. Thanks a lot, Kostas On Thu, Mar 12, 2020 at 11:17 AM Aljoscha Krettek wrote: > > +1 (binding) > > Aljoscha

Re: [DISCUSS] FLIP-85: Delayed Job Graph Generation

2020-03-10 Thread Kostas Kloudas
Hi all, The FLIP was updated under the section "First Version Deliverables". Cheers, Kostas On Tue, Mar 10, 2020 at 4:10 PM Kostas Kloudas wrote: > > Hi all, > > Yes I will do that. From the discussion, I will add that: > 1) for the cli, we are planning to add a &quo

Re: [DISCUSS] FLIP-85: Delayed Job Graph Generation

2020-03-10 Thread Kostas Kloudas
ncies/jars are built into the image. Cheers, Kostas On Tue, Mar 10, 2020 at 3:05 PM Yang Wang wrote: > > Thanks for your response. > > @Kostas Kloudas Could we update the cli changes and how to fetch the > user jars to FLIP document? I think other dev or users may have the similar > q

Re: [DISCUSS] FLIP-85: Delayed Job Graph Generation

2020-03-09 Thread Kostas Kloudas
terEntrypoint's responsibility. >>>>> >>>>> >Will there be an option of not tearing the cluster down? >>>>> >>>>> I think the answer is "No" because the cluster is designed to be bounded >>>>> with an Application. Us

[VOTE] [FLIP-85] Flink Application Mode

2020-03-06 Thread Kostas Kloudas
Hi all, I would like to start the vote for FLIP-85 [1], which is discussed and reached a consensus in the discussion thread [2]. Given that it is Friday, the vote will be open until March. 11th (72h starting on Monday), unless there is an objection or not enough votes. Thanks, Kostas [1]

Re: [PROPOSAL] Reverse the dependency from flink-streaming-java to flink-client

2020-03-05 Thread Kostas Kloudas
Big +1 also from my side. This will eliminate some work-arounds used so far to bypass the module structure (like code using reflection to extract a JobGraph from a Pipeline). I agree with Stephan that with proper documentation, release notes and tooling update, it will hopefully not be a big

Re: [DISCUSS] FLIP-85: Delayed Job Graph Generation

2020-03-05 Thread Kostas Kloudas
Also from my side +1 to start voting. Cheers, Kostas On Thu, Mar 5, 2020 at 7:45 AM tison wrote: > > +1 to star voting. > > Best, > tison. > > > Yang Wang 于2020年3月5日周四 下午2:29写道: >> >> Hi Peter, >> Really thanks for your response. >> >> Hi

Re: [DISCUSS] FLIP-85: Delayed Job Graph Generation

2020-03-03 Thread Kostas Kloudas
implementation for per job cluster. > > To minimize the confusion of users, I think it would be better just an option > of per job cluster for each type of cluster manager. How do you think? > > > Best Regards > Peter Huang > > > > > > > > > On Mon, Mar

Re: [DISCUSS] FLIP-85: Delayed Job Graph Generation

2020-03-02 Thread Kostas Kloudas
ce between per-job and application mode? > > For per-job, user `main()` is always executed on client side. And For > application mode, user > `main()` could be executed on client or master side(configured via cli > option). > Right? We need to have a clear concept. Otherwise, the users w

Re: [DISCUSS] FLIP-85: Delayed Job Graph Generation

2020-03-02 Thread Kostas Kloudas
in the ML :) Thanks for starting the discussion and I hope that soon we will be able to vote on the FLIP. Cheers, Kostas On Thu, Jan 16, 2020 at 3:40 AM Yang Wang wrote: > > Hi all, > > Thanks a lot for the feedback from @Kostas Kloudas. Your all concerns are > on point. The FLIP-85 is m

[jira] [Created] (FLINK-16369) Allow the StreamingFileSink to restore from a previous (old) savepoint.

2020-03-02 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-16369: -- Summary: Allow the StreamingFileSink to restore from a previous (old) savepoint. Key: FLINK-16369 URL: https://issues.apache.org/jira/browse/FLINK-16369 Project

Re: [DISCUSS] Improvements on FLIP Process

2020-02-18 Thread Kostas Kloudas
+1 to what Aljoscha and Timo are proposing. I would lean towards eliminating Google Docs altogether. I think they served a purpose when discussions were among 3 to 4 people but with the current size of the community and the amount of participants per discussion they become difficult to follow.

Re: [VOTE] Release 1.10.0, release candidate #3

2020-02-10 Thread Kostas Kloudas
Hi all, +1 (binding) - Built Flink locally - Tested quickstart by writing simple, WordCount-like streaming jobs - Submitted them to Yarn both "per-job" and "session" mode - used the yarn-session CLI to start/stop sessions Thanks a lot Gary and Yu for managing the release. Cheers, Kostas On

Re: performances of S3 writing with many buckets in parallel

2020-02-07 Thread Kostas Kloudas
Hi Enrico, Nice to hear from you and thanks for checking it out! This can be helpful for people using the BucketingSink but I would recommend you to switch to the StreamingFileSink which is the "new version" of the BucketingSink. In fact the BucketingSink is going to be removed in one of the

[jira] [Created] (FLINK-15926) Add DataStream.broadcast(StateDescriptor) to available transformations docs

2020-02-05 Thread Kostas Kloudas (Jira)
Kostas Kloudas created FLINK-15926: -- Summary: Add DataStream.broadcast(StateDescriptor) to available transformations docs Key: FLINK-15926 URL: https://issues.apache.org/jira/browse/FLINK-15926

Re: [VOTE] Release 1.10.0, release candidate #1

2020-02-03 Thread Kostas Kloudas
+1 (binding) - Built Flink locally - Tested quickstart by writing simple, WordCount-like jobs - Submitted them to Yarn both "per-job" and "session" mode For Thomas' comment, I agree that in this release we change how some of the execution options are propagated through the stack. This was done

Re: [VOTE] Release 1.9.2, release candidate #1

2020-01-30 Thread Kostas Kloudas
Hi all, +1 (binding) - Compiled locally - Built simple jobs using quickstart - Submitted on Yarn both per-job and session cluster Cheers, Kostas On Thu, Jan 30, 2020 at 6:26 AM Hequn Cheng wrote: > > Hi everyone, > > Thanks a lot for your checking and voting for the release! > I’ll summarize

  1   2   3   4   5   >