Re: Contributing Code Guide - Clarification

[Stephan Ewen]

My personal take is the following:

  - These improvements are welcome in general. Improving code quality is a
good idea.
  - At the same time, these refactorings can easily introduce new bugs
  - I would focus on issues where there is clearly duplication without
need, but the code and refactoring is straightforward enough that a
non-committer can well understand the implications.
  - Please be aware that such PRs might not be reviewed with the highest
priority, because there is not a bug or immediate user-improvement
associated with it. It also depends on where in the release cycle the
project currently is (close to feature freeze will be harder to get
attention for this).

Best,
Stephan


On Sat, Jun 29, 2019 at 12:24 AM Hugo Louro  wrote:

> Hi,
>
> I would like to ask if minor, focused, refactoring done by extracting
> duplicate code to private methods is welcome for existing code/classes. The
> reasoning behind my question is that I want to avoid [1], but at the same
> time I clearly identify with [2]. If it is indeed welcome, is it OK to
> commit it with a message along the lines
> "[hotfix][refactoring][component-name] - brief message", or is it more
> appropriate to create a JIRA ?
>
> [1] - "Is this a contribution just for the sake of getting a commit in an
> open source project (fixing typos, style changes merely for taste reasons".
> ~ contribute code guide
> 
>
> [2] - "Whenever you are about to copy/paste some code, or reproduce a
> similar type of functionality in a different place, think about the ways
> how to refactor/reuse/abstract the changes to avoid the duplication" ~
> "Apache
> Flink Code Style and Quality Guide
> <
> https://docs.google.com/document/d/1owKfK1DwXA-w6qnx3R7t2D_o0BsFkkukGlRhvl3XXjQ/edit#heading=h.7xph6o4i5xf0
> >
> "
>
> Thanks,
> Hugo
>

Re: [VOTE] Migrate to sponsored Travis account

[Stephan Ewen]

+1 to move to a private Travis account.

I can confirm that Ververica will sponsor a Travis CI plan that is
equivalent or a bit higher than the previous ASF quota (10 concurrent build
queues)

Best,
Stephan

On Thu, Jul 4, 2019 at 10:46 AM Chesnay Schepler  wrote:

> I've raised a JIRA
> with INFRA to inquire
> whether it would be possible to switch to a different Travis account,
> and if so what steps would need to be taken.
> We need a proper confirmation from INFRA since we are not in full
> control of the flink repository (for example, we cannot access the
> settings page).
>
> If this is indeed possible, Ververica is willing sponsor a Travis
> account for the Flink project.
> This would provide us with more than enough resources than we need.
>
> Since this makes the project more reliant on resources provided by
> external companies I would like to vote on this.
>
> Please vote on this proposal, as follows:
> [ ] +1, Approve the migration to a Ververica-sponsored Travis account,
> provided that INFRA approves
> [ ] -1, Do not approach the migration to a Ververica-sponsored Travis
> account
>
> The vote will be open for at least 24h, and until we have confirmation
> from INFRA. The voting period may be shorter than the usual 3 days since
> our current is effectively not working.
>
> On 04/07/2019 06:51, Bowen Li wrote:
> > Re: > Are they using their own Travis CI pool, or did the switch to an
> > entirely different CI service?
> >
> > I reached out to Wes and Krisztián from Apache Arrow PMC. They are
> > currently moving away from ASF's Travis to their own in-house metal
> > machines at [1] with custom CI application at [2]. They've seen
> > significant improvement w.r.t both much higher performance and
> > basically no resource waiting time, "night-and-day" difference quoting
> > Wes.
> >
> > Re: > If we can just switch to our own Travis pool, just for our
> > project, then this might be something we can do fairly quickly?
> >
> > I believe so, according to [3] and [4]
> >
> >
> > [1] https://ci.ursalabs.org/ 
> > [2] https://github.com/ursa-labs/ursabot
> > [3]
> > https://docs.travis-ci.com/user/migrate/open-source-repository-migration
> > [4] https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com
> >
> >
> >
> > On Wed, Jul 3, 2019 at 12:01 AM Chesnay Schepler  > > wrote:
> >
> > Are they using their own Travis CI pool, or did the switch to an
> > entirely different CI service?
> >
> > If we can just switch to our own Travis pool, just for our
> > project, then
> > this might be something we can do fairly quickly?
> >
> > On 03/07/2019 05:55, Bowen Li wrote:
> > > I responded in the INFRA ticket [1] that I believe they are
> > using a wrong
> > > metric against Flink and the total build time is a completely
> > different
> > > thing than guaranteed build capacity.
> > >
> > > My response:
> > >
> > > "As mentioned above, since I started to pay attention to Flink's
> > build
> > > queue a few tens of days ago, I'm in Seattle and I saw no build
> > was kicking
> > > off in PST daytime in weekdays for Flink. Our teammates in China
> > and Europe
> > > have also reported similar observations. So we need to evaluate
> > how the
> > > large total build time came from - if 1) your number and 2) our
> > > observations from three locations that cover pretty much a full
> > day, are
> > > all true, I **guess** one reason can be that - highly likely the
> > extra
> > > build time came from weekends when other Apache projects may be
> > idle and
> > > Flink just drains hard its congested queue.
> > >
> > > Please be aware of that we're not complaining about the lack of
> > resources
> > > in general, I'm complaining about the lack of **stable, dedicated**
> > > resources. An example for the latter one is, currently even if
> > no build is
> > > in Flink's queue and I submit a request to be the queue head in PST
> > > morning, my build won't even start in 6-8+h. That is an absurd
> > amount of
> > > waiting time.
> > >
> > > That's saying, if ASF INFRA decides to adopt a quota system and
> > grants
> > > Flink five DEDICATED servers that runs all the time only for
> > Flink, that'll
> > > be PERFECT and can totally solve our problem now.
> > >
> > > Please be aware of that we're not complaining about the lack of
> > resources
> > > in general, I'm complaining about the lack of **stable, dedicated**
> > > resources. An example for the latter one is, currently even if
> > no build is
> > > in Flink's queue and I submit a request to be the queue head in PST
> > > morning, my build won't even start in 6-8+h. That is an absurd
> > amount of
> > > waiting time.
> > >
> > 

Re: [DISCUSS] Flink framework and user log separation

[Stephan Ewen]

Is that something that can just be done by the right logging framework and
configuration?

Like having a log framework with two targets, one filtered on
"org.apache.flink" and the other one filtered on "my.company.project" or so?

On Fri, Mar 1, 2019 at 3:44 AM vino yang  wrote:

> Hi Jamie Grier,
>
> Thank you for your reply, let me add some explanations to this design.
>
> First of all, as stated in "Goal", it is mainly for the "Standalone"
> cluster model, although we have implemented it for Flink on YARN, this does
> not mean that we can't turn off this feature by means of options. It should
> be noted that the separation is basically based on the "log configuration
> file", it is very scalable and even allows users to define the log pattern
> of the configuration file (of course this is an extension feature, not
> mentioned in the design documentation). In fact, "multiple files are a
> special case of a single file", we can provide an option to keep it still
> the default behavior, it should be the scene you expect in the container.
>
> According to Flink's official 2016 adjustment report [1], users using the
> standalone mode are quite close to the yarn mode (unfortunately there is no
> data support in 2017). Although we mainly use Flink on Yarn now, we have
> used standalone in depth (close to the daily processing volume of 20
> trillion messages). In this scenario, the user logs generated by different
> job's tasks are mixed together, and it is very difficult to locate the
> issue. Moreover, as we configure the log file scrolling policy, we have to
> log in to the server to view it. Therefore, we expect that for the same
> task manager, the user logs generated by the tasks from the same job can be
> distinguished.
>
> In addition, I have tried MDC technology, but it can not achieve the goal.
> The underlying Flink is log4j 1.x and logback. We need to be compatible
> with both frameworks at the same time, and we don't allow large-scale
> changes to the active code, and no sense to the user.
>
> Some other points:
>
> 1) Many of our users have experience using Storm and Spark, and they are
> more accustomed to that style in standalone mode;
> 2) We split the user log by Job, which will help to implement the "business
> log aggregation" feature based on the Job.
>
> Best,
> Vino
>
> [1]: https://www.ververica.com/blog/flink-user-survey-2016-part-1
>
> Jamie Grier  于2019年3月1日周五 上午7:32写道：
>
> > I think maybe if I understood this correctly this design is going in the
> > wrong direction.  The problem with Flink logging, when you are running
> > multiple jobs in the same TMs, is not just about separating out the
> > business level logging into separate files.  The Flink framework itself
> > logs many things where there is clearly a single job in context but that
> > all ends up in the same log file and with no clear separation amongst the
> > log lines.
> >
> > Also, I don't think shooting to have multiple log files is a very good
> idea
> > either.  It's common, especially on container-based deployments, that the
> > expectation is that a process (like Flink) logs everything to stdout and
> > the surrounding tooling takes care of routing that log data somewhere.  I
> > think we should stick with that model and expect that there will be a
> > single log stream coming out of each Flink process.
> >
> > Instead, I think it would be better to enhance Flink's logging capability
> > such that the appropriate context can be added to each log line with the
> > exact format controlled by the end user.  It might make sense to take a
> > look at MDC, for example, as a way to approach this.
> >
> >
> > On Thu, Feb 28, 2019 at 4:24 AM vino yang  wrote:
> >
> > > Dear devs,
> > >
> > > Currently, for log output, Flink does not explicitly distinguish
> between
> > > framework logs and user logs. In Task Manager, logs from the framework
> > are
> > > intermixed with the user's business logs. In some deployment models,
> such
> > > as Standalone or YARN session, there are different task instances of
> > > different jobs deployed in the same Task Manager. It makes the log
> event
> > > flow more confusing unless the users explicitly use tags to distinguish
> > > them and it makes locating problems more difficult and inefficient. For
> > > YARN job cluster deployment model, this problem will not be very
> serious,
> > > but we still need to artificially distinguish between the framework and
> > the
> > > business log. Overall, we found that Flink's existing log model has the
> > > following problems:
> > >
> > >
> > >-
> > >
> > >Framework log and business log are mixed in the same log file. There
> > >is no way to make a clear distinction, which is not conducive to
> > problem
> > >location and analysis;
> > >-
> > >
> > >Not conducive to the independent collection of business logs;
> > >
> > >
> > > Therefore, we propose a mechanism to separate the framework and
> business
> > > log. It can split 

Re: [DISCUSS] Create "flink-playgrounds" repository

[Stephan Ewen]

Hi all!

I am fine with a separate repository.

Quick question. though: Have you considered putting the setup not under
"docs" but under "flink-quickstart" or so?
Would that be equally cumbersome for users?

Best,
Stephan


On Thu, Jul 11, 2019 at 12:19 PM Fabian Hueske  wrote:

> Hi,
>
> I think Quickstart should be as lightweight as possible and follow common
> practices.
> A Git repository for a few configuration files might feel like overkill,
> but IMO it makes sense because this ensures users can get started with 3
> commands:
>
> $ git clone .../flink-playground
> $ cd flink-playground
> $ docker-compose up -d
>
> So +1 to create a repository.
>
> Thanks, Fabian
>
> Am Do., 11. Juli 2019 um 12:07 Uhr schrieb Robert Metzger <
> rmetz...@apache.org>:
>
> > +1 to create a repo.
> >
> > On Thu, Jul 11, 2019 at 11:10 AM Konstantin Knauf <
> > konstan...@ververica.com>
> > wrote:
> >
> > > Hi everyone,
> > >
> > > in the course of implementing FLIP-42 we are currently reworking the
> > > Getting Started section of our documentation. As part of this, we are
> > > adding docker-compose-based playgrounds to get started with Flink
> > > operations and Flink SQL quickly.
> > >
> > > To reduce as much friction as possible for new users, we would like to
> > > maintain the required configuration files (docker-comose.yaml,
> > > flink-conf.yaml) in a separate new repository,
> > `apache/flink-playgrounds`.
> > >
> > > You can find more details and a brief discussion on this in the
> > > corresponding Jira ticket [2].
> > >
> > > What do you think?
> > >
> > > I am not sure, what kind of approval is required for such a change. So,
> > my
> > > suggestion would be that we have lazy majority within the next 24 hours
> > to
> > > create the repository, we proceed. Please let me know, if this
> requires a
> > > more formal approval.
> > >
> > > Best and thanks,
> > >
> > > Konstantin
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-42%3A+Rework+Flink+Documentation
> > > [2] https://issues.apache.org/jira/browse/FLINK-12749
> > >
> > >
> > > --
> > >
> > > Konstantin Knauf | Solutions Architect
> > >
> > > +49 160 91394525
> > >
> > >
> > > Planned Absences: 10.08.2019 - 31.08.2019, 05.09. - 06.09.2010
> > >
> > >
> > > --
> > >
> > > Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
> > >
> > > --
> > >
> > > Ververica GmbH
> > > Registered at Amtsgericht Charlottenburg: HRB 158244 B
> > > Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
> > >
> >
>

Re: [DISCUSS] ARM support for Flink

[Stephan Ewen]

I think an ARM release would be cool.

To actually support that properly, we would need something like an ARM
profile for the CI builds (at least in the nightly tests), otherwise ARM
support would probably be broken frequently.
Maybe that could be a way to start? Create a Travis CI ARM build (if
possible) and see what tests pass and which parts of the system would need
to be adjusted?

On Thu, Jul 11, 2019 at 9:24 AM Xiyuan Wang 
wrote:

> Hi yun:
>   I didn't try to build rocksdb with vagrant, but just `make -j8
> rocksdbjava` directly in an ARM machine.  We hit some issues as well. My
> colleague has created an issue in rocksdb[1]. Rocksdb doesn't contains ARM
> .so file in his offical jar package. If you have the same request, let's
> work together there.
>
> Thanks.
>
> [1]: https://github.com/facebook/rocksdb/issues/5559
>
> Yun Tang  于2019年7月11日周四 下午12:01写道：
>
> > Hi Xiyuan
> >
> > Have you ever tried to release RocksDB on ARM like official doc[1]
> > suggests? From our experience, cross-building for ARM did not work fine
> > with Vagrant and we have to build rocksDB's binary file on ARM
> separately.
> >
> > As frocksdb [2] might not always maintained in Flink, I think we'd better
> > support to release RocksDB-java with ARM officially.
> >
> >
> > [1] https://github.com/facebook/rocksdb/blob/master/java/RELEASE.md
> > [2] https://github.com/dataArtisans/frocksdb
> >
> > Best
> > Yun Tang
> >
> >
> > 
> > From: Xiyuan Wang 
> > Sent: Tuesday, July 9, 2019 10:52
> > To: dev@flink.apache.org
> > Subject: Re: [DISCUSS] ARM support for Flink
> >
> > Thanks for your help. I built the frocksdb locally on ARM and all the
> > related tests are passed now. Except some tests which can be fixed
> easily,
> > it seems that both building and testing are ran well on ARM.
> >
> > Basing on my test, Is it possible to support Flink on ARM officailly?
> Seem
> > the worklist is not too long. And I can help with the CI testing part.
> >
> > Need Flink team's idea.
> >
> > Thanks.
> >
> > Dian Fu  于2019年7月8日周一 上午10:23写道：
> >
> > > Hi Xiyuan,
> > >
> > > Thanks for bring the discussion.
> > >
> > > WRT the exception, it's because the native bundled in the rocksdb jar
> > file
> > > isn't compiled with cross platform support. You can refer [1] for how
> to
> > > build rocksdb which has ARM platform.
> > >
> > > WRT ARM support, the rocksdb currently used in Flink is hosted in the
> > > Ververica git [2], so it won't be difficult to make it support ARM.
> > > However, I guess this git exists just for temporary [3], not because we
> > > want to add much feature in rocksdb.
> > >
> > > [1] https://github.com/facebook/rocksdb/issues/678 <
> > > https://github.com/facebook/rocksdb/issues/678>
> > > [2] https://github.com/dataArtisans/frocksdb <
> > > https://github.com/dataArtisans/frocksdb>
> > > [3] https://issues.apache.org/jira/browse/FLINK-10471 <
> > > https://issues.apache.org/jira/browse/FLINK-10471>
> > >
> > > Regards,
> > > Dian
> > >
> > > > 在 2019年7月8日，上午9:17，Xiyuan Wang  写道：
> > > >
> > > > Hi Flink:
> > > >  Recently we meet a problem that we have to test and run Flink on ARM
> > > > arch. While after searching Flink community, I didn’t find an
> official
> > > ARM
> > > > release version.
> > > >
> > > > Since Flink is made by Java and Scala language which can be ran
> > > > cross-platform usually, I think Flink can be built and ran on ARM
> > > directly
> > > > as well. Then with my local test, Flink was built and deployed
> success
> > as
> > > > expected. But some tests were failed due to ARM arch. For example:
> > > >
> > > > 1. MemoryArchitectureTest.testArchitectureNotUnknown:34 Values should
> > be
> > > > different. Actual: UNKNOWN
> > > > 2. [ERROR]
> > > >
> > >
> >
> testIterator(org.apache.flink.contrib.streaming.state.RocksDBRocksStateKeysIteratorTest)
> > > > Time elapsed: 0.234 s  <<< ERROR!
> > > > java.io.IOException: Could not load the native RocksDB library
> > > > at
> > > >
> > >
> >
> org.apache.flink.contrib.streaming.state.RocksDBRocksStateKeysIteratorTest.testIteratorHelper(RocksDBRocksStateKeysIteratorTest.java:90)
> > > > at
> > > >
> > >
> >
> org.apache.flink.contrib.streaming.state.RocksDBRocksStateKeysIteratorTest.testIterator(RocksDBRocksStateKeysIteratorTest.java:63)
> > > > Caused by: java.lang.UnsatisfiedLinkError:
> > > >
> > >
> >
> /tmp/rocksdb-lib-81ca7930b92af2cca143a050c0338d34/librocksdbjni-linux64.so:
> > > >
> > >
> >
> /tmp/rocksdb-lib-81ca7930b92af2cca143a050c0338d34/librocksdbjni-linux64.so:
> > > > cannot open shared object file: No such file or directory (Possible
> > > cause:
> > > > can't load AMD 64-bit .so on a AARCH64-bit platform)
> > > > …
> > > >
> > > >  Since the test isn’t passed totally, we are not sure if Flink 100%
> > > > support ARM or not. Is it possible for Flink to support ARM release
> > > > officially? I guess it may be not a very huge work basing on Java. I
> > > notice
> > > > that Flink now uses trivis-ci w

Re: [ANNOUNCE] Feature freeze for Apache Flink 1.9.0 release

[Stephan Ewen]

Number (6) is not a feature but a bug fix, so no need to block on that...

On Thu, Jul 11, 2019 at 4:27 PM Kurt Young  wrote:

> Hi Chesnay,
>
> Here is the JIRA list I have collected, all of them are under reviewing:
>
> 1. Hive UDF support (FLINK-13024, FLINK-13225)
> 2. Partition prune support (FLINK-13115)
> 3. Set StreamGraph properties in blink planner (FLINK-13121)
> 4. Support HBase upsert sink (FLINK-10245)
> 5. Support JDBC TableFactory (FLINK-13118)
> 6. Fix the bug of throwing IOException while FileBufferReader#nextBuffer
> (FLINK-13110)
> 7. Bookkeeping of available resources of allocated slots in SlotPool
> (FLINK-12765)
> 8. Introduce ScheduleMode#LAZY_FROM_SOURCES_WITH_BATCH_SLOT_REQUEST
> (FLINK-13187)
> 9. Add support for batch slot requests to SlotPoolImpl (FLINK-13166)
> 10. Complete slot requests in request order (FLINK-13165)
>
> Best,
> Kurt
>
>
> On Thu, Jul 11, 2019 at 6:12 PM Chesnay Schepler 
> wrote:
>
> > Can we get JIRA's attached to these items so people out of the loop can
> > track the progress?
> >
> > On 05/07/2019 16:06, Kurt Young wrote:
> > > Here are the features I collected which are under actively developing
> and
> > > close
> > > to merge:
> > >
> > > 1. Bridge blink planner to unified table environment and remove
> > TableConfig
> > > from blink planner
> > > 2. Support timestamp with local time zone and partition pruning in
> blink
> > > planner
> > > 3. Support JDBC & HBase lookup function and upsert sink
> > > 4. StreamExecutionEnvironment supports executing job with StreamGraph,
> > and
> > > blink planner should set proper properties to StreamGraph
> > > 5. Set resource profiles to task and enable managed memory as resource
> > > profile
> > >
> > > Best,
> > > Kurt
> > >
> > >
> > > On Fri, Jul 5, 2019 at 9:37 PM Kurt Young  wrote:
> > >
> > >> Hi devs,
> > >>
> > >> It's July 5 now and we should announce feature freeze and cut the
> branch
> > >> as planned. However, some components seems still not ready yet and
> > >> various features are still under development or review.
> > >>
> > >> But we also can not extend the freeze day again which will further
> delay
> > >> the
> > >> release date. I think freeze new features today and have another
> couple
> > >> of buffer days, letting features which are almost ready have a chance
> to
> > >> get in is a reasonable solution.
> > >>
> > >> I hereby announce features of Flink 1.9.0 are freezed, *July 11* will
> be
> > >> the
> > >> day for cutting branch.  Since the feature freeze has effectively took
> > >> place,
> > >> I kindly ask committers to refrain from merging features that are
> > planned
> > >> for
> > >> future releases into the master branch for the time being before the
> 1.9
> > >> branch
> > >> is cut. We understand this might be a bit inconvenient, thanks for the
> > >> cooperation here.
> > >>
> > >> Best,
> > >> Kurt
> > >>
> > >>
> > >> On Fri, Jul 5, 2019 at 5:19 PM 罗齐  wrote:
> > >>
> > >>> Hi Gordon,
> > >>>
> > >>> Will branch 1.9 be cut out today? We're really looking forward to the
> > >>> blink features in 1.9.
> > >>>
> > >>> Thanks,
> > >>> Qi
> > >>>
> > >>> On Wed, Jun 26, 2019 at 7:18 PM Tzu-Li (Gordon) Tai <
> > tzuli...@apache.org>
> > >>> wrote:
> > >>>
> >  Thanks for the updates so far everyone!
> > 
> >  Since support for the new Blink-based Table / SQL runner and
> > fine-grained
> >  recovery are quite prominent features for 1.9.0,
> >  and developers involved in these topics have already expressed that
> > these
> >  could make good use for another week,
> >  I think it definitely makes sense to postpone the feature freeze.
> > 
> >  The new date for feature freeze and feature branch cut for 1.9.0
> will
> > be
> >  *July
> >  5*.
> > 
> >  Please update on this thread if there are any further concerns!
> > 
> >  Cheers,
> >  Gordon
> > 
> >  On Tue, Jun 25, 2019 at 9:05 PM Chesnay Schepler <
> ches...@apache.org>
> >  wrote:
> > 
> > > On the fine-grained recovery / batch scheduling side we could make
> > good
> > > use of another week.
> > > Currently we are on track to have the _feature_ merged, but without
> > > having done a great deal of end-to-end testing.
> > >
> > > On 25/06/2019 15:01, Kurt Young wrote:
> > >> Hi Aljoscha,
> > >>
> > >> I also feel an additional week can make the remaining work more
> >  easy. At
> > >> least
> > >> we don't have to check in lots of commits in both branches
> (master &
> > >> release-1.9).
> > >>
> > >> Best,
> > >> Kurt
> > >>
> > >>
> > >> On Tue, Jun 25, 2019 at 8:27 PM Aljoscha Krettek <
> >  aljos...@apache.org>
> > >> wrote:
> > >>
> > >>> A few threads are converging around supporting the new
> Blink-based
> >  Table
> > >>> API Runner/Planner. I think hitting the currently proposed
> feature
> > > freeze
> > >>> date is hard, if not imp

Re: [DISCUSS] Create "flink-playgrounds" repository

[Stephan Ewen]

I am fine with a separate repository, was just raising the other option as
a question.

+1 to go ahead

On Fri, Jul 12, 2019 at 9:49 AM Konstantin Knauf 
wrote:

> Hi Xuefu,
>
> thanks for having a look at this. I am sure this playground setup will need
> to be maintained and will go through revisions, too. So, we would still
> need to keep the content of the archive in some repository + the additional
> piece of automation to update the archive, when the documentation is build.
> To me this seems to be more overhead than a repository.
>
> Best,
>
> Konstantin
>
>
> On Thu, Jul 11, 2019 at 9:00 PM Xuefu Z  wrote:
>
> > The idea seems interesting, but I'm wondering if we have considered
> > publishing .tz file hosted somewhere in Flink site with a link in the
> doc.
> > This might avoid the "overkill" of introducing a repo, which is main used
> > for version control in development cycles. On the other hand, a docker
> > setup, once published, will seldom (if ever) go thru revisions.
> >
> > Thanks,
> > Xuefu
> >
> >
> >
> > On Thu, Jul 11, 2019 at 6:58 AM Konstantin Knauf <
> konstan...@ververica.com
> > >
> > wrote:
> >
> > > Hi Stephan,
> > >
> > > putting it under "flink-quickstarts" alone would not help. The user
> would
> > > still need to check out the whole `apache/flink` repository, which is a
> > bit
> > > overwhelming. The Java/Scala quickstarts use Maven archetypes. Is this
> > what
> > > you are suggesting? I think, this would be an option, but it seems
> > strange
> > > to manage a pure Docker setup (eventually maybe only one file) in a
> Maven
> > > project.
> > >
> > > Best,
> > >
> > > Konstantin
> > >
> > > On Thu, Jul 11, 2019 at 3:52 PM Stephan Ewen  wrote:
> > >
> > > > Hi all!
> > > >
> > > > I am fine with a separate repository.
> > > >
> > > > Quick question. though: Have you considered putting the setup not
> under
> > > > "docs" but under "flink-quickstart" or so?
> > > > Would that be equally cumbersome for users?
> > > >
> > > > Best,
> > > > Stephan
> > > >
> > > >
> > > > On Thu, Jul 11, 2019 at 12:19 PM Fabian Hueske 
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I think Quickstart should be as lightweight as possible and follow
> > > common
> > > > > practices.
> > > > > A Git repository for a few configuration files might feel like
> > > overkill,
> > > > > but IMO it makes sense because this ensures users can get started
> > with
> > > 3
> > > > > commands:
> > > > >
> > > > > $ git clone .../flink-playground
> > > > > $ cd flink-playground
> > > > > $ docker-compose up -d
> > > > >
> > > > > So +1 to create a repository.
> > > > >
> > > > > Thanks, Fabian
> > > > >
> > > > > Am Do., 11. Juli 2019 um 12:07 Uhr schrieb Robert Metzger <
> > > > > rmetz...@apache.org>:
> > > > >
> > > > > > +1 to create a repo.
> > > > > >
> > > > > > On Thu, Jul 11, 2019 at 11:10 AM Konstantin Knauf <
> > > > > > konstan...@ververica.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > in the course of implementing FLIP-42 we are currently
> reworking
> > > the
> > > > > > > Getting Started section of our documentation. As part of this,
> we
> > > are
> > > > > > > adding docker-compose-based playgrounds to get started with
> Flink
> > > > > > > operations and Flink SQL quickly.
> > > > > > >
> > > > > > > To reduce as much friction as possible for new users, we would
> > like
> > > > to
> > > > > > > maintain the required configuration files (docker-comose.yaml,
> > > > > > > flink-conf.yaml) in a separate new repository,
> > > > > > `apache/flink-playgrounds`.
> > > > > > >
> > > > > > > You can find more details and a brief discussion on this in the
> > > > > > &

Re: [DISCUSS] Publish the PyFlink into PyPI

[Stephan Ewen]

Hi!

Sorry for the late involvement. Here are some thoughts from my side:

Definitely +1 to publishing to PyPy, even if it is a binary release.
Community growth into other communities is great, and if this is the
natural way to reach developers in the Python community, let's do it. This
is not about our convenience, but reaching users.

I think the way to look at this is that this is a convenience distribution
channel, courtesy of the Flink community. It is not an Apache release, we
make this clear in the Readme.
Of course, this doesn't mean we don't try to uphold similar standards as
for our official release (like proper license information).

Concerning credentials sharing, I would be fine with whatever option. The
PMC doesn't own it (it is an initiative by some community members), but the
PMC needs to ensure trademark compliance, so slight preference for option
#1 (PMC would have means to correct problems).

I believe there is no need to differentiate between Scala versions, because
this is merely a convenience thing for pure Python users. Users that mix
python and scala (and thus depend on specific scala versions) can still
download from Apache or build themselves.

Best,
Stephan



On Thu, Jul 4, 2019 at 9:51 AM jincheng sun 
wrote:

> Hi All,
>
> Thanks for the feedback @Chesnay Schepler  @Dian!
>
> I think using `apache-flink` for the project name also makes sense to me.
> due to we should always keep in mind that Flink is owned by Apache. (And
> beam also using this pattern `apache-beam` for Python API)
>
> Regarding the Python API release with the JAVA JARs, I think the principle
> of consideration is the convenience of the user. So, Thanks for the
> explanation @Dian!
>
> And your right @Chesnay Schepler   we can't make a
> hasty decision and we need more people's opinions！
>
> So, I appreciate it if anyone can give us feedback and suggestions!
>
> Best,
> Jincheng
>
>
>
>
> Chesnay Schepler  于2019年7月3日周三 下午8:46写道：
>
> > So this would not be a source release then, but a full-blown binary
> > release.
> >
> > Maybe it is just me, but I find it a bit suspect to ship an entire java
> > application via PyPI, just because there's a Python API for it.
> >
> > We definitely need input from more people here.
> >
> > On 03/07/2019 14:09, Dian Fu wrote:
> > > Hi Chesnay,
> > >
> > > Thanks a lot for the suggestions.
> > >
> > > Regarding “distributing java/scala code to PyPI”:
> > > The Python Table API is just a wrapper of the Java Table API and
> without
> > the java/scala code, two steps will be needed to set up an environment to
> > execute a Python Table API program:
> > > 1) Install pyflink using "pip install apache-flink"
> > > 2) Download the flink distribution and set the FLINK_HOME to it.
> > > Besides, users have to make sure that the manually installed Flink is
> > compatible with the pip installed pyflink.
> > >
> > > Bundle the java/scala code inside the Python package will eliminate
> step
> > 2) and makes it more simple for users to install pyflink. There was a
> short
> > discussion  on this in
> > Spark community and they finally decide to package the java/scala code in
> > the python package. (BTW, PySpark only bundle the jars of scala 2.11).
> > >
> > > Regards,
> > > Dian
> > >
> > >> 在 2019年7月3日，下午7:13，Chesnay Schepler  写道：
> > >>
> > >> The existing artifact in the pyflink project was neither released by
> > the Flink project / anyone affiliated with it nor approved by the Flink
> PMC.
> > >>
> > >> As such, if we were to use this account I believe we should delete it
> > to not mislead users that this is in any way an apache-provided
> > distribution. Since this goes against the users wishes, I would be in
> favor
> > of creating a separate account, and giving back control over the pyflink
> > account.
> > >>
> > >> My take on the raised points:
> > >> 1.1) "apache-flink"
> > >> 1.2)  option 2
> > >> 2) Given that we only distribute python code there should be no reason
> > to differentiate between scala versions. We should not be distributing
> any
> > java/scala code and/or modules to PyPi. Currently, I'm a bit confused
> about
> > this question and wonder what exactly we are trying to publish here.
> > >> 3) The should be treated as any other source release; i.e., it needs a
> > LICENSE and NOTICE file, signatures and a PMC vote. My suggestion would
> be
> > to make this part of our normal release process. There will be _one_
> source
> > release on dist.apache.org encompassing everything, and a separate
> python
> > of focused source release that we push to PyPi. The LICENSE and NOTICE
> > contained in the python source release must also be present in the source
> > release of Flink; so basically the python source release is just the
> > contents of flink-python module the maven pom.xml, with no other special
> > sauce added during the release process.
> > >>
> > >> On 02/07/2019 05:42, jincheng sun wrote:
> > >>> Hi all,
> > >>>

Re: [DISCUSS] Flink client api enhancement for downstream project

[Stephan Ewen]

月17日周一 下午7:13写道：
> > >>>
> > >>> > Is there any possibility to have something like Apache Livy [1]
> also
> > >>> for
> > >>> > Flink in the future?
> > >>> >
> > >>> > [1] https://livy.apache.org/
> > >>> >
> > >>> > On Tue, Jun 11, 2019 at 5:23 PM Jeff Zhang 
> wrote:
> > >>> >
> > >>> > > >>>  Any API we expose should not have dependencies on the
> runtime
> > >>> > > (flink-runtime) package or other implementation details. To me,
> > this
> > >>> > means
> > >>> > > that the current ClusterClient cannot be exposed to users because
> > it
> > >>> >  uses
> > >>> > > quite some classes from the optimiser and runtime packages.
> > >>> > >
> > >>> > > We should change ClusterClient from class to interface.
> > >>> > > ExecutionEnvironment only use the interface ClusterClient which
> > >>> should be
> > >>> > > in flink-clients while the concrete implementation class could be
> > in
> > >>> > > flink-runtime.
> > >>> > >
> > >>> > > >>> What happens when a failure/restart in the client happens?
> > There
> > >>> need
> > >>> > > to be a way of re-establishing the connection to the job, set up
> > the
> > >>> > > listeners again, etc.
> > >>> > >
> > >>> > > Good point.  First we need to define what does failure/restart in
> > the
> > >>> > > client mean. IIUC, that usually mean network failure which will
> > >>> happen in
> > >>> > > class RestClient. If my understanding is correct, restart/retry
> > >>> mechanism
> > >>> > > should be done in RestClient.
> > >>> > >
> > >>> > >
> > >>> > >
> > >>> > >
> > >>> > >
> > >>> > > Aljoscha Krettek  于2019年6月11日周二 下午11:10写道：
> > >>> > >
> > >>> > > > Some points to consider:
> > >>> > > >
> > >>> > > > * Any API we expose should not have dependencies on the runtime
> > >>> > > > (flink-runtime) package or other implementation details. To me,
> > >>> this
> > >>> > > means
> > >>> > > > that the current ClusterClient cannot be exposed to users
> because
> > >>> it
> > >>> > >  uses
> > >>> > > > quite some classes from the optimiser and runtime packages.
> > >>> > > >
> > >>> > > > * What happens when a failure/restart in the client happens?
> > There
> > >>> need
> > >>> > > to
> > >>> > > > be a way of re-establishing the connection to the job, set up
> the
> > >>> > > listeners
> > >>> > > > again, etc.
> > >>> > > >
> > >>> > > > Aljoscha
> > >>> > > >
> > >>> > > > > On 29. May 2019, at 10:17, Jeff Zhang 
> > wrote:
> > >>> > > > >
> > >>> > > > > Sorry folks, the design doc is late as you expected. Here's
> the
> > >>> > design
> > >>> > > > doc
> > >>> > > > > I drafted, welcome any comments and feedback.
> > >>> > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> >
> https://docs.google.com/document/d/1VavBrYn8vJeZs-Mhu5VzKO6xrWCF40aY0nlQ_UVVTRg/edit?usp=sharing
> > >>> > > > >
> > >>> > > > >
> > >>> > > > >
> > >>> > > > > Stephan Ewen  于2019年2月14日周四 下午8:43写道：
> > >>> > > > >
> > >>> > > > >> Nice that this discussion is happening.
> > >>> > > > >>
> > >>> > > > >> In the FLIP, we could also revisit the entire role of the
> > >>> > environments
> > >>> > > > >> again.
> > >>> > > > >

Re: [DISCUSS] Adopting a Code Style and Quality Guide

[Stephan Ewen]

@hugo For your suggestions, I would ask to start a separate discussion
thread.
I think this mail thread has converged towards merging the initial
suggestion as a starting point and refining it later based on new
discussions.

Best,
Stephan


On Thu, Jun 27, 2019 at 10:48 PM Hugo Louro  wrote:

> +1. Thanks for working on the guide. It's very thorough and a good resource
> to learn good practices from.
>
> I would like use this thread as a placeholder for a couple of topics that
> may be deserving of further discussion on different threads:
>   - What's the best way to keep track of checkstyle version updates. For
> instance, currently there is a PR
> <https://github.com/apache/flink/pull/8870> proposing checkstyle to be
> updated because version 8.12 is no longer supported
>  - When classes import shaded dependencies, it is not trivial for IntelliJ
> to download and associate sources and javadocs, which makes debugging and
> navigate the code base harder. I tried installing the version of the
> library using maven from the CLI, and associate the sources "manually" on
> IntelliJ, but it seems it does not work (perhaps IntelliJ issue). Does
> anyone know of a good solution? If so, we should added here
> <
> https://ci.apache.org/projects/flink/flink-docs-release-1.8/flinkDev/ide_setup.html#intellij-idea
> >.
> I can volunteer for that if you tell me how to do it :)
> - did the community evaluate naming test methods according to these
> conventions <https://stackoverflow.com/a/1594049> ?
>
> Thanks
>
> On Mon, Jun 24, 2019 at 11:38 AM Stephan Ewen  wrote:
>
> > I think it makes sense to also start individual [DISCUSS] threads about
> > extensions and follow-ups.
> > Various suggestions came up, partly as comments in the doc, some as
> > questions in other threads.
> >
> > Examples:
> >   - use of null in return types versus Optional versus @Nullable/@Nonnull
> >   - initialization of collection sizes
> >   - logging
> >
> > I think these would be best discussed in separate threads each.
> > So, for contributors to whom these issues are dear, feel free to spawn
> > these additional threads.
> > (Bear in mind it is close to 1.9 feature freeze time, so please leave
> this
> > discussions a bit of time so that all community members have a chance to
> > participate)
> >
> >
> >
> > On Mon, Jun 24, 2019 at 7:51 PM Stephan Ewen  wrote:
> >
> > > Thanks for the pointer David.
> > >
> > > I was not aware of this tool and I have no experience with its
> practical
> > > impact. For example I would be curious what the effect of this is for
> > > existing PRs, merge conflicts, etc.
> > >
> > > If you want, feel free to start another discuss thread on the
> > introduction
> > > of such a tool.
> > >
> > > On Sun, Jun 23, 2019 at 6:32 PM David Morávek  wrote:
> > >
> > >> I love this kind of unification being pushed forward, the benefit it
> has
> > >> on
> > >> code reviews is enormous. Just my two cents, did you guys think about
> > >> adopting an automatic code formatter for java, the same way as Apache
> > Beam
> > >> did?
> > >>
> > >> It is super easy to use for contributors as they don't need to keep
> any
> > >> particular coding style in mind and they can only focus on
> functionality
> > >> they want to fix, and it's also great for reviewers, because they only
> > see
> > >> the important changes. This also eliminates need for any special
> editor
> > /
> > >> checkstyle configs as the code formatting is part of the build itself.
> > >>
> > >> The one Beam uses is https://github.com/diffplug/spotless with
> > >> GoogleJavaFormat, it may be worth to look into.
> > >>
> > >> Best,
> > >> David
> > >>
> > >> On Fri, Jun 21, 2019 at 4:40 PM Stephan Ewen 
> wrote:
> > >>
> > >> > Thanks, everyone, for the positive feedback :-)
> > >> >
> > >> > @Robert - It probably makes sense to break this down into various
> > pages,
> > >> > like PR, general code style guide, Java, component specific guides,
> > >> > formats, etc.
> > >> >
> > >> > Best,
> > >> > Stephan
> > >> >
> > >> >
> > >> > On Fri, Jun 21, 2019 at 4:29 PM Robert Metzger  >
> > >> > wrote:
> > >> >
> &

Re: [DISCUSS] Allow at-most-once delivery in case of failures

[Stephan Ewen]

Hi all!

This is an interesting discussion for sure.

Concerning user requests for changes modes, I also hear the following quite
often:
  - reduce the expensiveness of checkpoint alignment (unaligned
checkpoints) to make checkpoints fast/stable under high backpressure
  - more fine-grained failover while maintaining exactly-once (even if
costly)

Having also "at most once" to the mix is quite a long list of big changes
to the system.

My feeling is that on such a core system, the community can not push all
these efforts at the same time, especially because they touch overlapping
areas of the system and need the same committers involved.

On the other hand, the pluggable shuffle service and pluggable scheduler
could make it possible to have an external implementation of that.
  - of a network stack that supports "reconnects" of failed tasks with
continuing tasks
  - a scheduling strategy that restarts tasks individually even in
pipelined regions

I think contributors/committers could implements this separate from the
Flink core. The feature would be trial-run it through the community
packages. If it gains a lot of traction, the community could decide to put
in the effort to merge this into the core.

Best,
Stephan


On Tue, Jun 11, 2019 at 2:10 PM SHI Xiaogang  wrote:

> Hi All,
>
> It definitely requires a massive effort to allow at-most-once delivery in
> Flink. But as the feature is urgently demanded by many Flink users, i think
> every effort we made is worthy. Actually, the inability to support
> at-most-once delivery has become a major obstacle for Storm users to turn
> to Flink. It's undesirable for us to run different stream processing
> systems for different scenarios.
>
> I agree with Zhu Zhu that the guarantee we provide is the very first thing
> to be discussed. Recovering with checkpoints will lead to duplicated
> records, thus break the guarantee on at-most-once delivery.
>
> A method to achieve at-most-once guarantee is to completely disable
> checkpointing and let sources only read those records posted after they
> start. The method requires sources to allow the configuration to read
> latest records, which luckily is supported by many message queues including
> Kafka. As Flink relies sources' ability to rollback to achieve exact-only
> and at-least-once delivery, i think it's acceptable for Flink to rely
> sources' ability to read latest records to achieve at-most once delivery.
> This method does not require any modification to existing checkpointing
> mechanism. Besides, as there is no need to restoring from checkpoints,
> failed tasks can recover themselves at the fastest speed.
>
> Concerning the implementation efforts, i think we can benefit from some
> ongoing work including shuffle services and fine-grained recovery. For
> example, currently the exceptions in network connections will lead to
> failures of downstream and upstream tasks. To achieve at-most-once
> delivery, we should decouple intermediate results from tasks, reporting the
> exceptions of intermediate results to job master and letting the failover
> strategy to determine the actions taken. Some work is already done in the
> efforts to achieve fine-grained recovery, which can be extended to allow
> at-most-once delivery in Flink.
>
> But before starting the discussion on implementation details, as said at
> prior, we need to determine the guarantee we provide in the scenarios where
> timely recovery is needed.
> * What do you think of the at-most-once guarantee achieved by the proposed
> method?
> * Do we need checkpointing to reduce the amount of lost data?
> * Do we need deduplication to guarantee at-most-once delivery or just
> provide best-effort delivery?
>
> Regards,
> Xiaogang Shi
>
>
> Piotr Nowojski  于2019年6月11日周二 下午5:31写道：
>
> > Hi Xiaogang,
> >
> > It sounds interesting and definitely a useful feature, however the
> > questions for me would be how useful, how much effort would it require
> and
> > is it worth it? We simply can not do all things at once, and currently
> > people that could review/drive/mentor this effort are pretty much
> strained
> > :( For me one would have to investigate answers to those questions and
> > prioritise it compared to other ongoing efforts, before I could vote +1
> for
> > this.
> >
> > Couple of things to consider:
> > - would it be only a job manager/failure region recovery feature?
> > - would it require changes in CheckpointBarrierHandler,
> > CheckpointCoordinator classes?
> > - with `at-most-once` semantic theoretically speaking we could just drop
> > the current `CheckpointBarrier` handling/injecting code and avoid all of
> > the checkpoint alignment issues - we could just checkpoint all of the
> tasks
> > independently of one another. However maybe that could be a follow up
> > optimisation step?
> >
> > Piotrek
> >
> > > On 11 Jun 2019, at 10:53, Zili Chen  wrote:
> > >
> > > Hi Xiaogang,
> > >
> > > It is an interesting topic.
> > >
> > > Notice that there is some effo

Re: Probing of simple repartition hash join

[Stephan Ewen]

Hi!

The join implementations used for the DataSet API and for the Blink Planner
are quite intricate. They make use of these custom memory segments, to
operate as much as possible on bytes, to control JVM memory utilization and
to save serialization costs.
That makes the implementation super complicated.

Do your experiments need that kind of behavior?
If not, it might actually be simpler to just start with some very simple
object-based hash table.

If you want to actually work with these classes, I would suggest to first
change the MutableHashTable a bit. It is probably easier to remove the
logic to pull from iterators into build and probe side, but rather do this
outside the class.
(I am assuming you plan to work with a fork for research here).

Best,
Stephan




On Thu, Jul 25, 2019 at 11:58 AM Benjamin Burkhardt <
pri...@benjaminburkhardt.de> wrote:

> Hi,
>
> while doing a join, a MutableHashTable is created and filled while
> building. After building it is closed and the probing can begin.
> I would like to start probing while building the hash table still runs.
> (ignoring the fact that this would lead to join misses...)
>
> Anyone having an idea how one could do that?
>
> Am 24. Juli 2019, 04:14 +0200 schrieb Caizhi Weng :
> > Hi Benjamin,
> >
> > As you mentioned hash join I assume that you are referring to
> > `HashJoinOperator` in blink planner.
> >
> > The input is selected by `nextSelection` method. As you can see, it will
> > first read all records in the build side then read all records in the
> probe
> > side. So the probing will only start after the build side has been fully
> > received.
> >
> > Apart from this specific question, I'm interested in how you're going to
> > implement a hash join where any of the two sides can be read. Could you
> > share your ideas or give some hints about this? Thanks a lot.
> >
> > Benjamin Burkhardt  于2019年7月24日周三 上午1:24写道：
> >
> > > Hi all,
> > >
> > > Let’s imagine a simple repartition hash Join oft two tables.
> > >
> > > As soon as the first table is hashed completely (all EndOfPartition
> Events
> > > sent) the shipping and probing of the second table starts.
> > >
> > > What I can’t find:
> > >
> > > 1. What triggers to start the probing exactly?
> > > 2. Where can I find it in the code?
> > >
> > >
> > > My final goal is to change the 2-phase join mechanism to a mixed
> > > implementation where probing for finished subpartitions begins earlier.
> > >
> > > I appreciate any help.
> > >
> > > Thanks.
> > >
> > > Benjamin
> > >
>

Re: [DISCUSS] FLIP-27: Refactor Source Interface

[Stephan Ewen]

Hi Biao!

Thanks for reviving this. I would like to join this discussion, but am
quite occupied with the 1.9 release, so can we maybe pause this discussion
for a week or so?

In the meantime I can share some suggestion based on prior experiments:

How to do watermarks / timestamp extractors in a simpler and more flexible
way. I think that part is quite promising should be part of the new source
interface.
https://github.com/StephanEwen/flink/tree/source_interface/flink-core/src/main/java/org/apache/flink/api/common/eventtime

https://github.com/StephanEwen/flink/blob/source_interface/flink-core/src/main/java/org/apache/flink/api/common/src/SourceOutput.java



Some experiments on how to build the source reader and its library for
common threading/split patterns:
https://github.com/StephanEwen/flink/tree/source_interface/flink-core/src/main/java/org/apache/flink/api/common/src


Best,
Stephan


On Thu, Jul 25, 2019 at 10:03 AM Biao Liu  wrote:

> Hi devs,
>
> Since 1.9 is nearly released, I think we could get back to FLIP-27. I
> believe it should be included in 1.10.
>
> There are so many things mentioned in document of FLIP-27. [1] I think
> we'd better discuss them separately. However the wiki is not a good place
> to discuss. I wrote google doc about SplitReader API which misses some
> details in the document. [2]
>
> 1.
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27:+Refactor+Source+Interface
> 2.
> https://docs.google.com/document/d/1R1s_89T4S3CZwq7Tf31DciaMCqZwrLHGZFqPASu66oE/edit?usp=sharing
>
> CC Stephan, Aljoscha, Piotrek, Becket
>
>
> On Thu, Mar 28, 2019 at 4:38 PM Biao Liu  wrote:
>
>> Hi Steven,
>> Thank you for the feedback. Please take a look at the document FLIP-27
>> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface>
>>  which
>> is updated recently. A lot of details of enumerator were added in this
>> document. I think it would help.
>>
>> Steven Wu  于2019年3月28日周四 下午12:52写道：
>>
>>> This proposal mentioned that SplitEnumerator might run on the JobManager
>>> or
>>> in a single task on a TaskManager.
>>>
>>> if enumerator is a single task on a taskmanager, then the job DAG can
>>> never
>>> been embarrassingly parallel anymore. That will nullify the leverage of
>>> fine-grained recovery for embarrassingly parallel jobs.
>>>
>>> It's not clear to me what's the implication of running enumerator on the
>>> jobmanager. So I will leave that out for now.
>>>
>>> On Mon, Jan 28, 2019 at 3:05 AM Biao Liu  wrote:
>>>
>>> > Hi Stephan & Piotrek,
>>> >
>>> > Thank you for feedback.
>>> >
>>> > It seems that there are a lot of things to do in community. I am just
>>> > afraid that this discussion may be forgotten since there so many
>>> proposals
>>> > recently.
>>> > Anyway, wish to see the split topics soon :)
>>> >
>>> > Piotr Nowojski  于2019年1月24日周四 下午8:21写道：
>>> >
>>> > > Hi Biao!
>>> > >
>>> > > This discussion was stalled because of preparations for the open
>>> sourcing
>>> > > & merging Blink. I think before creating the tickets we should split
>>> this
>>> > > discussion into topics/areas outlined by Stephan and create Flips for
>>> > that.
>>> > >
>>> > > I think there is no chance for this to be completed in couple of
>>> > remaining
>>> > > weeks/1 month before 1.8 feature freeze, however it would be good to
>>> aim
>>> > > with those changes for 1.9.
>>> > >
>>> > > Piotrek
>>> > >
>>> > > > On 20 Jan 2019, at 16:08, Biao Liu  wrote:
>>> > > >
>>> > > > Hi community,
>>> > > > The summary of Stephan makes a lot sense to me. It is much clearer
>>> > indeed
>>> > > > after splitting the complex topic into small ones.
>>> > > > I was wondering is there any detail plan for next step? If not, I
>>> would
>>> > > > like to push this thing forward by creating some JIRA issues.
>>> > > > Another question is that should version 1.8 include these features?
>>> > > >
>>> > > > Stephan Ewen  于2018年12月1日周六 上午4:20写道：
>>> > > >
>>> > > >> Thanks everyone for the lively discussion. Let me try to summarize
>>> > > where I
>>> > > >> see convergence in 

Re: Fine Grained Recovery / FLIP-1

[Stephan Ewen]

Hi Thomas!

For Batch, this should be working in release 1.9.

For streaming, it is a bit more tricky, mainly because of the fact that you
have to deal with downstream correctness.
Either a recovery still needs to reset downstream tasks (which means on
average half of the tasks) or needs to wait before publishing the data
downstream until a persistent point for recovery has been reached.

I have looked a bit into the second variant here. This needs a bit more
thought (currently also busy with 1.9 release) but in the course of the
next release cycle we might be able to share some initial design.

Best,
Stephan



On Fri, Jul 26, 2019 at 3:46 AM Guowei Ma  wrote:

> Hi,
> 1. Currently, much work in FLINK-4256 is about failover improvements in the
> bouded dataset scenario.
> 2. For the streaming scenario,  a new shuffle plugin + proper failover
> strategy could avoid the "stop-the-word" recovery.
> 3. We have already done many works about the new shuffle in the old Flink
> shuffle architectures because many of our customers have the concern. We
> have a plan to move the work to the new Flink pluggable shuffle
> architecture.
>
> Best,
> Guowei
>
>
> Thomas Weise  于2019年7月26日周五 上午8:54写道：
>
> > Hi,
> >
> > We are using Flink for streaming and find the "stop-the-world" recovery
> > behavior of Flink prohibitive for use cases that prioritize availability.
> > Partial recovery as outlined in FLIP-1 would probably alleviate these
> > concerns.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-1+%3A+Fine+Grained+Recovery+from+Task+Failures
> >
> > Looking at the subtasks in
> > https://issues.apache.org/jira/browse/FLINK-4256 it
> > appears that much of the work was already done but not much recent
> > progress? What is missing (for streaming)? How close is version 2
> (recovery
> > from limited intermediate results)?
> >
> > Thanks!
> > Thomas
> >
>

Re: [DISCUSS] ARM support for Flink

[Stephan Ewen]

I don't think it is feasible for Flink to migrate CI completely.

Is there a way to add ARM tests on an external CI in addition?
@Chesnay what do you think?


On Fri, Jul 12, 2019 at 4:45 AM Xiyuan Wang 
wrote:

> Hi Stephan
>   yeah, we should add an ARM CI first. But Travis CI doesn't support ARM
> arch itself. OpenLab community support it. As I mentioned before, OpenLab
> is an opensource CI system like travis-ci.[1], it uses opensource CI
> project `zuul`[2] for its deployment. Now some opensource project has
> intergreted with it already. For example, `contained` project from
> CNCF community[3]. And I have a POC for Flink ARM build and test using
> OpenLab. Now the build is passed[4], and I'm working on debugging with the
> `test` part[5]. Is it fine for Flink to using?
>
> [1]: https://openlabtesting.org
> [2]: https://zuul-ci.org/docs/zuul/
> [3]: https://status.openlabtesting.org/projects
> [4]:
> https://status.openlabtesting.org/build/2aa33f1a87854679b70f36bd6f75a890
> [5]: https://github.com/theopenlab/flink/pull/1
>
>
> Stephan Ewen  于2019年7月11日周四 下午9:56写道：
>
> > I think an ARM release would be cool.
> >
> > To actually support that properly, we would need something like an ARM
> > profile for the CI builds (at least in the nightly tests), otherwise ARM
> > support would probably be broken frequently.
> > Maybe that could be a way to start? Create a Travis CI ARM build (if
> > possible) and see what tests pass and which parts of the system would
> need
> > to be adjusted?
> >
> > On Thu, Jul 11, 2019 at 9:24 AM Xiyuan Wang 
> > wrote:
> >
> > > Hi yun:
> > >   I didn't try to build rocksdb with vagrant, but just `make -j8
> > > rocksdbjava` directly in an ARM machine.  We hit some issues as well.
> My
> > > colleague has created an issue in rocksdb[1]. Rocksdb doesn't contains
> > ARM
> > > .so file in his offical jar package. If you have the same request,
> let's
> > > work together there.
> > >
> > > Thanks.
> > >
> > > [1]: https://github.com/facebook/rocksdb/issues/5559
> > >
> > > Yun Tang  于2019年7月11日周四 下午12:01写道：
> > >
> > > > Hi Xiyuan
> > > >
> > > > Have you ever tried to release RocksDB on ARM like official doc[1]
> > > > suggests? From our experience, cross-building for ARM did not work
> fine
> > > > with Vagrant and we have to build rocksDB's binary file on ARM
> > > separately.
> > > >
> > > > As frocksdb [2] might not always maintained in Flink, I think we'd
> > better
> > > > support to release RocksDB-java with ARM officially.
> > > >
> > > >
> > > > [1] https://github.com/facebook/rocksdb/blob/master/java/RELEASE.md
> > > > [2] https://github.com/dataArtisans/frocksdb
> > > >
> > > > Best
> > > > Yun Tang
> > > >
> > > >
> > > > 
> > > > From: Xiyuan Wang 
> > > > Sent: Tuesday, July 9, 2019 10:52
> > > > To: dev@flink.apache.org
> > > > Subject: Re: [DISCUSS] ARM support for Flink
> > > >
> > > > Thanks for your help. I built the frocksdb locally on ARM and all the
> > > > related tests are passed now. Except some tests which can be fixed
> > > easily,
> > > > it seems that both building and testing are ran well on ARM.
> > > >
> > > > Basing on my test, Is it possible to support Flink on ARM officailly?
> > > Seem
> > > > the worklist is not too long. And I can help with the CI testing
> part.
> > > >
> > > > Need Flink team's idea.
> > > >
> > > > Thanks.
> > > >
> > > > Dian Fu  于2019年7月8日周一 上午10:23写道：
> > > >
> > > > > Hi Xiyuan,
> > > > >
> > > > > Thanks for bring the discussion.
> > > > >
> > > > > WRT the exception, it's because the native bundled in the rocksdb
> jar
> > > > file
> > > > > isn't compiled with cross platform support. You can refer [1] for
> how
> > > to
> > > > > build rocksdb which has ARM platform.
> > > > >
> > > > > WRT ARM support, the rocksdb currently used in Flink is hosted in
> the
> > > > > Ververica git [2], so it won't be difficult to make it support ARM.
> > > > > However, I guess this git exists just for temporary [3], not
> 

Re: [DISCUSS] Removing the flink-mapr-fs module

[Stephan Ewen]

+1 to remove it

One should still be able to use MapR in the same way as any other vendor
Hadoop distribution.

On Mon, Jul 29, 2019 at 12:22 PM JingsongLee
 wrote:

> +1 for removing it. We never run mvn clean test success in China with
> mapr-fs...
> Best, Jingsong Lee
>
>
> --
> From:Biao Liu 
> Send Time:2019年7月29日(星期一) 12:05
> To:dev 
> Subject:Re: [DISCUSS] Removing the flink-mapr-fs module
>
> +1 for removing it.
>
> Actually I encountered this issue several times. I thought it might be
> blocked by firewall of China :(
>
> BTW, I think it should be included in release notes.
>
>
> On Mon, Jul 29, 2019 at 5:37 PM Aljoscha Krettek 
> wrote:
>
> > If we remove it, that would mean it’s not supported in Flink 1.9.0, yes.
> > Or we only remove it in Flink 1.10.0.
> >
> > Aljoscha
> >
> > > On 29. Jul 2019, at 11:35, Biao Liu  wrote:
> > >
> > > Hi Aljoscha,
> > >
> > > Does it mean the MapRFileSystem is no longer supported since 1.9.0?
> > >
> > > On Mon, Jul 29, 2019 at 5:19 PM Ufuk Celebi  wrote:
> > >
> > >> +1
> > >>
> > >>
> > >> On Mon, Jul 29, 2019 at 11:06 AM Jeff Zhang  wrote:
> > >>
> > >>> +1 to remove it.
> > >>>
> > >>> Aljoscha Krettek  于2019年7月29日周一 下午5:01写道：
> > >>>
> >  Hi,
> > 
> >  Because of recent problems in the dependencies of that module [1] I
> > >> would
> >  suggest that we remove it. If people are using it, they can use the
> > one
> >  from Flink 1.8.
> > 
> >  What do you think about it? It would a) solve the dependency problem
> > >> and
> >  b) make our build a tiny smidgen more lightweight.
> > 
> >  Aljoscha
> > 
> >  [1]
> > 
> > >>>
> > >>
> >
> https://lists.apache.org/thread.html/16c16db8f4b94dc47e638f059fd53be936d5da423376a8c1092eaad1@%3Cdev.flink.apache.org%3E
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Best Regards
> > >>>
> > >>> Jeff Zhang
> > >>>
> > >>
> >
> >
>

Re: [DISCUSS] Removing the flink-mapr-fs module

[Stephan Ewen]

It should be fairly straightforward to rewrite the code to not have a MapR
dependency.
Only one class from the MapR dependency is ever referenced, and we could
dynamically load that class.

That way we can drop the dependency without dropping the support.

On Mon, Jul 29, 2019 at 5:33 PM Aljoscha Krettek 
wrote:

> Just FYI, the MapRFileSystem does have some additional code on our
> (Hadoop)FileSystem class, so it might not be straightforward to use MapR
> with our vanilla HadoopFileSystem.
>
> (Still staying that we should remove it, though).
>
> Aljoscha
>
> > On 29. Jul 2019, at 15:16, Simon Su  wrote:
> >
> > +1 to remove it.
> >
> >
> > Thanks,
> > SImon
> >
> >
> > On 07/29/2019 21:00，Till Rohrmann wrote：
> > +1 to remove it.
> >
> > On Mon, Jul 29, 2019 at 1:27 PM Stephan Ewen  wrote:
> >
> > +1 to remove it
> >
> > One should still be able to use MapR in the same way as any other vendor
> > Hadoop distribution.
> >
> > On Mon, Jul 29, 2019 at 12:22 PM JingsongLee
> >  wrote:
> >
> > +1 for removing it. We never run mvn clean test success in China with
> > mapr-fs...
> > Best, Jingsong Lee
> >
> >
> > --
> > From:Biao Liu 
> > Send Time:2019年7月29日(星期一) 12:05
> > To:dev 
> > Subject:Re: [DISCUSS] Removing the flink-mapr-fs module
> >
> > +1 for removing it.
> >
> > Actually I encountered this issue several times. I thought it might be
> > blocked by firewall of China :(
> >
> > BTW, I think it should be included in release notes.
> >
> >
> > On Mon, Jul 29, 2019 at 5:37 PM Aljoscha Krettek 
> > wrote:
> >
> > If we remove it, that would mean it’s not supported in Flink 1.9.0,
> > yes.
> > Or we only remove it in Flink 1.10.0.
> >
> > Aljoscha
> >
> > On 29. Jul 2019, at 11:35, Biao Liu  wrote:
> >
> > Hi Aljoscha,
> >
> > Does it mean the MapRFileSystem is no longer supported since 1.9.0?
> >
> > On Mon, Jul 29, 2019 at 5:19 PM Ufuk Celebi  wrote:
> >
> > +1
> >
> >
> > On Mon, Jul 29, 2019 at 11:06 AM Jeff Zhang 
> > wrote:
> >
> > +1 to remove it.
> >
> > Aljoscha Krettek  于2019年7月29日周一 下午5:01写道：
> >
> > Hi,
> >
> > Because of recent problems in the dependencies of that module [1]
> > I
> > would
> > suggest that we remove it. If people are using it, they can use
> > the
> > one
> > from Flink 1.8.
> >
> > What do you think about it? It would a) solve the dependency
> > problem
> > and
> > b) make our build a tiny smidgen more lightweight.
> >
> > Aljoscha
> >
> > [1]
> >
> >
> >
> >
> >
> >
> https://lists.apache.org/thread.html/16c16db8f4b94dc47e638f059fd53be936d5da423376a8c1092eaad1@%3Cdev.flink.apache.org%3E
> >
> >
> >
> > --
> > Best Regards
> >
> > Jeff Zhang
> >
> >
> >
> >
> >
> >
>
>

Re: [DISCUSS] Removing the flink-mapr-fs module

[Stephan Ewen]

I will open a PR later today, changing the module to use reflection rather
than a hard MapR dependency.

On Tue, Jul 30, 2019 at 6:40 AM Rong Rong  wrote:

> We've also experienced some issues with our internal JFrog artifactory. I
> am suspecting some sort of mirroring problem but somehow it only occur to
> the mapr-fs module.
> So +1 to remove.
>
> On Mon, Jul 29, 2019 at 12:47 PM Stephan Ewen  wrote:
>
> > It should be fairly straightforward to rewrite the code to not have a
> MapR
> > dependency.
> > Only one class from the MapR dependency is ever referenced, and we could
> > dynamically load that class.
> >
> > That way we can drop the dependency without dropping the support.
> >
> > On Mon, Jul 29, 2019 at 5:33 PM Aljoscha Krettek 
> > wrote:
> >
> > > Just FYI, the MapRFileSystem does have some additional code on our
> > > (Hadoop)FileSystem class, so it might not be straightforward to use
> MapR
> > > with our vanilla HadoopFileSystem.
> > >
> > > (Still staying that we should remove it, though).
> > >
> > > Aljoscha
> > >
> > > > On 29. Jul 2019, at 15:16, Simon Su  wrote:
> > > >
> > > > +1 to remove it.
> > > >
> > > >
> > > > Thanks,
> > > > SImon
> > > >
> > > >
> > > > On 07/29/2019 21:00，Till Rohrmann wrote：
> > > > +1 to remove it.
> > > >
> > > > On Mon, Jul 29, 2019 at 1:27 PM Stephan Ewen 
> wrote:
> > > >
> > > > +1 to remove it
> > > >
> > > > One should still be able to use MapR in the same way as any other
> > vendor
> > > > Hadoop distribution.
> > > >
> > > > On Mon, Jul 29, 2019 at 12:22 PM JingsongLee
> > > >  wrote:
> > > >
> > > > +1 for removing it. We never run mvn clean test success in China with
> > > > mapr-fs...
> > > > Best, Jingsong Lee
> > > >
> > > >
> > > > --
> > > > From:Biao Liu 
> > > > Send Time:2019年7月29日(星期一) 12:05
> > > > To:dev 
> > > > Subject:Re: [DISCUSS] Removing the flink-mapr-fs module
> > > >
> > > > +1 for removing it.
> > > >
> > > > Actually I encountered this issue several times. I thought it might
> be
> > > > blocked by firewall of China :(
> > > >
> > > > BTW, I think it should be included in release notes.
> > > >
> > > >
> > > > On Mon, Jul 29, 2019 at 5:37 PM Aljoscha Krettek <
> aljos...@apache.org>
> > > > wrote:
> > > >
> > > > If we remove it, that would mean it’s not supported in Flink 1.9.0,
> > > > yes.
> > > > Or we only remove it in Flink 1.10.0.
> > > >
> > > > Aljoscha
> > > >
> > > > On 29. Jul 2019, at 11:35, Biao Liu  wrote:
> > > >
> > > > Hi Aljoscha,
> > > >
> > > > Does it mean the MapRFileSystem is no longer supported since 1.9.0?
> > > >
> > > > On Mon, Jul 29, 2019 at 5:19 PM Ufuk Celebi  wrote:
> > > >
> > > > +1
> > > >
> > > >
> > > > On Mon, Jul 29, 2019 at 11:06 AM Jeff Zhang 
> > > > wrote:
> > > >
> > > > +1 to remove it.
> > > >
> > > > Aljoscha Krettek  于2019年7月29日周一 下午5:01写道：
> > > >
> > > > Hi,
> > > >
> > > > Because of recent problems in the dependencies of that module [1]
> > > > I
> > > > would
> > > > suggest that we remove it. If people are using it, they can use
> > > > the
> > > > one
> > > > from Flink 1.8.
> > > >
> > > > What do you think about it? It would a) solve the dependency
> > > > problem
> > > > and
> > > > b) make our build a tiny smidgen more lightweight.
> > > >
> > > > Aljoscha
> > > >
> > > > [1]
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/16c16db8f4b94dc47e638f059fd53be936d5da423376a8c1092eaad1@%3Cdev.flink.apache.org%3E
> > > >
> > > >
> > > >
> > > > --
> > > > Best Regards
> > > >
> > > > Jeff Zhang
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> >
>

Re: Reading RocksDB contents from outside of Flink

[Stephan Ewen]

Hi!

Are you looking for online access or offline access?

For online access, you can to key lookups via queryable state.

For offline access, you can read and write rocksDB state using the new
state processor API in Flink 1.9
https://ci.apache.org/projects/flink/flink-docs-master/dev/libs/state_processor_api.html


Best,
Stephan


On Tue, Jul 30, 2019 at 2:39 PM taher koitawala  wrote:

> Hi Shilpa,
>The easiest way to do this is the make the Rocks DB state queryable.
> Then use the Flink queryable state client to access the state you have
> created.
>
>
> Regards
> Taher Koitawala
>
> On Tue, Jul 30, 2019, 4:58 PM Shilpa Deshpande 
> wrote:
>
> > Hello All,
> >
> > I am new to Apache Flink. In my company we are thinking of using Flink to
> > perform transformation of the data. The source of the data is Apache
> Kafka
> > topics. Each message that we receive on Kafka topic, we want to transform
> > it and store it on RocksDB. The messages can come out of order. We would
> > like to store the transformed output in RocksDB using Flink and retrieve
> it
> > from a Spring Boot application. The reason for Spring Boot Application is
> > outbound messaging involves some stitching as well as ordering of the
> data.
> > Is it possible and advisable to read data from RocksDB from Spring Boot
> > Application? Spring Boot Application will not change the data. The reason
> > we think Flink can help us is because we might be receiving millions of
> > messages during the day so want to make sure use a technology that scales
> > well. If you have a snippet of code to achieve this, please share it with
> > us.
> >
> > Thank you for your inputs!
> >
> > Regards,
> > Shilpa
> >
>

Re: Flink Kafka Issues

[Stephan Ewen]

Is the watermarking configured per-partition in Kafka, or per source?

If it is configured per partition, then a late (trailing) or early
(leading) partition would not affect the watermark as a whole.
There would not be any dropping of late data, simply a delay in the results
until the latest partition (watermark wise) has caught up.

Best,
Stephan

On Wed, Jul 31, 2019 at 8:00 AM Ramya Ramamurthy  wrote:

> Hi Jiangjie,
>
> Thanks for your response. I was able to figure out the issue.
> We have multiple end points from which we receive data. Out of which, one
> of the endpoints NTP was not set up/rather not getting synced to ntp
> properly. so those VM's were sending packets which was 2 mins ahead of
> time. So all the rest of the packets from various other sources coming into
> that Kafka cluster was dropped. Increased the watermark to 3 mins [from 1
> min] and this stopped the "laterecordsdropped".
>
> But this is kind of worrisome, if there is anything like this, it would
> affect our systems badly, as we cannot afford to lose data. Is there any
> better way to approach this? The Flink tables do not have side outputs to
> collect these lost packets as well, which is also a concern.  Is this a
> feature in making ??
> I could see that the Flink Tables arent yet evolved like Streams. Let me
> know what you think.
>
> Regards,
> ~Ramya.
>
> On Mon, Jul 29, 2019 at 6:17 PM Becket Qin  wrote:
>
> > Hi Ramya,
> >
> > I am a little confused in the situation here. Is the following what you
> > saw:
> >
> > 1. The Kafka topic A has a traffic drop from 22:00 UTC to 00:30 UTC.
> > 2. Your Flink job (say job 1) reads from Topic A, but it has a high fetch
> > rate with an abnormally high CPU consumption during the period mentioned
> in
> > (1).
> > 3. Your Flink job (say job 2) reads from Topic B, and it sees a lot of
> > messages got dropped due to late arrivals. i.e. the timestamp of those
> > messages is larger than the watermark + max-allowed-lateness.
> >
> > What is the relationship between job 1 and job 2? Are they the same job?
> Is
> > there any producer sending "old" messages to the Kafka cluster which may
> > cause those messages to be dropped by Flink due to their old timestamp?
> >
> > Unfortunately the image does not work in apache mailing list. Can you
> post
> > the image somewhere and send the link instead?
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> >
> >
> > On Thu, Jul 18, 2019 at 9:36 AM Ramya Ramamurthy 
> > wrote:
> >
> > > Hi,
> > >
> > > We are facing a serious production issue with Flink. Any help would be
> > > appreciated.
> > >
> > > We receive packets from a Kafka Cluster - This cluster has a sudden
> drop
> > > in the packets from 22:00 UTC till 00:30 UTC everyday [on a specific
> > topic,
> > > say "topic A"]. Though our job reads from a different topic [say "topic
> > > B"], we see that we drop a lot of packets here [due to
> > "laterecordsDropped"
> > > metric]. At the same time, we see the job which reads from "topic A"
> has
> > > high fetch rate. We also observed one of the brokers of this cluster
> had
> > an
> > > abnormal CPU rise [which i attributed to the high fetch rates].
> > >
> > > We have a tumbling window of 1 min [with 10 seconds of
> > > watermarksPeriodicBounded].  This is based on the packets' event time.
> Is
> > > there any reason why my job reading from "topic B" can higher records
> > > dropped.
> > >
> > > The picture below has a screenshot where
> > > Laterecords dropped corresponds to job reading from "topic B"
> > > Fetch and Consume rates relates to job reading from "topic A" [which
> has
> > > the downward trend in traffic in the mentioned times].
> > >
> > > [image: image.png]
> > >
> > > All these graphs are correlated and we are unable to isolate this
> > problem.
> > > there are other modules which consumes from this topic, and we have no
> > slow
> > > records logged here, which is why we are not sure of there is this
> issue
> > > with Flink alone.
> > >
> > > Thanks.
> > >
> >
>

Re: [DISCUSS] ARM support for Flink

[Stephan Ewen]

Wow, that is pretty nice work, thanks a lot!

We need some support from Apache Infra to see if we can connect the Flink
Github Repo with the OpenLab CI.
We would also need a discussion on the developer mailing list, to get
community agreement.

Have you looked at whether we need to run all tests with ARM, or whether
maybe only the "core" and "tests" profile would be enough to get confidence
that Flink runs on ARM?
Just asking because Flink has a lot of long running tests by now that can
easily eat up a lot of CI capacity.

Best,
Stephan



On Tue, Jul 30, 2019 at 3:45 AM Xiyuan Wang 
wrote:

> Hi Stephan,
>   Maybe I misled you in the previous email. We don't need to migrate CI
> completely, travis-ci is still there working for X86 arch. What we need to
> do is to add another CI tool for ARM arch.
>
>   There are some ways to do it. As I wrote on
> https://issues.apache.org/jira/browse/FLINK-13199 to @Chesnay:
>
> 1. Add OpenLab CI system for ARM arch test.OpenLab is very similar with
> travis-ci. What Flilnk need to do is adding the openlab github app to the
> repo, then add the job define files inner Flink repo, Here is a POC by me:
> https://github.com/theopenlab/flink/pull/1
> 2. OpenLab will donate ARM resouces to Apache Infra team as well. Then
> Flink can use the Apache offical  Jenkins system for Flink ARM test in the
> future. https://builds.apache.org/
> 3. Use Drony CI which support ARM arch as well. https://drone.io/
>
> Since I'm from OpenLab community, if Flink choose OpenLab CI, My OpenLab
> colleague and I can keep helping and maintaining the ARM CI job. If choose
> the 2nd way, the CI maintainance work may be handled by apache-infra team I
> guess.  If choose the 3rd Drony CI, what we can help is very limited.
> AFAIK, Drony use container for CI test, which may not satisfy some
> requiremnts. And OpenLab use VM for test.
>
> Need Flink core team's decision and reply.
>
> Thanks.
>
>
> Stephan Ewen  于2019年7月29日周一 下午6:05写道：
>
> > I don't think it is feasible for Flink to migrate CI completely.
> >
> > Is there a way to add ARM tests on an external CI in addition?
> > @Chesnay what do you think?
> >
> >
> > On Fri, Jul 12, 2019 at 4:45 AM Xiyuan Wang 
> > wrote:
> >
> > > Hi Stephan
> > >   yeah, we should add an ARM CI first. But Travis CI doesn't support
> ARM
> > > arch itself. OpenLab community support it. As I mentioned before,
> OpenLab
> > > is an opensource CI system like travis-ci.[1], it uses opensource CI
> > > project `zuul`[2] for its deployment. Now some opensource project has
> > > intergreted with it already. For example, `contained` project from
> > > CNCF community[3]. And I have a POC for Flink ARM build and test using
> > > OpenLab. Now the build is passed[4], and I'm working on debugging with
> > the
> > > `test` part[5]. Is it fine for Flink to using?
> > >
> > > [1]: https://openlabtesting.org
> > > [2]: https://zuul-ci.org/docs/zuul/
> > > [3]: https://status.openlabtesting.org/projects
> > > [4]:
> > >
> https://status.openlabtesting.org/build/2aa33f1a87854679b70f36bd6f75a890
> > > [5]: https://github.com/theopenlab/flink/pull/1
> > >
> > >
> > > Stephan Ewen  于2019年7月11日周四 下午9:56写道：
> > >
> > > > I think an ARM release would be cool.
> > > >
> > > > To actually support that properly, we would need something like an
> ARM
> > > > profile for the CI builds (at least in the nightly tests), otherwise
> > ARM
> > > > support would probably be broken frequently.
> > > > Maybe that could be a way to start? Create a Travis CI ARM build (if
> > > > possible) and see what tests pass and which parts of the system would
> > > need
> > > > to be adjusted?
> > > >
> > > > On Thu, Jul 11, 2019 at 9:24 AM Xiyuan Wang <
> wangxiyuan1...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi yun:
> > > > >   I didn't try to build rocksdb with vagrant, but just `make -j8
> > > > > rocksdbjava` directly in an ARM machine.  We hit some issues as
> well.
> > > My
> > > > > colleague has created an issue in rocksdb[1]. Rocksdb doesn't
> > contains
> > > > ARM
> > > > > .so file in his offical jar package. If you have the same request,
> > > let's
> > > > > work together there.
> > > > >
> > > > > Thanks.
> > > > >
> > > &g

Re: [DISCUSS] Removing the flink-mapr-fs module

[Stephan Ewen]

This has been fixed in master and 1.9, see
https://issues.apache.org/jira/browse/FLINK-13499

On Tue, Jul 30, 2019 at 3:28 PM Stephan Ewen  wrote:

> I will open a PR later today, changing the module to use reflection rather
> than a hard MapR dependency.
>
> On Tue, Jul 30, 2019 at 6:40 AM Rong Rong  wrote:
>
>> We've also experienced some issues with our internal JFrog artifactory. I
>> am suspecting some sort of mirroring problem but somehow it only occur to
>> the mapr-fs module.
>> So +1 to remove.
>>
>> On Mon, Jul 29, 2019 at 12:47 PM Stephan Ewen  wrote:
>>
>> > It should be fairly straightforward to rewrite the code to not have a
>> MapR
>> > dependency.
>> > Only one class from the MapR dependency is ever referenced, and we could
>> > dynamically load that class.
>> >
>> > That way we can drop the dependency without dropping the support.
>> >
>> > On Mon, Jul 29, 2019 at 5:33 PM Aljoscha Krettek 
>> > wrote:
>> >
>> > > Just FYI, the MapRFileSystem does have some additional code on our
>> > > (Hadoop)FileSystem class, so it might not be straightforward to use
>> MapR
>> > > with our vanilla HadoopFileSystem.
>> > >
>> > > (Still staying that we should remove it, though).
>> > >
>> > > Aljoscha
>> > >
>> > > > On 29. Jul 2019, at 15:16, Simon Su  wrote:
>> > > >
>> > > > +1 to remove it.
>> > > >
>> > > >
>> > > > Thanks,
>> > > > SImon
>> > > >
>> > > >
>> > > > On 07/29/2019 21:00，Till Rohrmann wrote：
>> > > > +1 to remove it.
>> > > >
>> > > > On Mon, Jul 29, 2019 at 1:27 PM Stephan Ewen 
>> wrote:
>> > > >
>> > > > +1 to remove it
>> > > >
>> > > > One should still be able to use MapR in the same way as any other
>> > vendor
>> > > > Hadoop distribution.
>> > > >
>> > > > On Mon, Jul 29, 2019 at 12:22 PM JingsongLee
>> > > >  wrote:
>> > > >
>> > > > +1 for removing it. We never run mvn clean test success in China
>> with
>> > > > mapr-fs...
>> > > > Best, Jingsong Lee
>> > > >
>> > > >
>> > > > --
>> > > > From:Biao Liu 
>> > > > Send Time:2019年7月29日(星期一) 12:05
>> > > > To:dev 
>> > > > Subject:Re: [DISCUSS] Removing the flink-mapr-fs module
>> > > >
>> > > > +1 for removing it.
>> > > >
>> > > > Actually I encountered this issue several times. I thought it might
>> be
>> > > > blocked by firewall of China :(
>> > > >
>> > > > BTW, I think it should be included in release notes.
>> > > >
>> > > >
>> > > > On Mon, Jul 29, 2019 at 5:37 PM Aljoscha Krettek <
>> aljos...@apache.org>
>> > > > wrote:
>> > > >
>> > > > If we remove it, that would mean it’s not supported in Flink 1.9.0,
>> > > > yes.
>> > > > Or we only remove it in Flink 1.10.0.
>> > > >
>> > > > Aljoscha
>> > > >
>> > > > On 29. Jul 2019, at 11:35, Biao Liu  wrote:
>> > > >
>> > > > Hi Aljoscha,
>> > > >
>> > > > Does it mean the MapRFileSystem is no longer supported since 1.9.0?
>> > > >
>> > > > On Mon, Jul 29, 2019 at 5:19 PM Ufuk Celebi  wrote:
>> > > >
>> > > > +1
>> > > >
>> > > >
>> > > > On Mon, Jul 29, 2019 at 11:06 AM Jeff Zhang 
>> > > > wrote:
>> > > >
>> > > > +1 to remove it.
>> > > >
>> > > > Aljoscha Krettek  于2019年7月29日周一 下午5:01写道：
>> > > >
>> > > > Hi,
>> > > >
>> > > > Because of recent problems in the dependencies of that module [1]
>> > > > I
>> > > > would
>> > > > suggest that we remove it. If people are using it, they can use
>> > > > the
>> > > > one
>> > > > from Flink 1.8.
>> > > >
>> > > > What do you think about it? It would a) solve the dependency
>> > > > problem
>> > > > and
>> > > > b) make our build a tiny smidgen more lightweight.
>> > > >
>> > > > Aljoscha
>> > > >
>> > > > [1]
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > >
>> >
>> https://lists.apache.org/thread.html/16c16db8f4b94dc47e638f059fd53be936d5da423376a8c1092eaad1@%3Cdev.flink.apache.org%3E
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Best Regards
>> > > >
>> > > > Jeff Zhang
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > >
>> > >
>> >
>>
>

Re: [VOTE] Publish the PyFlink into PyPI

[Stephan Ewen]

+1 (binding)

On Thu, Aug 1, 2019 at 9:52 AM Dian Fu  wrote:

> Hi Jincheng,
>
> Thanks a lot for driving this.
> +1 (non-binding).
>
> Regards,
> Dian
>
> > 在 2019年8月1日，下午3:24，jincheng sun  写道：
> >
> > Hi all,
> >
> > Publish the PyFlink into PyPI is very important for our user, Please vote
> > on the following proposal:
> >
> > 1. Create PyPI Project for Apache Flink Python API, named: "apache-flink"
> > 2. Release one binary with the default Scala version same with flink
> > default config.
> > 3. Create an account, named "pyflink" as owner(only PMC can manage it).
> PMC
> > can add account for the Release Manager, but Release Manager can not
> delete
> > the release.
> >
> > [ ] +1, Approve the proposal.
> > [ ] -1, Disapprove the proposal, because ...
> >
> > The vote will be open for at least 72 hours. It is adopted by a simple
> > majority with a minimum of three positive votes.
> >
> > See discussion threads for more details [1].
> >
> > Thanks,
> > Jincheng
> >
> > [1]
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Publish-the-PyFlink-into-PyPI-td30095.html
>
>

Re: [DISCUSS] ARM support for Flink

[Stephan Ewen]

Asking INFRA to add support means filing a JIRA ticket.

That works the same way as filing a FLINK Jira ticket, but selecting INFRA
as the project to file the ticket for.

On Thu, Aug 1, 2019 at 4:17 AM Xiyuan Wang  wrote:

> Thanks for your reply.
>
> We are now keeping investigating and debugging Flink on ARM.  It's hard for
> us to say How many kinds of test are enough for ARM support at this moment,
> but `core` and `test` are necessary of cause I think. What we do now is
> following travis-ci, added all the module that tarvis-ci contains.
>
> During out local test, there are just few tests failed[1]. We have
> solutions for some of them, others are still under debugging. Flink team's
> idea is welcome. And very thanks for your jira issue[2], we will keep
> updating it then.
>
> It'll be great if Infra Team could add OpenLab App[3](or other CI if Flink
> choose) to Flink repo. I'm not  clear how to talk with Infra Team, should
> Flink team start the discussion? Or I send a mail list to Infra? Need your
> help.
>
> Then once app is added, perhaps we can add `core` and `test` jobs as the
> first step, making them run stable and successful and then adding more
> modules if needed.
>
> [1]: https://etherpad.net/p/flink_arm64_support
> [2]: https://issues.apache.org/jira/browse/FLINK-13448
> [3]: https://github.com/apps/theopenlab-ci
>
> Regards
> wangxiyuan
>
> Stephan Ewen  于2019年7月31日周三 下午9:46写道：
>
> > Wow, that is pretty nice work, thanks a lot!
> >
> > We need some support from Apache Infra to see if we can connect the Flink
> > Github Repo with the OpenLab CI.
> > We would also need a discussion on the developer mailing list, to get
> > community agreement.
> >
> > Have you looked at whether we need to run all tests with ARM, or whether
> > maybe only the "core" and "tests" profile would be enough to get
> confidence
> > that Flink runs on ARM?
> > Just asking because Flink has a lot of long running tests by now that can
> > easily eat up a lot of CI capacity.
> >
> > Best,
> > Stephan
> >
> >
> >
> > On Tue, Jul 30, 2019 at 3:45 AM Xiyuan Wang 
> > wrote:
> >
> > > Hi Stephan,
> > >   Maybe I misled you in the previous email. We don't need to migrate CI
> > > completely, travis-ci is still there working for X86 arch. What we need
> > to
> > > do is to add another CI tool for ARM arch.
> > >
> > >   There are some ways to do it. As I wrote on
> > > https://issues.apache.org/jira/browse/FLINK-13199 to @Chesnay:
> > >
> > > 1. Add OpenLab CI system for ARM arch test.OpenLab is very similar with
> > > travis-ci. What Flilnk need to do is adding the openlab github app to
> the
> > > repo, then add the job define files inner Flink repo, Here is a POC by
> > me:
> > > https://github.com/theopenlab/flink/pull/1
> > > 2. OpenLab will donate ARM resouces to Apache Infra team as well. Then
> > > Flink can use the Apache offical  Jenkins system for Flink ARM test in
> > the
> > > future. https://builds.apache.org/
> > > 3. Use Drony CI which support ARM arch as well. https://drone.io/
> > >
> > > Since I'm from OpenLab community, if Flink choose OpenLab CI, My
> OpenLab
> > > colleague and I can keep helping and maintaining the ARM CI job. If
> > choose
> > > the 2nd way, the CI maintainance work may be handled by apache-infra
> > team I
> > > guess.  If choose the 3rd Drony CI, what we can help is very limited.
> > > AFAIK, Drony use container for CI test, which may not satisfy some
> > > requiremnts. And OpenLab use VM for test.
> > >
> > > Need Flink core team's decision and reply.
> > >
> > > Thanks.
> > >
> > >
> > > Stephan Ewen  于2019年7月29日周一 下午6:05写道：
> > >
> > > > I don't think it is feasible for Flink to migrate CI completely.
> > > >
> > > > Is there a way to add ARM tests on an external CI in addition?
> > > > @Chesnay what do you think?
> > > >
> > > >
> > > > On Fri, Jul 12, 2019 at 4:45 AM Xiyuan Wang <
> wangxiyuan1...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Stephan
> > > > >   yeah, we should add an ARM CI first. But Travis CI doesn't
> support
> > > ARM
> > > > > arch itself. OpenLab community support it. As I mentioned before,
> > > OpenLab
> > > > > is an opensource CI system like travis-ci.[1], it use

[DISCUSS] Backport FLINK-13326 to 1.9 release

[Stephan Ewen]

Hi all!

I would like to backport a minor chance from 'master' to 'release-1.9'.
It is a very minor change

I am checking here because this is not technically a bug fix, but a way of
exposing the raw keyed state stream in tasks a bit different. It would
unblock some work in a project that tries to fix the fault tolerance of
streaming iterations.

Any concerns?

Best,
Stephan

Re: [DISCUSS] Backport FLINK-13326 to 1.9 release

[Stephan Ewen]

For clarification: It does not fix the loop fault tolerance in the existing
API. There, the concepts are a bit messy and this needs a bigger overhaul.

It is a building block for building fault tolerant loops in downstream
projects.

On Thu, Aug 1, 2019 at 11:45 AM Becket Qin  wrote:

> +1 as well. If this affects the fault tolerance of streaming iteration, I'd
> consider this as a bug fix.
>
> On Thu, Aug 1, 2019 at 11:44 AM Till Rohrmann 
> wrote:
>
> > I've quickly glanced over the changes and I would be ok with backporting
> it
> > if it helps fixing fault tolerance of streaming iterations. Hence +1 from
> > my side.
> >
> > Cheers,
> > Till
> >
> > On Thu, Aug 1, 2019 at 11:20 AM Ufuk Celebi  wrote:
> >
> > > Thanks for checking. No concerns on my side. +1 to back port. Fixing
> > fault
> > > tolerance of streaming iterations sounds like a very valuable thing to
> > > unblock with this release.
> > >
> > > – Ufuk
> > >
> > >
> > > On Thu, Aug 1, 2019 at 11:02 AM Stephan Ewen  wrote:
> > >
> > > > Hi all!
> > > >
> > > > I would like to backport a minor chance from 'master' to
> 'release-1.9'.
> > > > It is a very minor change
> > > >
> > > > I am checking here because this is not technically a bug fix, but a
> way
> > > of
> > > > exposing the raw keyed state stream in tasks a bit different. It
> would
> > > > unblock some work in a project that tries to fix the fault tolerance
> of
> > > > streaming iterations.
> > > >
> > > > Any concerns?
> > > >
> > > > Best,
> > > > Stephan
> > > >
> > >
> >
>

Re: [DISCUSS] Flink project bylaws

[Stephan Ewen]

I added a clarification to the table, clarifying that the current phrasing
means that committers do not need another +1 for their commits.

On Mon, Jul 29, 2019 at 2:11 PM Fabian Hueske  wrote:

> Hi Becket,
>
> Thanks a lot for pushing this forward and addressing the feedback.
> I'm very happy with the current state of the bylaws.
> +1 to wait until Friday before starting a vote.
>
> Best, Fabian
>
> Am Mo., 29. Juli 2019 um 13:47 Uhr schrieb Becket Qin <
> becket@gmail.com
> >:
>
> > Hi Everyone,
> >
> > Thanks for all the discussion and feedback.
> >
> > It seems that we have almost reached consensus. I'll leave the discussion
> > thread open until this Friday. If there is no more concerns raised, I'll
> > start a voting thread after that.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Fri, Jul 26, 2019 at 6:49 PM Yu Li  wrote:
> >
> > > Hi Becket,
> > >
> > > Thanks for noticing and resolving my comment around PMC removal and ASF
> > > rules of PMC membership change process, which you seem to neglect in
> the
> > > summary of updates (smile).
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Wed, 24 Jul 2019 at 04:32, Becket Qin  wrote:
> > >
> > > > Hi folks,
> > > >
> > > > Thanks for all the feedback.
> > > >
> > > > It seems that there are a few concerns over the emeritus status
> after 6
> > > > months of inactivity. Given that the main purpose is just to make
> sure
> > > 2/3
> > > > majority can pass and we sort of have a solution for that, I just
> > updated
> > > > the draft with the following changes:
> > > >
> > > > 1. Removed the inactivity term for emeritus committers / PMCs. A
> > > committer
> > > > / PMC will only be considered emeritus by their own claim.
> > > > 2. Removed the approval process for reinstatement of the emeritus
> > > > committers / PMCs. An emeritus committer / PMC will be reinstated
> when
> > > they
> > > > send an email to the priv...@flink.apache.org.
> > > > 3. Adde the term to ensure 2/3 majority voting is still doable when
> > there
> > > > are non-emeritus committers / PMCs who do not cast the vote.
> > > >
> > > > Please let me know if you have any further thoughts.
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > > On Tue, Jul 23, 2019 at 10:18 AM Becket Qin 
> > > wrote:
> > > >
> > > > > Hi Fabian,
> > > > >
> > > > > Thanks for the feedback.
> > > > >
> > > > > I agree that if we don't like emeritus committers / PMCs, we don't
> > have
> > > > to
> > > > > do it. That said, emeritus status simply means whether an
> individual
> > is
> > > > > active / inactive in the community. It does not make the merits
> > earned
> > > to
> > > > > go away. For our purpose, emeritus status is mostly just a way to
> > make
> > > > 2/3
> > > > > majority possible. As you noticed, since reaching out to binding
> > voters
> > > > who
> > > > > have not voted can achieve the same goal, the emeritus status is
> more
> > > of
> > > > an
> > > > > optimization so we don't have to ping the inactive binding voters
> > every
> > > > > time and wait for long. However, given that 2/3 majority votings
> are
> > > > rare,
> > > > > such communication cost is probably OK. So I think we can remove
> that
> > > > > emeritus part from the bylaws.
> > > > >
> > > > > 1) We should add to the requirements of the PMC that they need to
> > make
> > > > >> sure the project complies with brand issues and reports misuse of
> > ASF
> > > > >> brands.
> > > > >
> > > > > Good point. Added.
> > > > >
> > > > > 2) Do we want to restrict voting days to working days, i.e., a 3
> day
> > > vote
> > > > >> that starts on Friday 11:00am ends on Wednesday 11:00am?
> > > > >
> > > > > This might be a little tricky because people are from countries in
> > > > > different time zones and with different holidays, and so on. If we
> > are
> > > > > worrying about 3 days minimum length is not enough for those who
> want
> > > to
> > > > > give feedback, we can make it 5 days.
> > > > >
> > > > > 3) Do we need a process do decide about removal of features (like
> the
> > > > >> DataSet API for instance or the legacy DataSet/DataStream Python
> > > APIs)?
> > > > >
> > > > > I assume such action should be covered by FLIP as it is a change to
> > the
> > > > > API and probably needs a migration plan. It would be useful to
> have a
> > > > > formal deprecation procedure. But that might be better to be put
> into
> > > > > somewhere else because the bylaws are primarily focusing on the
> > > > > non-technical rules, whereas the deprecation seems more on the
> > > technical
> > > > > side.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > >
> > > > > On Mon, Jul 22, 2019 at 11:42 PM Fabian Hueske 
> > > > wrote:
> > > > >
> > > > >> Hi all,
> > > > >>
> > > > >> First of all thank you very much Becket for starting this
> > discussion!
> > > > >> I think it is a very good idea and overdue to formally define some
> > of
> > > > our
> > > > >> comm

[DISCUSS] Merging new features post-feature-freeze

[Stephan Ewen]

Hi all!

I would like to bring this topic up, because we saw quite a few "secret"
post-feature-freeze feature merges.
The latest example was https://issues.apache.org/jira/browse/FLINK-13225

I would like to make sure that we are all on the same page on what a
feature freeze means and how to handle possible additions after the feature
freeze.
My understanding was the following, and I assume that this was also the
understanding of the community when we started establishing the release
practice:

  - Feature freeze is the date until new features can be merged.
  - After the feature freeze, we only merge bug fixes, release relevant
tests (end to end tests), and documentation.
  - Features should already be stable and have tests. It is not okay to
"get a foot in the door" before feature freeze by merging something that is
not ready (as a placeholder) and then fixing it post feature freeze.
  - Extending functionality to new components is not a bug fix, it is a
feature.
  - If someone wants to add a minor feature after the feature freeze, and
there is a good reason for that, it should be explicitly discussed. If
there is no objection, it can be merged.

Please let me know if you have a different understanding of what feature
freeze means.

Regarding the issue of FLINK-13225
?
  - Should we keep it?
  - Should we revert it in the release-1.9 branch and only keep it for
master?

Best,
Stephan

Re: [DISCUSS] Merging new features post-feature-freeze

[Stephan Ewen]

e to
> >>> discuss what
> >>> does *feature freeze* really means. At least to me, seems I have some
> >>> misunderstandings with this comparing to other community members. But
> as
> >>> you
> >>> pointed out in the jira and also in this mail, I think your
> understanding
> >>> makes sense
> >>> to me.
> >>>
> >>> Maybe we can have a conclusion in the thread and put this into the
> >> project
> >>> bylaws
> >>> which are under discussion?
> >>>
> >>> Regarding to FLINK-13225, I would like to hear other's opinion since I
> >>> merged it. But
> >>> I would like to revert it if someone voted for reverting it.
> >>>
> >>> Sorry for the inconvenience I caused.
> >>>
> >>> Best,
> >>> Kurt
> >>>
> >>>
> >>> On Thu, Aug 8, 2019 at 4:46 PM Stephan Ewen  wrote:
> >>>
> >>>> Hi all!
> >>>>
> >>>> I would like to bring this topic up, because we saw quite a few
> "secret"
> >>>> post-feature-freeze feature merges.
> >>>> The latest example was
> >> https://issues.apache.org/jira/browse/FLINK-13225
> >>>> I would like to make sure that we are all on the same page on what a
> >>>> feature freeze means and how to handle possible additions after the
> >> feature
> >>>> freeze.
> >>>> My understanding was the following, and I assume that this was also
> the
> >>>> understanding of the community when we started establishing the
> release
> >>>> practice:
> >>>>
> >>>> - Feature freeze is the date until new features can be merged.
> >>>> - After the feature freeze, we only merge bug fixes, release
> relevant
> >>>> tests (end to end tests), and documentation.
> >>>> - Features should already be stable and have tests. It is not
> okay to
> >>>> "get a foot in the door" before feature freeze by merging something
> >> that is
> >>>> not ready (as a placeholder) and then fixing it post feature freeze.
> >>>> - Extending functionality to new components is not a bug fix, it
> is a
> >>>> feature.
> >>>> - If someone wants to add a minor feature after the feature
> freeze,
> >> and
> >>>> there is a good reason for that, it should be explicitly discussed. If
> >>>> there is no objection, it can be merged.
> >>>>
> >>>> Please let me know if you have a different understanding of what
> feature
> >>>> freeze means.
> >>>>
> >>>> Regarding the issue of FLINK-13225
> >>>> <https://issues.apache.org/jira/browse/FLINK-13225>?
> >>>> - Should we keep it?
> >>>> - Should we revert it in the release-1.9 branch and only keep it
> for
> >>>> master?
> >>>>
> >>>> Best,
> >>>> Stephan
> >>>>
> >>
>
>

Re: [DISCUSS] Flink Docker Playgrounds

[Stephan Ewen]

I remember that Patrick (who maintained the docker-flink images so far)
frequently raised the point that its good practice to have the images
decoupled from the project release cycle.
Changes to the images can be done frequently and released fast that way.

In addition, one typically supports images for multiple releases, meaning
the versioning is different than in the main code base.

On Thu, Aug 8, 2019 at 6:35 PM Till Rohrmann  wrote:

> I would be in favour of option 1.
>
> We could also think about making the flink-playgrounds and the Flink docker
> image release part of the Flink release process [1] if we don't want to
> have independent release cycles. I think at the moment the official Flink
> docker image is too often forgotten.
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release
>
> Cheers,
> Till
>
> On Thu, Aug 8, 2019 at 6:25 PM Seth Wiesman  wrote:
>
> > Hey Fabian,
> >
> > I support option 1.
> >
> >  As per FLIP-42, playgrounds are going to become core to flinks getting
> > started experience and I believe it is worth the effort to get this
> right.
> >
> > - As you mentioned, we may (and in my opinion definitely will) add more
> > images in the future. Setting up an integration now will set the stage
> for
> > those future additions.
> >
> > - These images will be many users first exposure to Flink and having a
> > proper release cycle to ensure they work properly may be worth the effort
> > in and of itself. We already found during the first PR to that repo that
> we
> > needed to find users with different OSs to test.
> >
> > - Similarly to the above point, having the images hosted under an
> official
> > Apache account adds a certain amount of credibility and shows the
> community
> > that we take on-boarding new users seriously.
> >
> > - I am generally opposed having the official flink docs rely on something
> > that is hosted under someone’s personal account. I don’t want bug fixes
> or
> > updates to be blocked by your (or some else’s ) availability.
> >
> > Seth
> >
> > > On Aug 8, 2019, at 10:36 AM, Fabian Hueske  wrote:
> > >
> > > Hi everyone,
> > >
> > > As you might know, some of us are currently working on Docker-based
> > > playgrounds that make it very easy for first-time Flink users to try
> out
> > > and play with Flink [0].
> > >
> > > Our current setup (still work in progress with some parts merged to the
> > > master branch) looks as follows:
> > > * The playground is a Docker Compose environment [1] consisting of
> Flink,
> > > Kafka, and Zookeeper images (ZK for Kafka). The playground is based on
> a
> > > specific Flink job.
> > > * We had planned to add the example job of the playground as an example
> > to
> > > the flink main repository to bundle it with the Flink distribution.
> > Hence,
> > > it would have been included in the Docker-hub-official (soon to be
> > > published) Flink 1.9 Docker image [2].
> > > * The main motivation of adding the job to the examples module in the
> > flink
> > > main repo was to avoid the maintenance overhead for a customized Docker
> > > image.
> > >
> > > When discussing to backport the playground job (and its data generator)
> > to
> > > include it in the Flink 1.9 examples, concerns were raised about their
> > > Kafka dependency which will become a problem, if the community agrees
> on
> > > the recently proposed repository split, which would remove flink-kafka
> > from
> > > the main repository [3]. I think this is a fair concern, that we did
> not
> > > consider when designing the playground (also the repo split was not
> > > proposed yet).
> > >
> > > If we don't add the playground job to the examples, we need to put it
> > > somewhere else. The obvious choice would be the flink-playgrounds [4]
> > > repository, which was intended for the docker-compose configuration
> > files.
> > > However, we would not be able to include it in the Docker-hub-official
> > > Flink image any more and would need to maintain a custom Docker image,
> > what
> > > we tried to avoid. The custom image would of course be based on the
> > > Docker-hub-official Flink image.
> > >
> > > There are different approaches for this:
> > >
> > > 1) Building one (or more) official ASF images
> > > There is an official Apache Docker Hub user [5] and a bunch of projects
> > > publish Docker images via this user. Apache Infra seems to support an
> > > process that automatically builds and publishes Docker images when a
> > > release tag is added to a repository. This feature needs to be
> enabled. I
> > > haven't found detailed documentation on this but there is a bunch of
> > INFRA
> > > Jira tickets that discuss this mechanism.
> > > This approach would mean that we need a formal Apache release for
> > > flink-playgrounds (similar to flink-shaded). The obvious benefits are
> > that
> > > these images would be ASF-official Docker images. In case we can
> publish
> > > more than one image per repo, we could also publish images f

Re: [DISCUSS] Repository split

[Stephan Ewen]

Thank you all for the good discussion.

I was one of the folks that thinking about such a repository split together
with Chesnay, but due to lack of prior experience, happy to hear all the
points that

Let's investigate a bit what would be alternatives to this that solve the
two problems.

   (1) Build time: Seems that some sophisticated CI setup can go a long way
there. Would that also be possible on Travis (via artifact caching?)

   (2) Huge amount of PRs against the same repo, PRs getting lost, fast
moving master makes it hard to catch up and have a clean rebase-test-merge
run.



Just as a thought experiment: What if we "mimic" the repo split in the mono
repo, by re-arranging the modules in the mono-repo as they would be in
separate repos?
A developer could simply "cd flink-main" and "mvn clean verify" in there.
Would be a simple experience, no need to understand build profiles, etc.

Would that help us with anything? Maybe even with point (2), if we start
treating the top level directories as organizational units?

flink
   +-- flink-main
   +-- flink-core
   +-- flink-datastream-java
   +-- flink-runtime
   +-- etc
   +-- flink-connectors
   +-- flink-kafka
   +-- flink-kinesis
   +-- etc
   +-- flink-libs
   +-- flink-ml
   +-- flink-
   +-- flink-core
   +-- etc

Best,
Stephan






On Sun, Aug 11, 2019 at 4:36 PM Maximilian Michels  wrote:

> Apart from a technical explanation, the initial suggestion does not
> propose how the repository should be split up. The only meaningful split I
> see is for the connectors.
>
> This discussion dates back a few years:
> https://lists.apache.org/thread.html/4ee502667a5801d23d76a01406e747e1a934417dc67ef7d26fb7f79c@1449757911@%3Cdev.flink.apache.org%3E
>
> I would be in favor of keeping the mono repository. Like already mentioned
> here, there are other ways to resolve build time issue. For instance, in
> Beam we have granular build triggers that allow to test only specific
> components and their dependencies:
> https://github.com/apache/beam/blob/a2b57e3b55a09d641cee8c3b796cc6971a008db0/.test-infra/jenkins/job_PreCommit_Java.groovy#L26
>
> Thanks,
> Max
>
> On 09.08.19 09:14, Biao Liu wrote:
> > Hi folks,
> >
> > Thanks for bringing this discussion Chesnay.
> >
> > +1 for the motivation. It's really a bad experience of waiting Travis
> > building for a long time.
> >
> > WRT the solution, personally I agree with Dawid/David.
> >
> > IMO the biggest benefit of splitting repository is reducing build time. I
> > think it could be achieved without splitting the repository. That's the
> > best solution for me.
> >
> > And there would be several pains I do really care about if we split the
> > repository.
> >
> > 1. Most of our users are developer. The non-developer users probably do
> not
> > care the code structure at all. They might use the released binary file
> > directly. For developers, the multiple repositories are not so friendly
> to
> > read, build or test the codes. I think it's a big regression.
> >
> > 2. It's definitely a nightmare to work across repositories. As Piotr
> said,
> > it's should be a rare case. However Jack raised a good example,
> debugging a
> > sub-repository IT case. Image the scenario, I'm debugging an unstable
> Kafka
> > IT case. I need to add some logs in runtime components to find some
> clues.
> > What should I do? I have to locally install the flink-main project for
> each
> > time after adding logs. And it's easy to make mistakes with switching
> > between repositories time after time.
> >
> > To sum up, at least for now I agree with Dawid that we should go toward
> > splitting the CI builds not the repository.
> >
> > Thanks,
> > Biao /'bɪ.aʊ/
> >
> >
> >
> > On Fri, Aug 9, 2019 at 12:55 AM Jark Wu  wrote:
> >
> > > Hi,
> > >
> > > First of all, I agree with Dawid and David's point.
> > >
> > > I will share some experience on the repository split. We have been
> through
> > > it for Alibaba Blink, which is the most worthwhile project to learn
> from I
> > > think.
> > > We split Blink project into "blink-connectors" and "blink", but we
> didn't
> > > get much benefit for better development process. In the contrary, it
> slow
> > > down the development sometimes.
> > > We have suffered from the following issues after split as far as I can
> see:
> > >
> > > 1. Unstable build and test:
> > > The interface or behavior changes in the underlying (e.g. core, table)
> will
> > > lead to build fail and tests fail in the connectors repo. AFAIK, table
> api
> > > are still under heavy evolution.
> > > This will make connectors repo more unstable and makes us busy to fix
> the
> > > build problems and tests problems **after-commit**.
> > > First, it's not easy to locate which commit of main repo lead to the
> > > connectors repo fail (we have over 70+ commits every day in flink
> master
> > > now and it is growing).
> > > Second, when 2 or 3 build/t

Re: [DISCUSS] Repository split

[Stephan Ewen]

Just in case we decide to pursue the repo split in the end, some thoughts
on Chesnay's questions:

(1) Git History

We can also use "git filter-branch" to rewrite the history to only contain
the connectors.
It changes commit hashes, but not sure that this is a problem. The commit
hashes are still valid in the main repo, so one can look up the commits
that fixed an earlier issue.

(2) Maven

+1 to a shared flink-parent pom.xml file

(3) Docs

One option would be to not integrate the docs.
That would mean a top level navigation between Flink, Connectors, Libraries
(for example as a horizontal bar at the top) and then per repository
navigation as we currently have it.
Of course, sharing docs build setup would be desirable.

(4) End-2-End tests

I think we absolutely need those on the other repos.
As Piotr pointed out, some of the end to end test coverage depends on
connectors and libraries.

While ideally that would not be necessary, I believe that realistically,
targeted test coverage in the core will never absolutely perfect. So a
certain amount of additional coverage (especially for bugs due to
distributed race conditions) will be caught by the extended test coverage
we get from connector and library end-to-end tests.

  Let's find a way to keep that, maybe not as per-commit tests, but as
nightly ones.

On Wed, Aug 7, 2019 at 1:14 PM Chesnay Schepler  wrote:

> Hello everyone,
>
> The Flink project sees an ever-increasing amount of dev activity, both
> in terms of reworked and new features.
>
> This is of course an excellent situation to be in, but we are getting to
> a point where the associate downsides are becoming increasingly
> troublesome.
>
> The ever increasing build times, in addition to unstable tests,
> significantly slow down the develoment process.
> Additionally, pull requests for smaller features frequently slip through
> the crasks as they are being buried under a mountain of other pull
> requests.
>
> As a result I'd like to start a discussion on splitting the Flink
> repository.
>
> In this mail I will outline the core idea, and what problems I currently
> envision.
>
> I'd specifically like to encourage those who were part of similar
> initiatives in other projects to share the experiences and ideas.
>
>
> General Idea
>
> For starters, the idea is to create a new repository for
> "flink-connectors".
> For the remainder of this mail, the current Flink repository is referred
> to as "flink-main".
>
> There are also other candidates that we could discuss in the future,
> like flink-libraries (the next top-priority repo to ease flink-ml
> development), metric reporters, filesystems and flink-formats.
>
> Moving out flink-connectors provides the most benefits, as we straight
> away save at-least an hour of testing time, and not being included in
> the binary distribution simplifies a few things.
>
>
> Problems to solve
>
> To make this a reality there's a number of questions we have to discuss;
> some in the short-term, others in the long-term.
>
> 1) Git history
>
> We have to decide whether we want to rewrite the history of sub
> repositories to only contain diffs/commits related to this part of
> Flink, or whether we just fork from some commit in flink-main and
> add a commit to the connector repo that "transforms" it from
> flink-main to flink-connectors (i.e., remove everything unrelated to
> connectors + update module structure etc.).
>
> The latter option would have the advantage that our commit book
> keeping in JIRA would still be correct, but it would create a
> significant divide between the current and past state of the
> repository.
>
> 2) Maven
>
> We should look into whether there's a way to share dependency/plugin
> configurations and similar, so we don't have to keep them in sync
> manually across multiple repositories.
>
> A new parent Flink pom that all repositories define as their parent
> could work; this would imply splicing out part of the current room
> pom.xml.
>
> 3) Documentation
>
> Splitting the repository realistically also implies splitting the
> documentation source files (At the beginning we can get by with
> having it still in flink-main).
> We could just move the relevant files to the respective repository
> (while maintaining the directory structure), and merge them when
> building the docs.
>
> We also have to look at how we can handle java-/scaladocs; e.g.
> whether it is possible to aggregate them across projects.
>
> 4) CI (end-to-end tests)
>
> The very basic question we have to answer is whether we want E2E
> tests in the sub repositories. If so, we need to find a way to share
> e2e-tooling.
>
> 5) Releases
>
> We have to discuss how our release process will look like. This may
> also have repercussions on how repositories may depend on each other
> (SNAPSHOT vs LATEST). Note that this should be discussed for each
> 

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

[Stephan Ewen]

Hi Gyula!

Thanks for reporting this.

Can you try to simply build Flink without Hadoop and then exporting
HADOOP_CLASSPATH to your CloudEra libs?
That is the recommended way these days.

Best,
Stephan



On Mon, Aug 12, 2019 at 10:48 AM Gyula Fóra  wrote:

> Thanks Dawid,
>
> In the meantime I also figured out that I need to build the
> https://github.com/apache/flink-shaded project locally with
> -Dhadoop.version set to the specific hadoop version if I want something
> different.
>
> Cheers,
> Gyula
>
> On Mon, Aug 12, 2019 at 9:54 AM Dawid Wysakowicz 
> wrote:
>
> > Hi Gyula,
> >
> > As for the issues with mapr maven repository, you might have a look at
> > this message:
> >
> >
> https://lists.apache.org/thread.html/77f4db930216e6da0d6121065149cef43ff3ea33c9ffe9b1a3047210@%3Cdev.flink.apache.org%3E
> >
> > Try using the "unsafe-mapr-repo" profile.
> >
> > Best,
> >
> > Dawid
> >
> > On 11/08/2019 19:31, Gyula Fóra wrote:
> > > Hi again,
> > >
> > > How do I build the RC locally with the hadoop version specified? Seems
> > like
> > > no matter what I do I run into dependency problems with the shaded
> hadoop
> > > dependencies.
> > > This seems to have worked in the past.
> > >
> > > There might be some documentation somewhere that I couldnt find, so I
> > would
> > > appreciate any pointers :)
> > >
> > > Thanks!
> > > Gyula
> > >
> > > On Sun, Aug 11, 2019 at 6:57 PM Gyula Fóra 
> wrote:
> > >
> > >> Hi!
> > >>
> > >> I am trying to build 1.9.0-rc2 with the -Pvendor-repos profile
> enabled.
> > I
> > >> get the following error:
> > >>
> > >> mvn clean install -DskipTests -Pvendor-repos -Dhadoop.version=2.6.0
> > >> -Pinclude-hadoop (ignore that the hadoop version is not a vendor
> hadoop
> > >> version)
> > >>
> > >> [ERROR] Failed to execute goal on project flink-hadoop-fs: Could not
> > >> resolve dependencies for project
> > >> org.apache.flink:flink-hadoop-fs:jar:1.9.0: Failed to collect
> > dependencies
> > >> at org.apache.flink:flink-shaded-hadoop-2:jar:2.6.0-7.0: Failed to
> read
> > >> artifact descriptor for
> > >> org.apache.flink:flink-shaded-hadoop-2:jar:2.6.0-7.0: Could not
> transfer
> > >> artifact org.apache.flink:flink-shaded-hadoop-2:pom:2.6.0-7.0 from/to
> > >> mapr-releases (https://repository.mapr.com/maven/):
> > >> sun.security.validator.ValidatorException: PKIX path building failed:
> > >> sun.security.provider.certpath.SunCertPathBuilderException: unable to
> > find
> > >> valid certification path to requested target -> [Help 1]
> > >>
> > >> This looks like a TLS error. Might not be related to the release but
> it
> > >> could be good to know.
> > >>
> > >> Cheers,
> > >> Gyula
> > >>
> > >> On Fri, Aug 9, 2019 at 6:26 PM Tzu-Li (Gordon) Tai <
> tzuli...@apache.org
> > >
> > >> wrote:
> > >>
> > >>> Please note that the unresolved issues that are still tagged with a
> fix
> > >>> version "1.9.0", as seen in the JIRA release notes [1], are issues to
> > >>> update documents for new features.
> > >>> I've left them still associated with 1.9.0 since these should still
> be
> > >>> updated for 1.9.0 soon along with the official release.
> > >>>
> > >>> [1]
> > >>>
> > >>>
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344601
> > >>>
> > >>> On Fri, Aug 9, 2019 at 6:17 PM Tzu-Li (Gordon) Tai <
> > tzuli...@apache.org>
> > >>> wrote:
> > >>>
> >  Hi all,
> > 
> >  Release candidate #2 for Apache Flink 1.9.0 is now ready for your
> > >>> review.
> >  This is the first voting candidate for 1.9.0, following the preview
> >  candidates RC0 and RC1.
> > 
> >  Please review and vote on release candidate #2 for version 1.9.0, as
> >  follows:
> >  [ ] +1, Approve the release
> >  [ ] -1, Do not approve the release (please provide specific
> comments)
> > 
> >  The complete staging area is available for your review, which
> > includes:
> >  * JIRA release notes [1],
> >  * the official Apache source release and binary convenience releases
> > to
> > >>> be
> >  deployed to dist.apache.org [2], which are signed with the key with
> >  fingerprint 1C1E2394D3194E1944613488F320986D35C33D6A [3],
> >  * all artifacts to be deployed to the Maven Central Repository [4],
> >  * source code tag “release-1.9.0-rc2” [5].
> > 
> >  Robert is also preparing a pull request for the announcement blog
> post
> > >>> in
> >  the works, and will update this voting thread with a link to the
> pull
> >  request shortly afterwards.
> > 
> >  The vote will be open for *at least 72 hours*.
> >  Please cast your votes before *Aug. 14th (Wed.) 2019, 17:00 PM
> CET*.It
> > >>> is
> >  adopted by majority approval, with at least 3 PMC affirmative votes.
> >  Thanks,
> >  Gordon[1]
> > 
> > >>>
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344601
> >  [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.9.0-rc2/

Re: [DISCUSS] Integrate new SourceReader with Mailbox Model in StreamTask

[Stephan Ewen]

+1 to looking at the Source Reader interface as converged with respect to
its integration with the runtime.

Especially the semantics around the availability future and "emitNext" seem
to have reach consensus.

On Sat, Aug 10, 2019 at 10:51 PM zhijiang
 wrote:

>
> Hi all,
>
> As mentioned in FLIP-27[1], the proposed SourceReader interface is part of
> refactoring source interface.
> AFAIK FLIP-27 has not been voted yet and might still need more time as a
> whole compared to the
>
>
> SourceReader discussion which seems more or less converged.
>
>
> We would like to start an initial progress on runtime side based on the
> SourceReader interface to integrate
> the execution of SourceReader with existing mailbox thread model[2] in
> StreamTask. It is one of the steps for
>
>
> finally integrating all the actions in task (processing timer, checkpoint)
> into unified mailbox model. The benefits
>
> are simpler processing logics because only one single thread handles all
> the actions without concurrent issue,
>
> and further getting rid of lock dependency which might cause unfair lock
> concern in checkpoint process.
>
> We still need to support the legacy source in some releases which would
> probably be used for a while, especially
> for the scenario of performance concern.
>
> Welcome any feedbacks or comments in design doc[3].
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
> [2]
> https://docs.google.com/document/d/1eDpsUKv2FqwZiS1Pm6gYO5eFHScBHfULKmH1-ZEWB4g/edit
> [3]
> https://docs.google.com/document/d/13x9M7k1SRqkOFXP0bETcJemIRyJzoqGgkdy11pz5qHM/edit?usp=sharing
>
> Best,
> Zhijiang
>

Re: Watermarking in Src and Timestamping downstream

[Stephan Ewen]

Do you know what part of the code happens to block off your watermark?
Maybe a method that is overridden in AbstractStreamOperator in your code?

On Sat, Aug 10, 2019 at 4:06 AM Roshan Naik 
wrote:

> Have streaming use cases where it is useful & easier to generate the
> watermark in the Source (via ctx.emitWatermark() ) and assign timestamps
> in a downstream custom operator which calls  output.collect(new
> StreamRecord(msg, time)).
> When doing so, I see that the watermark reaches the downstream operator,
> but does not flow past it and consequently further downstream windows
> remain open. (I am using Flink 1.6).
> -roshan
>
>

Re: [VOTE] Flink Project Bylaws

[Stephan Ewen]

+1

On Tue, Aug 13, 2019 at 12:22 PM Maximilian Michels  wrote:

> +1 It's good that we formalize this.
>
> On 13.08.19 10:41, Fabian Hueske wrote:
> > +1 for the proposed bylaws.
> > Thanks for pushing this Becket!
> >
> > Cheers, Fabian
> >
> > Am Mo., 12. Aug. 2019 um 16:31 Uhr schrieb Robert Metzger <
> > rmetz...@apache.org>:
> >
> > > I changed the permissions of the page.
> > >
> > > On Mon, Aug 12, 2019 at 4:21 PM Till Rohrmann 
> > > wrote:
> > >
> > >> +1 for the proposal. Thanks a lot for driving this discussion Becket!
> > >>
> > >> Cheers,
> > >> Till
> > >>
> > >> On Mon, Aug 12, 2019 at 3:02 PM Becket Qin 
> wrote:
> > >>
> > >>> Hi Robert,
> > >>>
> > >>> That's a good suggestion. Will you help to change the permission on
> > > that
> > >>> page?
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Jiangjie (Becket) Qin
> > >>>
> > >>> On Mon, Aug 12, 2019 at 2:41 PM Robert Metzger 
> > >>> wrote:
> > >>>
> >  Thanks for starting the vote.
> >  How about putting a specific version in the wiki up for voting, or
> >  restricting edit access to the page to the PMC?
> >  There were already two changes (very minor) to the page since the
> > > vote
> > >>> has
> >  started:
> > 
> > 
> > >>>
> > >>
> > >
> https://cwiki.apache.org/confluence/pages/viewpreviousversions.action?pageId=120731026
> >  I suggest to restrict edit access to the page.
> > 
> > 
> > 
> >  On Mon, Aug 12, 2019 at 11:43 AM Timo Walther 
> > >>> wrote:
> > 
> > > +1
> > >
> > > Thanks for all the efforts you put into this for documenting how
> > > the
> > > project operates.
> > >
> > > Regards,
> > > Timo
> > >
> > > Am 12.08.19 um 10:44 schrieb Aljoscha Krettek:
> > >> +1
> > >>
> > >>> On 11. Aug 2019, at 10:07, Becket Qin 
> > >> wrote:
> > >>>
> > >>> Hi all,
> > >>>
> > >>> I would like to start a voting thread on the project bylaws of
> > >>> Flink.
> >  It
> > >>> aims to help the community coordinate more smoothly. Please see
> > >> the
> > > bylaws
> > >>> wiki page below for details.
> > >>>
> > >>>
> > >
> > 
> > >>>
> > >>
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120731026
> > >>>
> > >>> The discussion thread is following:
> > >>>
> > >>>
> > >
> > 
> > >>>
> > >>
> > >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-project-bylaws-td30409.html
> > >>>
> > >>> The vote will be open for at least 6 days. PMC members' votes
> > > are
> > >>> considered as binding. The vote requires 2/3 majority of the
> > >> binding
> > > +1s to
> > >>> pass.
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Jiangjie (Becket) Qin
> > >
> > >
> > >
> > 
> > >>>
> > >>
> > >
> >
>
>

Re: Apache flink 1.7.2 security issues

[Stephan Ewen]

Hi!

Thank you for reporting this!

At the moment, the Flink REST endpoint is not secure in the way that you
can expose it publicly. After all, you can submit Flink jobs to it which by
definition support executing arbitrary code.
Given that access to the REST endpoint allows by design arbitrary code
execution (running a Flink job), these reported vulnerabilities are
probably not as critical.

In light of that, the REST endpoint needs to be exposed in a secure way
(SSL mutual auth, an authenticating proxy, etc.).

Nevertheless, let us see whether we can update at least the web UI
dependencies to newer versions which are not subject to these exploits, to
take a step towards making the REST endpoint more suitable to be public
facing.

Best,
Stephan



On Sun, Aug 11, 2019 at 6:20 PM V N, Suchithra (Nokia - IN/Bangalore) <
suchithra@nokia.com> wrote:

> Hello,
>
>
>
> We are using Apache Flink 1.7.2 version. During our security scans
> following issues are reported by our scan tool. Please let us know your
> comments on these issues.
>
>
>
> *[1] 150085 Slow HTTP POST vulnerability*
>
> *Severity *Potential Vulnerability - Level 3
>
> *Group *Information Disclosure
>
>
>
> *Threat*
>
> The web application is possibly vulnerable to a "slow HTTP POST" Denial of
> Service (DoS) attack. This is an application-level DoS that consumes server
>
> resources by maintaining open connections for an extended period of time
> by slowly sending traffic to the server. If the server maintains too many
> connections
>
> open at once, then it may not be able to respond to new, legitimate
> connections.
>
>
>
> *#1 Request*
>
> *Payload *N/A
>
> *Request *POST https://:/
>
> #1 Host: :
>
> #3 Accept: */*
>
> #4 Content-Type: application/x-www-form-urlencoded
>
>
>
> *#1 Response*
>
> Vulnerable to slow HTTP POST attack
>
> Connection with partial POST body remained open for: 312932 milliseconds
>
>
>
> *[2] 150124 Clickjacking - Framable Page (10)*
>
> *Severity *Confirmed Vulnerability - Level 3
>
> *Group *Information Disclosure
>
> *CVSS Base *6.4 *CVSS Temporal*5.8
>
>
>
> *Threat*
>
> The web page can be framed. This means that clickjacking attacks against
> users are possible.
>
>
>
> *#1 Request*
>
> *Payload *N/A
>
> *Request *GET https://:/
>
> #1 Host: :
>
> #3 Accept: */*
>
>
>
> *#1 Response*
>
> The URI was framed.
>
>
>
> Below url’s have also reported the same issues and response was same.
>
>
>
> *Request *GET https://:/partials/jobs/running-jobs.html
>
> *Request *GET https://:/partials/submit.html
>
> *Request *GET https://:/partials/jobmanager/stdout.html
>
> *Request *GET https://:/partials/jobs/completed-jobs.html
>
> *Request *GET https://:/partials/taskmanager/index.html
>
> *Request *GET https://:/partials/jobmanager/log.html
> 
>
> *Request *GET https://:/partials/jobmanager/index.html
>
> *Request *GET https:///partials/overview.html
>
> *Request *GET https://:/partials/jobmanager/config.html
>
>
>
> *[3] 150162 Use of JavaScript Library with Known Vulnerability (4)*
>
>
>
> *Threat*
>
> The web application is using a JavaScript library that is known to contain
> at least one vulnerability.
>
>
>
> *#1 Request*
>
> *Payload *-
>
> *Request *GET https://:/
>
> #1 Host: :
>
> #3 Accept: */*
>
>
>
> *#1 Response*
>
> *Vulnerable javascript library: jQuery*
>
> *version: 2.2.0*
>
> Details:
>
> CVE-2015-9251: jQuery versions on or above 1.4.0 and below 1.12.0 (version
> 1.12.3 and above but below 3.0.0-beta1 as well) are vulnerable to XSS via
> 3rd party text/javascript responses(3rd party
>
> CORS request may execute). (https://github.com/jquery/jquery/issues/2432).
>
> Solution: jQuery version 3.0.0 has been released to address the issue (
> http://blog.jquery.com/2016/01/08/jquery-2-2-and-1-12-released/). Please
> refer to vendor documentation (https://blog.jquery.com/)
>
> for the latest security updates.
>
>
>
> Found on the following pages (only first 10 pages are reported):
>
> https://:/
>
> https://:/#/completed-jobs
>
> https://:/#/jobmanager/config
>
> https://:/#/overview
>
> https://:/#/running-jobs
>
> https://:/#/submit
>
> https://:/#/taskmanagers
>
> https://:/#/jobmanager/log
>
> https://:/#/jobmanager/stdout
>
> https://:/#/taskmanager/100474b27dcd8eeb9f3ff38c952977c9/log
>
>
>
>
>
> *#1 Response*
>
> *Vulnerable javascript library: Angular*
>
> *version: 1.4.8*
>
> Details:
>
> In angular versions below 1.6.5 both Firefox and Safari are vulnerable to
> XSS in $sanitize if an inert document created via
> `document.implementation.createHTMLDocument()` is used. Angular version
>
> 1.6.5 checks for these vulnerabilities and then use a DOMParser or XHR
> strategy if needed. Please refer to vendor documentation (
> https://github.com/angular/angular.js/commit/
>
> 8f31f1ff43b673a24f84422d5c13d6312b2c4d94) for latest security updates.
>
> Found on the following pages (only first 10 pages are reported):
>
> https://:/
>
> https://:/#/

Re: [DISCUSS] Drop stale class Program

[Stephan Ewen]

+1 to drop it.

It's one of the oldest pieces of legacy.

On Wed, Aug 14, 2019 at 12:07 PM Aljoscha Krettek 
wrote:

> Hi,
>
> I would be in favour of removing Program (and the code paths that support
> it) for Flink 1.10. Most users of Flink don’t actually know it exists and
> it is only making our code more complicated. Going forward with the new
> Client API discussions will be a lot easier without it as well.
>
> Best,
> Aljoscha
>
> > On 14. Aug 2019, at 11:08, Kostas Kloudas  wrote:
> >
> > Hi all,
> >
> > It is nice to have this discussion.
> >
> > I am totally up for removing the unused Program interface, as this will
> > simplify a lot of other code paths in the ClusterClient and elsewhere.
> >
> > Also about the easier integration of Flink with other frameworks, there
> > is another discussion in the mailing list with exactly this topic:
> > [DISCUSS] Flink client api enhancement for downstream project
> >
> > Cheers,
> > Kostas
> >
> >
> > On Tue, Jul 30, 2019 at 1:38 PM Zili Chen  wrote:
> >
> >> Hi,
> >>
> >> With a one-week survey in user list[1], nobody expect Flavio and Jeff
> >> participant the thread.
> >>
> >> Flavio shared his experience with a revised Program like interface.
> >> This could be regraded as downstream integration and in client api
> >> enhancements document we propose rich interface for this integration.
> >> Anyway, the Flink scope Program is less functional than it should be.
> >>
> >> With no objection I'd like to push on this thread. We need a committer
> >> participant this thread to shepherd the removal/deprecation of Program,
> a
> >> @PublicEvolving interface. Anybody has spare time? Or anything I can do
> >> to make progress?
> >>
> >> Best,
> >> tison.
> >>
> >> [1]
> >>
> >>
> https://lists.apache.org/thread.html/37445e43729cf7eaeb0aa09133d3980b62f891c5ee69d2c3c3e76ab5@%3Cuser.flink.apache.org%3E
> >>
> >>
> >> Zili Chen  于2019年7月22日周一 下午8:38写道：
> >>
> >>> Hi,
> >>>
> >>> I created a thread for survey in user list[1]. Please take participate
> in
> >>> if interested.
> >>>
> >>> Best,
> >>> tison.
> >>>
> >>> [1]
> >>>
> >>
> https://lists.apache.org/thread.html/37445e43729cf7eaeb0aa09133d3980b62f891c5ee69d2c3c3e76ab5@%3Cuser.flink.apache.org%3E
> >>>
> >>>
> >>> Flavio Pompermaier  于2019年7月19日周五 下午5:18写道：
> >>>
>  +1 to remove directly the Program class (I think nobody use it and
> it's
>  not
>  supported at all by REST services and UI).
>  Moreover it requires a lot of transitive dependencies while it should
> >> be a
>  very simple thing..
>  +1 to add this discussion to "Flink client api enhancement"
> 
>  On Fri, Jul 19, 2019 at 11:14 AM Biao Liu  wrote:
> 
> > To Flavio, good point for the integration suggestion.
> >
> > I think it should be considered in the "Flink client api enhancement"
> > discussion. But the outdated API should be deprecated somehow.
> >
> > Flavio Pompermaier  于2019年7月19日周五 下午4:21写道：
> >
> >> In my experience a basic "official" (but optional) program
> >> description
> >> would be very useful indeed (in order to ease the integration with
>  other
> >> frameworks).
> >>
> >> Of course it should be extended and integrated with the REST
> >> services
>  and
> >> the Web UI (when defined) in order to be useful..
> >> It ease to show to the user what a job does and which parameters it
> >> requires (optional or mandatory) and with a proper help description.
> >> Indeed, when we write a Flink job we implement the following
>  interface:
> >>
> >> public interface FlinkJob {
> >>  String getDescription();
> >>  List getParameters();
> >> boolean isStreamingOrBatch();
> >> }
> >>
> >> public class ClusterJobParameter {
> >>
> >>  private String paramName;
> >>  private String paramType = "string";
> >>  private String paramDesc;
> >>  private String paramDefaultValue;
> >>  private Set choices;
> >>  private boolean mandatory;
> >> }
> >>
> >> This really helps to launch a Flink job by a frontend (if the rest
> > services
> >> returns back those infos).
> >>
> >> Best,
> >> Flavio
> >>
> >> On Fri, Jul 19, 2019 at 9:57 AM Biao Liu 
> >> wrote:
> >>
> >>> Hi Zili,
> >>>
> >>> Thank you for bring us this discussion.
> >>>
> >>> My gut feeling is +1 for dropping it.
> >>> Usually it costs some time to deprecate a public (actually it's
> >>> `PublicEvolving`) API. Ideally it should be marked as `Deprecated`
> > first.
> >>> Then it might be abandoned it in some later version.
> >>>
> >>> I'm not sure how big the burden is to make it compatible with the
> >> enhanced
> >>> client API. If it's a critical blocker, I support dropping it
>  radically
> >> in
> >>> 1.10. Of course a survey is necessary. And the result of survey is
> >>> acceptable.
> >>>
> >>>
>

Re: Checkpointing under backpressure

[Stephan Ewen]

Hi all!

Yes, the first proposal of "unaligend checkpoints" (probably two years back
now) drew a major inspiration from Chandy Lamport, as did actually the
original checkpointing algorithm.

"Logging data between first and last barrier" versus "barrier jumping over
buffer and storing those buffers" is pretty close same.
However, there are a few nice benefits of the proposal of unaligned
checkpoints over Chandy-Lamport.

*## Benefits of Unaligned Checkpoints*

(1) It is very similar to the original algorithm (can be seen an an
optional feature purely in the network stack) and thus can share lot's of
code paths.

(2) Less data stored. If we make the "jump over buffers" part timeout based
(for example barrier overtakes buffers if not flushed within 10ms) then
checkpoints are in the common case of flowing pipelines aligned without
in-flight data. Only back pressured cases store some in-flight data, which
means we don't regress in the common case and only fix the back pressure
case.

(3) Faster checkpoints. Chandy Lamport still waits for all barriers to
arrive naturally, logging on the way. If data processing is slow, this can
still take quite a while.

==> I think both these points are strong reasons to not change the
mechanism away from "trigger sources" and start with CL-style "trigger all".


*## Possible ways to combine Chandy Lamport and Unaligned Checkpoints*

We can think about something like "take state snapshot on first barrier"
and then store buffers until the other barriers arrive. Inside the network
stack, barriers could still overtake and persist buffers.
The benefit would be less latency increase in the channels which already
have received barriers.
However, as mentioned before, not prioritizing the inputs from which
barriers are still missing can also have an adverse effect.


*## Concerning upgrades*

I think it is a fair restriction to say that upgrades need to happen on
aligned checkpoints. It is a rare enough operation.


*## Concerning re-scaling (changing parallelism)*

We need to support that on unaligned checkpoints as well. There are several
feature proposals about automatic scaling, especially down scaling in case
of missing resources. The last snapshot might be a regular checkpoint, so
all checkpoints need to support rescaling.


*## Concerning end-to-end checkpoint duration and "trigger sources" versus
"trigger all"*

I think for the end-to-end checkpoint duration, an "overtake buffers"
approach yields faster checkpoints, as mentioned above (Chandy Lamport
logging still needs to wait for barrier to flow).

I don't see the benefit of a "trigger all tasks via RPC concurrently"
approach. Bear in mind that it is still a globally coordinated approach and
you need to wait for the global checkpoint to complete before committing
any side effects.
I believe that the checkpoint time is more determined by the state
checkpoint writing, and the global coordination and metadata commit, than
by the difference in alignment time between "trigger from source and jump
over buffers" versus "trigger all tasks concurrently".

Trying to optimize a few tens of milliseconds out of the network stack
sends (and changing the overall checkpointing approach completely for that)
while staying with a globally coordinated checkpoint will send us down a
path to a dead end.

To really bring task persistence latency down to 10s of milliseconds (so we
can frequently commit in sinks), we need to take an approach without any
global coordination. Tasks need to establish a persistent recovery point
individually and at their own discretion, only then can it be frequent
enough. To get there, they would need to decouple themselves from the
predecessor and successor tasks (via something like persistent channels).
This is a different discussion, though, somewhat orthogonal to this one
here.

Best,
Stephan


On Wed, Aug 14, 2019 at 12:37 PM Piotr Nowojski  wrote:

> Hi again,
>
> Zhu Zhu let me think about this more. Maybe as Paris is writing, we do not
> need to block any channels at all, at least assuming credit base flow
> control. Regarding what should happen with the following checkpoint is
> another question. Also, should we support concurrent checkpoints and
> subsuming checkpoints as we do now? Maybe not…
>
> Paris
>
> Re
> I. 2. a) and b) - yes, this would have to be taken into an account
> I. 2. c) and IV. 2. - without those, end to end checkpoint time will
> probably be longer than it could be. It might affect external systems. For
> example Kafka, which automatically time outs lingering transactions, and
> for us, the transaction time is equal to the time between two checkpoints.
>
> II 1. - I’m confused. To make things straight. Flink is currently
> snapshotting once it receives all of the checkpoint barriers from all of
> the input channels and only then it broadcasts the checkpoint barrier down
> the stream. And this is correct from exactly-once perspective.
>
> As far as I understand, your proposal based on Chandy

Re: [DISCUSS] FLIP-52: Remove legacy Program interface.

[Stephan Ewen]

+1

the "main" method is the overwhelming default. getting rid of "two ways to
do things" is a good idea.

On Wed, Aug 14, 2019 at 1:42 PM Kostas Kloudas  wrote:

> Hi all,
>
> As discussed in [1] , the Program interface seems to be outdated and
> there seems to be
> no objection to remove it.
>
> Given that this interface is PublicEvolving, its removal should pass
> through a FLIP and
> this discussion and the associated FLIP are exactly for that purpose.
>
> Please let me know what you think and if it is ok to proceed with its
> removal.
>
> Cheers,
> Kostas
>
> link to FLIP-52 :
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=125308637
>
> [1]
> https://lists.apache.org/x/thread.html/7ffc9936a384b891dbcf0a481d26c6d13b2125607c200577780d1e18@%3Cdev.flink.apache.org%3E
>

Re: Checkpointing under backpressure

[Stephan Ewen]

Scaling with unaligned checkpoints might be a necessity.

Let's assume the job failed due to a lost TaskManager, but no new
TaskManager becomes available.
In that case we need to scale down based on the latest complete checkpoint,
because we cannot produce a new checkpoint.


On Wed, Aug 14, 2019 at 2:05 PM Paris Carbone 
wrote:

> +1 I think we are on the same page Stephan.
>
> Rescaling on unaligned checkpoint sounds challenging and a bit
> unnecessary. No?
> Why not sticking to aligned snapshots for live reconfiguration/rescaling?
> It’s a pretty rare operation and it would simplify things by a lot.
> Everything can be “staged” upon alignment including replacing channels and
> tasks.
>
> -Paris
>
> > On 14 Aug 2019, at 13:39, Stephan Ewen  wrote:
> >
> > Hi all!
> >
> > Yes, the first proposal of "unaligend checkpoints" (probably two years
> back
> > now) drew a major inspiration from Chandy Lamport, as did actually the
> > original checkpointing algorithm.
> >
> > "Logging data between first and last barrier" versus "barrier jumping
> over
> > buffer and storing those buffers" is pretty close same.
> > However, there are a few nice benefits of the proposal of unaligned
> > checkpoints over Chandy-Lamport.
> >
> > *## Benefits of Unaligned Checkpoints*
> >
> > (1) It is very similar to the original algorithm (can be seen an an
> > optional feature purely in the network stack) and thus can share lot's of
> > code paths.
> >
> > (2) Less data stored. If we make the "jump over buffers" part timeout
> based
> > (for example barrier overtakes buffers if not flushed within 10ms) then
> > checkpoints are in the common case of flowing pipelines aligned without
> > in-flight data. Only back pressured cases store some in-flight data,
> which
> > means we don't regress in the common case and only fix the back pressure
> > case.
> >
> > (3) Faster checkpoints. Chandy Lamport still waits for all barriers to
> > arrive naturally, logging on the way. If data processing is slow, this
> can
> > still take quite a while.
> >
> > ==> I think both these points are strong reasons to not change the
> > mechanism away from "trigger sources" and start with CL-style "trigger
> all".
> >
> >
> > *## Possible ways to combine Chandy Lamport and Unaligned Checkpoints*
> >
> > We can think about something like "take state snapshot on first barrier"
> > and then store buffers until the other barriers arrive. Inside the
> network
> > stack, barriers could still overtake and persist buffers.
> > The benefit would be less latency increase in the channels which already
> > have received barriers.
> > However, as mentioned before, not prioritizing the inputs from which
> > barriers are still missing can also have an adverse effect.
> >
> >
> > *## Concerning upgrades*
> >
> > I think it is a fair restriction to say that upgrades need to happen on
> > aligned checkpoints. It is a rare enough operation.
> >
> >
> > *## Concerning re-scaling (changing parallelism)*
> >
> > We need to support that on unaligned checkpoints as well. There are
> several
> > feature proposals about automatic scaling, especially down scaling in
> case
> > of missing resources. The last snapshot might be a regular checkpoint, so
> > all checkpoints need to support rescaling.
> >
> >
> > *## Concerning end-to-end checkpoint duration and "trigger sources"
> versus
> > "trigger all"*
> >
> > I think for the end-to-end checkpoint duration, an "overtake buffers"
> > approach yields faster checkpoints, as mentioned above (Chandy Lamport
> > logging still needs to wait for barrier to flow).
> >
> > I don't see the benefit of a "trigger all tasks via RPC concurrently"
> > approach. Bear in mind that it is still a globally coordinated approach
> and
> > you need to wait for the global checkpoint to complete before committing
> > any side effects.
> > I believe that the checkpoint time is more determined by the state
> > checkpoint writing, and the global coordination and metadata commit, than
> > by the difference in alignment time between "trigger from source and jump
> > over buffers" versus "trigger all tasks concurrently".
> >
> > Trying to optimize a few tens of milliseconds out of the network stack
> > sends (and changing the overall checkpointing approach completely for
> that)
> > whil

Re: Checkpointing under backpressure

[Stephan Ewen]

Quick note: The current implementation is

Align -> Forward -> Sync Snapshot Part (-> Async Snapshot Part)

On Wed, Aug 14, 2019 at 5:21 PM Piotr Nowojski  wrote:

> > Thanks for the great ideas so far.
>
> +1
>
> Regarding other things raised, I mostly agree with Stephan.
>
> I like the idea of simultaneously starting the checkpoint everywhere via
> RPC call (especially in cases where Tasks are busy doing some synchronous
> operations for example for tens of milliseconds. In that case every network
> exchange adds tens of milliseconds of delay in propagating the checkpoint).
> However I agree that this might be a premature optimisation assuming the
> current state of our code (we already have checkpoint barriers).
>
> However I like the idea of switching from:
>
> 1. A -> S -> F (Align -> snapshot -> forward markers)
>
> To
>
> 2. S -> F -> L (Snapshot -> forward markers -> log pending channels)
>
> Or even to
>
> 6. F -> S -> L (Forward markers -> snapshot -> log pending channels)
>
> It feels to me like this would decouple propagation of checkpoints from
> costs of synchronous snapshots and waiting for all of the checkpoint
> barriers to arrive (even if they will overtake in-flight records, this
> might take some time).
>
> > What I like about the Chandy Lamport approach (2.) initiated from
> sources is that:
> >   - Snapshotting imposes no modification to normal processing.
>
> Yes, I agree that would be nice. Currently, during the alignment and
> blocking of the input channels, we might be wasting CPU cycles of up stream
> tasks. If we succeed in designing new checkpointing mechanism to not
> disrupt/block regular data processing (% the extra IO cost for logging the
> in-flight records), that would be a huge improvement.
>
> Piotrek
>
> > On 14 Aug 2019, at 14:56, Paris Carbone  wrote:
> >
> > Sure I see. In cases when no periodic aligned snapshots are employed
> this is the only option.
> >
> > Two things that were not highlighted enough so far on the proposed
> protocol (included my mails):
> >   - The Recovery/Reconfiguration strategy should strictly prioritise
> processing logged events before entering normal task input operation.
> Otherwise causality can be violated. This also means dataflow recovery will
> be expected to be slower to the one employed on an aligned snapshot.
> >   - Same as with state capture, markers should be forwarded upon
> first marker received on input. No later than that. Otherwise we have
> duplicate side effects.
> >
> > Thanks for the great ideas so far.
> >
> > Paris
> >
> >> On 14 Aug 2019, at 14:33, Stephan Ewen  wrote:
> >>
> >> Scaling with unaligned checkpoints might be a necessity.
> >>
> >> Let's assume the job failed due to a lost TaskManager, but no new
> >> TaskManager becomes available.
> >> In that case we need to scale down based on the latest complete
> checkpoint,
> >> because we cannot produce a new checkpoint.
> >>
> >>
> >> On Wed, Aug 14, 2019 at 2:05 PM Paris Carbone 
> >> wrote:
> >>
> >>> +1 I think we are on the same page Stephan.
> >>>
> >>> Rescaling on unaligned checkpoint sounds challenging and a bit
> >>> unnecessary. No?
> >>> Why not sticking to aligned snapshots for live
> reconfiguration/rescaling?
> >>> It’s a pretty rare operation and it would simplify things by a lot.
> >>> Everything can be “staged” upon alignment including replacing channels
> and
> >>> tasks.
> >>>
> >>> -Paris
> >>>
> >>>> On 14 Aug 2019, at 13:39, Stephan Ewen  wrote:
> >>>>
> >>>> Hi all!
> >>>>
> >>>> Yes, the first proposal of "unaligend checkpoints" (probably two years
> >>> back
> >>>> now) drew a major inspiration from Chandy Lamport, as did actually the
> >>>> original checkpointing algorithm.
> >>>>
> >>>> "Logging data between first and last barrier" versus "barrier jumping
> >>> over
> >>>> buffer and storing those buffers" is pretty close same.
> >>>> However, there are a few nice benefits of the proposal of unaligned
> >>>> checkpoints over Chandy-Lamport.
> >>>>
> >>>> *## Benefits of Unaligned Checkpoints*
> >>>>
> >>>> (1) It is very similar to the original algorithm (can be seen an an
> >>>> optional feat

Re: Checkpointing under backpressure

[Stephan Ewen]

; > Subject:Re: Checkpointing under backpressure
> >
> > Quick note: The current implementation is
> >
> > Align -> Forward -> Sync Snapshot Part (-> Async Snapshot Part)
> >
> > On Wed, Aug 14, 2019 at 5:21 PM Piotr Nowojski 
> > wrote:
> >
> > > > Thanks for the great ideas so far.
> > >
> > > +1
> > >
> > > Regarding other things raised, I mostly agree with Stephan.
> > >
> > > I like the idea of simultaneously starting the checkpoint everywhere
> via
> > > RPC call (especially in cases where Tasks are busy doing some
> synchronous
> > > operations for example for tens of milliseconds. In that case every
> > network
> > > exchange adds tens of milliseconds of delay in propagating the
> > checkpoint).
> > > However I agree that this might be a premature optimisation assuming
> the
> > > current state of our code (we already have checkpoint barriers).
> > >
> > > However I like the idea of switching from:
> > >
> > > 1. A -> S -> F (Align -> snapshot -> forward markers)
> > >
> > > To
> > >
> > > 2. S -> F -> L (Snapshot -> forward markers -> log pending channels)
> > >
> > > Or even to
> > >
> > > 6. F -> S -> L (Forward markers -> snapshot -> log pending channels)
> > >
> > > It feels to me like this would decouple propagation of checkpoints from
> > > costs of synchronous snapshots and waiting for all of the checkpoint
> > > barriers to arrive (even if they will overtake in-flight records, this
> > > might take some time).
> > >
> > > > What I like about the Chandy Lamport approach (2.) initiated from
> > > sources is that:
> > > >   - Snapshotting imposes no modification to normal processing.
> > >
> > > Yes, I agree that would be nice. Currently, during the alignment and
> > > blocking of the input channels, we might be wasting CPU cycles of up
> > stream
> > > tasks. If we succeed in designing new checkpointing mechanism to not
> > > disrupt/block regular data processing (% the extra IO cost for logging
> > the
> > > in-flight records), that would be a huge improvement.
> > >
> > > Piotrek
> > >
> > > > On 14 Aug 2019, at 14:56, Paris Carbone 
> > wrote:
> > > >
> > > > Sure I see. In cases when no periodic aligned snapshots are employed
> > > this is the only option.
> > > >
> > > > Two things that were not highlighted enough so far on the proposed
> > > protocol (included my mails):
> > > >   - The Recovery/Reconfiguration strategy should strictly
> > prioritise
> > > processing logged events before entering normal task input operation.
> > > Otherwise causality can be violated. This also means dataflow recovery
> > will
> > > be expected to be slower to the one employed on an aligned snapshot.
> > > >   - Same as with state capture, markers should be forwarded upon
> > > first marker received on input. No later than that. Otherwise we have
> > > duplicate side effects.
> > > >
> > > > Thanks for the great ideas so far.
> > > >
> > > > Paris
> > > >
> > > >> On 14 Aug 2019, at 14:33, Stephan Ewen  wrote:
> > > >>
> > > >> Scaling with unaligned checkpoints might be a necessity.
> > > >>
> > > >> Let's assume the job failed due to a lost TaskManager, but no new
> > > >> TaskManager becomes available.
> > > >> In that case we need to scale down based on the latest complete
> > > checkpoint,
> > > >> because we cannot produce a new checkpoint.
> > > >>
> > > >>
> > > >> On Wed, Aug 14, 2019 at 2:05 PM Paris Carbone <
> > seniorcarb...@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> +1 I think we are on the same page Stephan.
> > > >>>
> > > >>> Rescaling on unaligned checkpoint sounds challenging and a bit
> > > >>> unnecessary. No?
> > > >>> Why not sticking to aligned snapshots for live
> > > reconfiguration/rescaling?
> > > >>> It’s a pretty rare operation and it would simplify things by a lot.
> > > >>> Everything can be “staged” upon alignment including replacing
> > channels
> > > and
> > 

Re: [DISCUSS] FLIP-50: Spill-able Heap Keyed State Backend

[Stephan Ewen]

+1 for this feature. I think this will be appreciated by users, as a way to
use the HeapStateBackend with a safety-net against OOM errors.
And having had major production exposure is great.

>From the implementation plan, it looks like this exists purely in a new
module and does not require any changes in other parts of Flink's code. Can
you confirm that?

Other that that, I have no further questions and we could proceed to vote
on this FLIP, from my side.

Best,
Stephan


On Tue, Aug 13, 2019 at 10:00 PM Yu Li  wrote:

> Sorry for forgetting to give the link of the FLIP, here it is:
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend
>
> Thanks!
>
> Best Regards,
> Yu
>
>
> On Tue, 13 Aug 2019 at 18:06, Yu Li  wrote:
>
> > Hi All,
> >
> > We ever held a discussion about this feature before [1] but now opening
> > another thread because after a second thought introducing a new backend
> > instead of modifying the existing heap backend is a better option to
> > prevent causing any regression or surprise to existing in-production
> usage.
> > And since introducing a new backend is relatively big change, we regard
> it
> > as a FLIP and need another discussion and voting process according to our
> > newly drafted bylaw [2].
> >
> > Please allow me to quote the brief description from the old thread [1]
> for
> > the convenience of those who noticed this feature for the first time:
> >
> >
> > *HeapKeyedStateBackend is one of the two KeyedStateBackends in Flink,
> > since state lives as Java objects on the heap in HeapKeyedStateBackend
> and
> > the de/serialization only happens during state snapshot and restore, it
> > outperforms RocksDBKeyeStateBackend when all data could reside in
> memory.**However,
> > along with the advantage, HeapKeyedStateBackend also has its
> shortcomings,
> > and the most painful one is the difficulty to estimate the maximum heap
> > size (Xmx) to set, and we will suffer from GC impact once the heap memory
> > is not enough to hold all state data. There’re several (inevitable)
> causes
> > for such scenario, including (but not limited to):*
> >
> >
> >
> > ** Memory overhead of Java object representation (tens of times of the
> > serialized data size).* Data flood caused by burst traffic.* Data
> > accumulation caused by source malfunction.**To resolve this problem, we
> > proposed a solution to support spilling state data to disk before heap
> > memory is exhausted. We will monitor the heap usage and choose the
> coldest
> > data to spill, and reload them when heap memory is regained after data
> > removing or TTL expiration, automatically. Furthermore, *to prevent
> > causing unexpected regression to existing usage of HeapKeyedStateBackend,
> > we plan to introduce a new SpillableHeapKeyedStateBackend and change it
> to
> > default in future if proven to be stable.
> >
> > Please let us know your point of the feature and any comment is
> > welcomed/appreciated. Thanks.
> >
> > [1] https://s.apache.org/pxeif
> > [2]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120731026
> >
> > Best Regards,
> > Yu
> >
>

Re: [DISCUSS] FLIP-48: Pluggable Intermediate Result Storage

[Stephan Ewen]

Sorry for the late response. So many FLIPs these days.

I am a bit unsure about the motivation here, and that this need to be a
part of Flink. It sounds like this can be perfectly built around Flink as a
minimal library on top of it, without any change in the core APIs or
runtime.

The proposal to handle "caching intermediate results" (to make them
reusable across jobs in a session), and "writing them in different formats
/ indexing them" doesn't sound like it should be the same mechanism.

  - The caching part is a transparent low-level primitive. It avoid
re-executing a part of the job graph, but otherwise is completely
transparent to the consumer job.

  - Writing data out in a sink, compressing/indexing it and then reading it
in another job is also a way of reusing a previous result, but on a
completely different abstraction level. It is not the same intermediate
result any more. When the consumer reads from it and applies predicate
pushdown, etc. then the consumer job looks completely different from a job
that consumed the original result. It hence needs to be solved on the API
level via a sink and a source.

I would suggest to keep these concepts separate: Caching (possibly
automatically) for jobs in a session, and long term writing/sharing of data
sets.

Solving the "long term writing/sharing" in a library rather than in the
runtime also has the advantage of not pushing yet more stuff into Flink's
core, which I believe is also an important criterion.

Best,
Stephan


On Thu, Jul 25, 2019 at 4:53 AM Xuannan Su  wrote:

> Hi folks,
>
> I would like to start the FLIP discussion thread about the pluggable
> intermediate result storage.
>
> This is phase 2 of FLIP-36: Support Interactive Programming in Flink Skip
> to end of metadata. While the FLIP-36 provides a default implementation of
> the intermediate result storage using the shuffle service, we would like to
> make the intermediate result storage pluggable so that the user can easily
> swap the storage.
>
> We are looking forward to your thought!
>
> The FLIP link is the following:
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-48%3A+Pluggable+Intermediate+Result+Storage
> <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-48:+Pluggable+Intermediate+Result+Storage
> >
>
> Best,
> Xuannan
>

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

[Stephan Ewen]

n fix it into 1.9.1.
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> Thanks,
> > > > > > > > > > >>>>>>>>>>> Jark
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> On Tue, 13 Aug 2019 at 15:10, Richard
> > > Deurwaarder <
> > > > > > > > > > >>> rich...@xeli.eu
> > > > > > > > > > >>>>>>>>>> wrote:
> > > > > > > > > > >>>>>>>>>>>> Hello all,
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> I noticed the PubSub example jar is not
> > included
> > > > in
> > > > > > the
> > > > > > > > > > >>> examples/
> > > > > > > > > > >>>>>> dir
> > > > > > > > > > >>>>>>>>>> of
> > > > > > > > > > >>>>>>>>>>>> flink-dist. I've created
> > > > > > > > > > >>>>>>>>>>>
> > > https://issues.apache.org/jira/browse/FLINK-13700
> > > > > > > > > > >>>>>>>>>>>> +
> > > https://github.com/apache/flink/pull/9424/files
> > > > > to
> > > > > > > fix
> > > > > > > > > > this.
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> I will leave it up to you to decide if we
> want
> > > to
> > > > > add
> > > > > > > this
> > > > > > > > > to
> > > > > > > > > > >>>>>> 1.9.0.
> > > > > > > > > > >>>>>>>>>>>> Regards,
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> Richard
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> On Tue, Aug 13, 2019 at 9:04 AM Till
> Rohrmann
> > <
> > > > > > > > > > >>>>>> trohrm...@apache.org>
> > > > > > > > > > >>>>>>>>>>>> wrote:
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> Hi Jark,
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> thanks for reporting this issue. Could this
> > be
> > > a
> > > > > > > > documented
> > > > > > > > > > >>>>>>>>>> limitation
> > > > > > > > > > >>>>>>>>>>> of
> > > > > > > > > > >>>>>>>>>>>>> Blink's preview version? I think we have
> > agreed
> > > > > that
> > > > > > > the
> > > > > > > > > > Blink
> > > > > > > > > > >>>> SQL
> > > > > > > > > > >>>>>>>>>>>> planner
> > > > > > > > > > >>>>>>>>>>>>> will be rather a preview feature than
> > > production
> > > > > > ready.
> > > > > > > > > Hence
> > > > > > > > > > >>> it
> > > > > > > > > > >>>>&g

Re: [VOTE] FLIP-50: Spill-able Heap State Backend

[Stephan Ewen]

+1

On Sun, Aug 18, 2019 at 3:31 PM Till Rohrmann  wrote:

> +1
>
> On Fri, Aug 16, 2019 at 4:54 PM Yu Li  wrote:
>
> > Hi All,
> >
> > Since we have reached a consensus in the discussion thread [1], I'd like
> to
> > start the voting for FLIP-50 [2].
> >
> > This vote will be open for at least 72 hours. Unless objection I will try
> > to close it by end of Tuesday August 20, 2019 if we have sufficient
> votes.
> > Thanks.
> >
> > [1] https://s.apache.org/cq358
> > [2]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend
> >
> > Best Regards,
> > Yu
> >
>

Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

[Stephan Ewen]

I like the idea of enhancing the configuration and to do early validation.

I feel that some of the ideas in the FLIP seem a bit ad hoc, though. For
example, having a boolean "isList" is a clear indication of not having
thought through the type/category system.
Also, having a more clear category system makes validation simpler.

For example, I have seen systems distinguishing between numeric parameters
(valid ranges), category parameters (set of possible values), quantities
like duration and memory size (need measure and unit), which results in an
elegant system for validation.


On Fri, Aug 16, 2019 at 5:22 PM JingsongLee 
wrote:

> +1 to this, thanks Timo and Dawid for the design.
> This allows the currently cluttered configuration of various
>  modules to be unified.
> This is also first step of one of the keys to making new unified
> TableEnvironment available for production.
>
> Previously, we did encounter complex configurations, such as
> specifying the skewed values of column in DDL. The skew may
>  be a single field or a combination of multiple fields. So the
>  configuration is very troublesome. We used JSON string to
>  configure it.
>
> Best,
> Jingsong Lee
>
>
>
> --
> From:Jark Wu 
> Send Time:2019年8月16日(星期五) 16:44
> To:dev 
> Subject:Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration
>
> Thanks for starting this design Timo and Dawid,
>
> Improving ConfigOption has been hovering in my mind for a long time.
> We have seen the benefit when developing blink configurations and connector
> properties in 1.9 release.
> Thanks for bringing it up and make such a detailed design.
> I will leave my thoughts and comments there.
>
> Cheers,
> Jark
>
>
> On Fri, 16 Aug 2019 at 22:30, Zili Chen  wrote:
>
> > Hi Timo,
> >
> > It looks interesting. Thanks for preparing this FLIP!
> >
> > Client API enhancement benefit from this evolution which
> > hopefully provides a better view of configuration of Flink.
> > In client API enhancement, we likely make the deployment
> > of cluster and submission of job totally defined by configuration.
> >
> > Will take a look at the document in days.
> >
> > Best,
> > tison.
> >
> >
> > Timo Walther  于2019年8月16日周五 下午10:12写道：
> >
> > > Hi everyone,
> > >
> > > Dawid and I are working on making parts of ExecutionConfig and
> > > TableConfig configurable via config options. This is necessary to make
> > > all properties also available in SQL. Additionally, with the new SQL
> DDL
> > > based on properties as well as more connectors and formats coming up,
> > > unified configuration becomes more important.
> > >
> > > We need more features around string-based configuration in the future,
> > > which is why Dawid and I would like to propose FLIP-54 for evolving the
> > > ConfigOption and Configuration classes:
> > >
> > >
> > >
> >
> https://docs.google.com/document/d/1IQ7nwXqmhCy900t2vQLEL3N2HIdMg-JO8vTzo1BtyKU/edit
> > >
> > > In summary it adds:
> > > - documented types and validation
> > > - more common types such as memory size, duration, list
> > > - simple non-nested object types
> > >
> > > Looking forward to your feedback,
> > > Timo
> > >
> > >
> >
>
>

Re: [DISCUSS] Update our Roadmap

[Stephan Ewen]

I could help with that.

On Fri, Aug 16, 2019 at 2:36 PM Robert Metzger  wrote:

> Flink 1.9 is feature freezed and almost released.
> I guess it makes sense to update the roadmap on the website again.
>
> Who feels like having a good overview of what's coming up?
>
> On Tue, May 7, 2019 at 4:33 PM Fabian Hueske  wrote:
>
> > Yes, that's a very good proposal Jark.
> > +1
> >
> > Best, Fabian
> >
> > Am Mo., 6. Mai 2019 um 16:33 Uhr schrieb Till Rohrmann <
> > trohrm...@apache.org
> > >:
> >
> > > I think this is a good idea Jark. Putting the last update date on the
> > > roadmap would also force us to regularly update it.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Mon, May 6, 2019 at 4:14 AM Jark Wu  wrote:
> > >
> > > > Hi,
> > > >
> > > > One suggestion for the roadmap:
> > > >
> > > > Shall we add a `latest-update-time` to the top of Roadmap page? So
> that
> > > > users can know this is a up-to-date Roadmap.
> > > >
> > > > On Thu, 2 May 2019 at 04:49, Bowen Li  wrote:
> > > >
> > > > > +1
> > > > >
> > > > > On Mon, Apr 29, 2019 at 11:41 PM jincheng sun <
> > > sunjincheng...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Jeff&Fabian,
> > > > > >
> > > > > > I have open the PR about add Python Table API section to the
> > > roadmap. I
> > > > > > appreciate if you have time to look at it. :)
> > > > > >
> > > > > > https://github.com/apache/flink-web/pull/204
> > > > > >
> > > > > > Regards,
> > > > > > Jincheng
> > > > > >
> > > > > > jincheng sun  于2019年4月29日周一 下午11:12写道：
> > > > > >
> > > > > > > Sure, I will do it！I think the python table api info should in
> > the
> > > > > > >  roadmap! Thank you @Jeff @Fabian
> > > > > > >
> > > > > > > Fabian Hueske 于2019年4月29日 周一23:05写道：
> > > > > > >
> > > > > > >> Great, thanks Jeff and Timo!
> > > > > > >>
> > > > > > >> @Jincheng do you want to write a paragraph about the Python
> > effort
> > > > and
> > > > > > >> open a PR for it?
> > > > > > >>
> > > > > > >> I'll remove the issue about Hadoop convenience builds
> > > (FLINK-11266).
> > > > > > >>
> > > > > > >> Best, Fabian
> > > > > > >>
> > > > > > >> Am Mo., 29. Apr. 2019 um 16:37 Uhr schrieb Jeff Zhang <
> > > > > zjf...@gmail.com
> > > > > > >:
> > > > > > >>
> > > > > > >>> jincheng(cc) is driving the python effort, I think he can
> help
> > to
> > > > > > >>> prepare it.
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> Fabian Hueske  于2019年4月29日周一 下午10:15写道：
> > > > > > >>>
> > > > > >  Hi everyone,
> > > > > > 
> > > > > >  Since we had no more comments on this thread, I think we
> > proceed
> > > > to
> > > > > >  update the roadmap.
> > > > > > 
> > > > > >  @Jeff Zhang  I agree, we should add the
> > > Python
> > > > > >  efforts to the roadmap.
> > > > > >  Do you want to prepare a short paragraph that we can add to
> > the
> > > > > >  document?
> > > > > > 
> > > > > >  Best, Fabian
> > > > > > 
> > > > > >  Am Mi., 17. Apr. 2019 um 15:04 Uhr schrieb Jeff Zhang <
> > > > > > zjf...@gmail.com
> > > > > >  >:
> > > > > > 
> > > > > > > Hi Fabian,
> > > > > > >
> > > > > > > One thing missing is python api and python udf, we already
> > > > > discussed
> > > > > > > it in
> > > > > > > community, and it is very close to reach consensus.
> > > > > > >
> > > > > > >
> > > > > > > Fabian Hueske  于2019年4月17日周三 下午7:51写道：
> > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > We recently added a roadmap to our project website [1]
> and
> > > > > decided
> > > > > > to
> > > > > > > > update it after every release. Flink 1.8.0 was released a
> > few
> > > > > days
> > > > > > > ago, so
> > > > > > > > I think it we should check and remove from the roadmap
> what
> > > was
> > > > > > > achieved so
> > > > > > > > far and add features / improvements that we plan for the
> > > > future.
> > > > > > > >
> > > > > > > > I had a look at the roadmap and found that
> > > > > > > >
> > > > > > > > > We are changing the build setup to not bundle Hadoop by
> > > > > default,
> > > > > > > but
> > > > > > > > rather offer pre-packaged
> > > > > > > > > Hadoop libraries for the use with Yarn, HDFS, etc. as
> > > > > convenience
> > > > > > > > downloads FLINK-11266 <
> > > > > > > https://issues.apache.org/jira/browse/FLINK-11266>.
> > > > > > > >
> > > > > > > > was implemented for 1.8.0 and should be removed from the
> > > > roadmap.
> > > > > > > > All other issues are still ongoing efforts.
> > > > > > > >
> > > > > > > > Are there any other efforts that we want to put on the
> > > roadmap?
> > > > > > > >
> > > > > > > > Best, Fabian
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best Regards
> > > > > > >
> > > > > > > Jeff Zhang
> > > > > > >
> > > > > > 
> > > > > > >>>
> > > > > > >>> --
> > > > > > >>> B

Re: [DISCUSS] Release flink-shaded 8.0

[Stephan Ewen]

Are we fine with the current Netty version, or would be want to bump it?

On Fri, Aug 16, 2019 at 10:30 AM Chesnay Schepler 
wrote:

> Hello,
>
> I would like to kick off the next flink-shaded release next week. There
> are 2 ongoing efforts that are blocked on this release:
>
>   * [FLINK-13467] Java 11 support requires a bump to ASM to correctly
> handle Java 11 bytecode
>   * [FLINK-11767] Reworking the typeSerializerSnapshotMigrationTestBase
> requires asm-commons to be added to flink-shaded-asm
>
> Are there any other changes on anyone's radar that we will have to make
> for 1.10? (will bumping calcite require anything, for example)
>
>
>

Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

[Stephan Ewen]

A "List Type" sounds like a good direction to me.

The comment on the type system was a bit brief, I agree. The idea is to see
if something like that can ease validation. Especially the correlation
system seems quite complex (proxies to work around order of initialization).

For example, let's assume we don't think primarily about "java types" but
would define types as one of the following (just examples, haven't thought
all the details through):

  (a) category type: implies string, and a fix set of possible values.
Those would be passes and naturally make it into the docs and validation.
Maps to a String or Enum in Java.

  (b) numeric integer type: implies long (or optionally integer, if we want
to automatically check overflow / underflow). would take typical domain
validators, like non-negative, etc.

  (c) numeric real type: same as above (double or float)

  (d) numeric interval type: either defined as an interval, or references
other parameter by key. validation by valid interval.

  (e) quantity: a measure and a unit. separately parsable. The measure's
type could be any of the numeric types above, with same validation rules.

With a system like the above, would we still correlation validators? Are
there still cases that we need to catch early (config loading) or are the
remaining cases sufficiently rare and runtime or setup specific, that it is
fine to handle them in component initialization?


On Sun, Aug 18, 2019 at 6:36 PM Dawid Wysakowicz 
wrote:

> Hi Stephan,
>
> Thank you for your opinion.
>
> Actually list/composite types are the topics we spent the most of the
> time. I understand that from a perspective of a full blown type system,
> a field like isList may look weird. Please let me elaborate a bit more
> on the reason behind it though. Maybe we weren't clear enough about it
> in the FLIP. The key feature of all the conifg options is that they must
> have a string representation as they might come from a configuration
> file. Moreover it must be a human readable format, so that the values
> might be manually adjusted. Having that in mind we did not want to add a
> support of an arbitrary nesting and we decided to allow for lists only
> (and flat objects - I think though in the current design there is a
> mistake around the Configurable interface). I think though you have a
> point here and it would be better to have a ListConfigOption instead of
> this field. Does it make sense to you?
>
> As for the second part of your message. I am not sure if I understood
> it. The validators work with parse/deserialized values from
> Configuration that means they can be bound to the generic parameter of
> Configuration. You can have a RangeValidator Comparable/Number>. I don't think the type hierarchy in the ConfigOption
> has anything to do with the validation logic. Could you elaborate a bit
> more what did you mean?
>
> Best,
>
> Dawid
>
> On 18/08/2019 16:42, Stephan Ewen wrote:
> > I like the idea of enhancing the configuration and to do early
> validation.
> >
> > I feel that some of the ideas in the FLIP seem a bit ad hoc, though. For
> > example, having a boolean "isList" is a clear indication of not having
> > thought through the type/category system.
> > Also, having a more clear category system makes validation simpler.
> >
> > For example, I have seen systems distinguishing between numeric
> parameters
> > (valid ranges), category parameters (set of possible values), quantities
> > like duration and memory size (need measure and unit), which results in
> an
> > elegant system for validation.
> >
> >
> > On Fri, Aug 16, 2019 at 5:22 PM JingsongLee  .invalid>
> > wrote:
> >
> >> +1 to this, thanks Timo and Dawid for the design.
> >> This allows the currently cluttered configuration of various
> >>  modules to be unified.
> >> This is also first step of one of the keys to making new unified
> >> TableEnvironment available for production.
> >>
> >> Previously, we did encounter complex configurations, such as
> >> specifying the skewed values of column in DDL. The skew may
> >>  be a single field or a combination of multiple fields. So the
> >>  configuration is very troublesome. We used JSON string to
> >>  configure it.
> >>
> >> Best,
> >> Jingsong Lee
> >>
> >>
> >>
> >> --
> >> From:Jark Wu 
> >> Send Time:2019年8月16日(星期五) 16:44
> >> To:dev 
> >> Subject:Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration
> >>
> >> Thanks for starting this design Ti

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

[Stephan Ewen]

+1 for Gordon's approach.

If we do that, we can probably skip re-testing everything and mainly need
to verify the release artifacts (signatures, build from source, etc.).

If we open the RC up for changes, I fear a lot of small issues will rush in
and destabilize the candidate again, meaning we have to do another larger
testing effort.



On Mon, Aug 19, 2019 at 9:48 AM Becket Qin  wrote:

> Hi Gordon,
>
> I remember we mentioned earlier that if there is an additional RC, we can
> piggyback the GCP PubSub API change (
> https://issues.apache.org/jira/browse/FLINK-13231). It is a small patch to
> avoid future API change. So should be able to merge it very shortly. Would
> it be possible to include that into RC3 as well?
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Mon, Aug 19, 2019 at 9:43 AM Tzu-Li (Gordon) Tai 
> wrote:
>
> > Hi,
> >
> > https://issues.apache.org/jira/browse/FLINK-13752 turns out to be an
> > actual
> > blocker, so we would have to close this RC now in favor of a new one.
> >
> > Since we are already quite past the planned release time for 1.9.0, I
> would
> > like to limit the new changes included in RC3 to only the following:
> > - https://issues.apache.org/jira/browse/FLINK-13752
> > - Fix license and notice file issues that Kurt had found with
> > flink-runtime-web and flink-state-processing-api
> >
> > This means that I will not be creating RC3 with the release-1.9 branch as
> > is, but essentially only cherry-picking the above mentioned changes on
> top
> > of RC2.
> > The minimal set of changes on top of RC2 should allow us to carry most if
> > not all of the already existing votes without another round of extensive
> > testing, and allow us to have a shortened voting time.
> >
> > I understand that there are other issues mentioned in this thread that
> are
> > already spotted and merged to release-1.9, especially for the Blink
> planner
> > and DDL, but I suggest not to include them in RC3.
> > I think it would be better to collect all the remaining issues for those
> > over a period of time, and include them as 1.9.1 which can ideally also
> > happen a few weeks soon after 1.9.0.
> >
> > What do you think? If there are not objections, I would proceed with this
> > plan and push out a new RC by the end of today (Aug. 19th CET).
> >
> > Regards,
> > Gordon
> >
> > On Mon, Aug 19, 2019 at 4:09 AM Zili Chen  wrote:
> >
> > > We should investigate the performance regression but regardless the
> > > regression I vote +1
> > >
> > > Have verified following things
> > >
> > > - Jobs running on YARN x (Session & Per Job) with high-availability
> > > enabled.
> > > - Simulate JM and TM failures.
> > > - Simulate temporary network partition.
> > >
> > > Best,
> > > tison.
> > >
> > >
> > > Stephan Ewen  于2019年8月18日周日 下午10:12写道：
> > >
> > > > For reference, this is the JIRA issue about the regression in
> question:
> > > >
> > > > https://issues.apache.org/jira/browse/FLINK-13752
> > > >
> > > >
> > > > On Fri, Aug 16, 2019 at 10:57 AM Guowei Ma 
> > wrote:
> > > >
> > > > > Hi, till
> > > > > I can send the job to you offline.
> > > > > It is just a datastream job and does not use
> > > > TwoInputSelectableStreamTask.
> > > > > A->B
> > > > >  \
> > > > >C
> > > > >  /
> > > > > D->E
> > > > > Best,
> > > > > Guowei
> > > > >
> > > > >
> > > > > Till Rohrmann  于2019年8月16日周五 下午4:34写道：
> > > > >
> > > > > > Thanks for reporting this issue Guowei. Could you share a bit
> more
> > > > > details
> > > > > > what the job exactly does and which operators it uses? Does the
> job
> > > > uses
> > > > > > the new `TwoInputSelectableStreamTask` which might cause the
> > > > performance
> > > > > > regression?
> > > > > >
> > > > > > I think it is important to understand where the problem comes
> from
> > > > before
> > > > > > we proceed with the release.
> > > > > >
> > > > > > Cheers,
> > > > > > Till
> > > > > >
> > > > > > On Fri, Aug 16, 2019 at 10:27 AM Guowei Ma 

Re: [DISCUSS] FLIP-49: Unified Memory Configuration for TaskExecutors

[Stephan Ewen]

About the "-XX:MaxDirectMemorySize" discussion, maybe let me summarize it a
bit differently:

We have the following two options:

(1) We let MemorySegments be de-allocated by the GC. That makes it segfault
safe. But then we need a way to trigger GC in case de-allocation and
re-allocation of a bunch of segments happens quickly, which is often the
case during batch scheduling or task restart.
  - The "-XX:MaxDirectMemorySize" (option 1.1) is one way to do this
  - Another way could be to have a dedicated bookkeeping in the
MemoryManager (option 1.2), so that this is a number independent of the
"-XX:MaxDirectMemorySize" parameter.

(2) We manually allocate and de-allocate the memory for the MemorySegments
(option 2). That way we need not worry about triggering GC by some
threshold or bookkeeping, but it is harder to prevent segfaults. We need to
be very careful about when we release the memory segments (only in the
cleanup phase of the main thread).

If we go with option 1.1, we probably need to set
"-XX:MaxDirectMemorySize" to "off_heap_managed_memory + direct_memory" and
have "direct_memory" as a separate reserved memory pool. Because if we just
set "-XX:MaxDirectMemorySize" to "off_heap_managed_memory + jvm_overhead",
then there will be times when that entire memory is allocated by direct
buffers and we have nothing left for the JVM overhead. So we either need a
way to compensate for that (again some safety margin cutoff value) or we
will exceed container memory.

If we go with option 1.2, we need to be aware that it takes elaborate logic
to push recycling of direct buffers without always triggering a full GC.


My first guess is that the options will be easiest to do in the following
order:

  - Option 1.1 with a dedicated direct_memory parameter, as discussed
above. We would need to find a way to set the direct_memory parameter by
default. We could start with 64 MB and see how it goes in practice. One
danger I see is that setting this loo low can cause a bunch of additional
GCs compared to before (we need to watch this carefully).

  - Option 2. It is actually quite simple to implement, we could try how
segfault safe we are at the moment.

  - Option 1.2: We would not touch the "-XX:MaxDirectMemorySize" parameter
at all and assume that all the direct memory allocations that the JVM and
Netty do are infrequent enough to be cleaned up fast enough through regular
GC. I am not sure if that is a valid assumption, though.

Best,
Stephan



On Fri, Aug 16, 2019 at 2:16 PM Xintong Song  wrote:

> Thanks for sharing your opinion Till.
>
> I'm also in favor of alternative 2. I was wondering whether we can avoid
> using Unsafe.allocate() for off-heap managed memory and network memory with
> alternative 3. But after giving it a second thought, I think even for
> alternative 3 using direct memory for off-heap managed memory could cause
> problems.
>
> Hi Yang,
>
> Regarding your concern, I think what proposed in this FLIP it to have both
> off-heap managed memory and network memory allocated through
> Unsafe.allocate(), which means they are practically native memory and not
> limited by JVM max direct memory. The only parts of memory limited by JVM
> max direct memory are task off-heap memory and JVM overhead, which are
> exactly alternative 2 suggests to set the JVM max direct memory to.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Fri, Aug 16, 2019 at 1:48 PM Till Rohrmann 
> wrote:
>
> > Thanks for the clarification Xintong. I understand the two alternatives
> > now.
> >
> > I would be in favour of option 2 because it makes things explicit. If we
> > don't limit the direct memory, I fear that we might end up in a similar
> > situation as we are currently in: The user might see that her process
> gets
> > killed by the OS and does not know why this is the case. Consequently,
> she
> > tries to decrease the process memory size (similar to increasing the
> cutoff
> > ratio) in order to accommodate for the extra direct memory. Even worse,
> she
> > tries to decrease memory budgets which are not fully used and hence won't
> > change the overall memory consumption.
> >
> > Cheers,
> > Till
> >
> > On Fri, Aug 16, 2019 at 11:01 AM Xintong Song 
> > wrote:
> >
> > > Let me explain this with a concrete example Till.
> > >
> > > Let's say we have the following scenario.
> > >
> > > Total Process Memory: 1GB
> > > JVM Direct Memory (Task Off-Heap Memory + JVM Overhead): 200MB
> > > Other Memory (JVM Heap Memory, JVM Metaspace, Off-Heap Managed Memory
> and
> > > Network Memory): 800MB
> > >
> > >
> > > For alternative 2, we set -XX:MaxDirectMemorySize to 200MB.
> > > For alternative 3, we set -XX:MaxDirectMemorySize to a very large
> value,
> > > let's say 1TB.
> > >
> > > If the actual direct memory usage of Task Off-Heap Memory and JVM
> > Overhead
> > > do not exceed 200MB, then alternative 2 and alternative 3 should have
> the
> > > same utility. Setting larger -XX:MaxDirectMemorySize will not reduce
> the
> > > sizes of t

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

[Stephan Ewen]

Looking at FLINK-13699, it seems to be very local to Table API and HBase
connector.
We can cherry-pick that without re-running distributed tests.


On Mon, Aug 19, 2019 at 1:46 PM Till Rohrmann  wrote:

> I've merged the fix for FLINK-13752. Hence we are good to go to create the
> new RC.
>
> Cheers,
> Till
>
> On Mon, Aug 19, 2019 at 1:30 PM Timo Walther  wrote:
>
> > I support Jark's fix for FLINK-13699 because it would be disappointing
> > if both DDL and connectors are ready to handle DATE/TIME/TIMESTAMP but a
> > little component in the middle of the stack is preventing an otherwise
> > usable feature. The changes are minor.
> >
> > Thanks,
> > Timo
> >
> >
> > Am 19.08.19 um 13:24 schrieb Jark Wu:
> > > Hi Gordon,
> > >
> > > I agree that we should pick the minimal set of changes to shorten the
> > > release testing time.
> > > However, I would like to include FLINK-13699 into RC3. FLINK-13699 is a
> > > critical DDL issue, and is a small change to flink table (won't affect
> > the
> > > runtime feature and stability).
> > > I will do some tests around sql and blink planner if the RC3 include
> this
> > > fix.
> > >
> > > But if the community against to include it, I'm also fine with having
> it
> > in
> > > the next minor release.
> > >
> > > Thanks,
> > > Jark
> > >
> > > On Mon, 19 Aug 2019 at 16:16, Stephan Ewen  wrote:
> > >
> > >> +1 for Gordon's approach.
> > >>
> > >> If we do that, we can probably skip re-testing everything and mainly
> > need
> > >> to verify the release artifacts (signatures, build from source, etc.).
> > >>
> > >> If we open the RC up for changes, I fear a lot of small issues will
> > rush in
> > >> and destabilize the candidate again, meaning we have to do another
> > larger
> > >> testing effort.
> > >>
> > >>
> > >>
> > >> On Mon, Aug 19, 2019 at 9:48 AM Becket Qin 
> > wrote:
> > >>
> > >>> Hi Gordon,
> > >>>
> > >>> I remember we mentioned earlier that if there is an additional RC, we
> > can
> > >>> piggyback the GCP PubSub API change (
> > >>> https://issues.apache.org/jira/browse/FLINK-13231). It is a small
> > patch
> > >> to
> > >>> avoid future API change. So should be able to merge it very shortly.
> > >> Would
> > >>> it be possible to include that into RC3 as well?
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Jiangjie (Becket) Qin
> > >>>
> > >>> On Mon, Aug 19, 2019 at 9:43 AM Tzu-Li (Gordon) Tai <
> > tzuli...@apache.org
> > >>>
> > >>> wrote:
> > >>>
> > >>>> Hi,
> > >>>>
> > >>>> https://issues.apache.org/jira/browse/FLINK-13752 turns out to be
> an
> > >>>> actual
> > >>>> blocker, so we would have to close this RC now in favor of a new
> one.
> > >>>>
> > >>>> Since we are already quite past the planned release time for 1.9.0,
> I
> > >>> would
> > >>>> like to limit the new changes included in RC3 to only the following:
> > >>>> - https://issues.apache.org/jira/browse/FLINK-13752
> > >>>> - Fix license and notice file issues that Kurt had found with
> > >>>> flink-runtime-web and flink-state-processing-api
> > >>>>
> > >>>> This means that I will not be creating RC3 with the release-1.9
> branch
> > >> as
> > >>>> is, but essentially only cherry-picking the above mentioned changes
> on
> > >>> top
> > >>>> of RC2.
> > >>>> The minimal set of changes on top of RC2 should allow us to carry
> most
> > >> if
> > >>>> not all of the already existing votes without another round of
> > >> extensive
> > >>>> testing, and allow us to have a shortened voting time.
> > >>>>
> > >>>> I understand that there are other issues mentioned in this thread
> that
> > >>> are
> > >>>> already spotted and merged to release-1.9, especially for the Blink
> > >>> planner
> > >>>> and DDL, but I suggest not to include them in RC3.
> &

Re: [DISCUSS] FLIP-49: Unified Memory Configuration for TaskExecutors

[Stephan Ewen]

My main concern with option 2 (manually release memory) is that segfaults
in the JVM send off all sorts of alarms on user ends. So we need to
guarantee that this never happens.

The trickyness is in tasks that uses data structures / algorithms with
additional threads, like hash table spill/read and sorting threads. We need
to ensure that these cleanly shut down before we can exit the task.
I am not sure that we have that guaranteed already, that's why option 1.1
seemed simpler to me.

On Mon, Aug 19, 2019 at 3:42 PM Xintong Song  wrote:

> Thanks for the comments, Stephan. Summarized in this way really makes
> things easier to understand.
>
> I'm in favor of option 2, at least for the moment. I think it is not that
> difficult to keep it segfault safe for memory manager, as long as we always
> de-allocate the memory segment when it is released from the memory
> consumers. Only if the memory consumer continue using the buffer of memory
> segment after releasing it, in which case we do want the job to fail so we
> detect the memory leak early.
>
> For option 1.2, I don't think this is a good idea. Not only because the
> assumption (regular GC is enough to clean direct buffers) may not always be
> true, but also it makes harder for finding problems in cases of memory
> overuse. E.g., user configured some direct memory for the user libraries.
> If the library actually use more direct memory then configured, which
> cannot be cleaned by GC because they are still in use, may lead to overuse
> of the total container memory. In that case, if it didn't touch the JVM
> default max direct memory limit, we cannot get a direct memory OOM and it
> will become super hard to understand which part of the configuration need
> to be updated.
>
> For option 1.1, it has the similar problem as 1.2, if the exceeded direct
> memory does not reach the max direct memory limit specified by the
> dedicated parameter. I think it is slightly better than 1.2, only because
> we can tune the parameter.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Mon, Aug 19, 2019 at 2:53 PM Stephan Ewen  wrote:
>
> > About the "-XX:MaxDirectMemorySize" discussion, maybe let me summarize
> it a
> > bit differently:
> >
> > We have the following two options:
> >
> > (1) We let MemorySegments be de-allocated by the GC. That makes it
> segfault
> > safe. But then we need a way to trigger GC in case de-allocation and
> > re-allocation of a bunch of segments happens quickly, which is often the
> > case during batch scheduling or task restart.
> >   - The "-XX:MaxDirectMemorySize" (option 1.1) is one way to do this
> >   - Another way could be to have a dedicated bookkeeping in the
> > MemoryManager (option 1.2), so that this is a number independent of the
> > "-XX:MaxDirectMemorySize" parameter.
> >
> > (2) We manually allocate and de-allocate the memory for the
> MemorySegments
> > (option 2). That way we need not worry about triggering GC by some
> > threshold or bookkeeping, but it is harder to prevent segfaults. We need
> to
> > be very careful about when we release the memory segments (only in the
> > cleanup phase of the main thread).
> >
> > If we go with option 1.1, we probably need to set
> > "-XX:MaxDirectMemorySize" to "off_heap_managed_memory + direct_memory"
> and
> > have "direct_memory" as a separate reserved memory pool. Because if we
> just
> > set "-XX:MaxDirectMemorySize" to "off_heap_managed_memory +
> jvm_overhead",
> > then there will be times when that entire memory is allocated by direct
> > buffers and we have nothing left for the JVM overhead. So we either need
> a
> > way to compensate for that (again some safety margin cutoff value) or we
> > will exceed container memory.
> >
> > If we go with option 1.2, we need to be aware that it takes elaborate
> logic
> > to push recycling of direct buffers without always triggering a full GC.
> >
> >
> > My first guess is that the options will be easiest to do in the following
> > order:
> >
> >   - Option 1.1 with a dedicated direct_memory parameter, as discussed
> > above. We would need to find a way to set the direct_memory parameter by
> > default. We could start with 64 MB and see how it goes in practice. One
> > danger I see is that setting this loo low can cause a bunch of additional
> > GCs compared to before (we need to watch this carefully).
> >
> >   - Option 2. It is actually quite simple to implement, we could try how
> > segfault safe we are at the moment.
> >
> >   - Option 1.2: We

Re: [DISCUSS][CODE STYLE] Usage of Java Optional

[Stephan Ewen]

For the use of optional in private methods: It sounds fine to me, because
it means it is strictly class-internal (between methods and helper methods)
and does not leak beyond that.


On Mon, Aug 19, 2019 at 5:53 PM Andrey Zagrebin 
wrote:

> Hi all,
>
> Sorry for not getting back to this discussion for some time.
> It looks like in general we agree on the initially suggested points:
>
>- Always use Optional only to return nullable values in the API/public
>methods
>   - Only if you can prove that Optional usage would lead to a
>   performance degradation in critical code then return nullable value
>   directly and annotate it with @Nullable
>- Passing an Optional argument to a method can be allowed if it is
>within a private helper method and simplifies the code
>- Optional should not be used for class fields
>
> The first point can be also elaborated by explicitly forbiding
> Optional/Nullable parameters in public methods.
> In general we can always avoid Optional parameters by either overloading
> the method or using a pojo with a builder to pass a set of parameters.
>
> The third point does not prevent from using @Nullable/@Nonnull for class
> fields.
> If we agree to not use Optional for fields then not sure I see any use case
> for SerializableOptional (please comment on that if you have more details).
>
> @Jingsong Lee
> Using Optional in Maps.
> I can see this as a possible use case.
> I would leave this decision to the developer/reviewer to reason about it
> and keep the scope of this discussion to the variables/parameters as it
> seems to be the biggest point of friction atm.
>
> Finally, I see a split regarding the second point:  argument to a method can be allowed if it is within a private helper method
> and simplifies the code>.
> There are people who have a strong opinion against allowing it.
> Let's vote then for whether to allow it or not.
> So far as I see we have the following votes (correct me if wrong and add
> more if you want):
> Piotr+1
> Biao+1
> Timo   -1
> Yu   -1
> Aljoscha -1
> Till  +1
> Igal+1
> Dawid-1
> Me +1
>
> So far: +5 / -4 (Optional arguments in private methods)
>
> Best,
> Andrey
>
>
> On Tue, Aug 6, 2019 at 8:53 AM Piotr Nowojski  wrote:
>
> > Hi Qi,
> >
> > > For example, SingleInputGate is already creating Optional for every
> > BufferOrEvent in getNextBufferOrEvent(). How much performance gain would
> we
> > get if it’s replaced by null check?
> >
> > When I was introducing it there, I have micro-benchmarked this change,
> and
> > there was no visible throughput change in a pure network only micro
> > benchmark (with whole Flink running around it any changes would be even
> > less visible).
> >
> > Piotrek
> >
> > > On 5 Aug 2019, at 15:20, Till Rohrmann  wrote:
> > >
> > > I'd be in favour of
> > >
> > > - Optional for method return values if not performance critical
> > > - Optional can be used for internal methods if it makes sense
> > > - No optional fields
> > >
> > > Cheers,
> > > Till
> > >
> > > On Mon, Aug 5, 2019 at 12:07 PM Aljoscha Krettek 
> > > wrote:
> > >
> > >> Hi,
> > >>
> > >> I’m also in favour of using Optional only for method return values. I
> > >> could be persuaded to allow them for parameters of internal methods
> but
> > I’m
> > >> sceptical about that.
> > >>
> > >> Aljoscha
> > >>
> > >>> On 2. Aug 2019, at 15:35, Yu Li  wrote:
> > >>>
> > >>> TL; DR: I second Timo that we should use Optional only as method
> return
> > >>> type for non-performance critical code.
> > >>>
> > >>> From the example given on our AvroFactory [1] I also noticed that
> > >> Jetbrains
> > >>> marks the OptionalUsedAsFieldOrParameterType inspection as a warning.
> > >> It's
> > >>> relatively easy to understand why it's not suggested to use
> (java.util)
> > >>> Optional as a field since it's not serializable. What made me feel
> > >> curious
> > >>> is that why we shouldn't use it as a parameter type, so I did some
> > >>> investigation and here is what I found:
> > >>>
> > >>> There's a JB blog talking about java8 top tips [2] where we could
> find
> > >> the
> > >>> advice around Optional, there I found another blog telling about the
> > >>> pragmatic approach of using Optional [3]. Reading further we could
> see
> > >> the
> > >>> reason why we shouldn't use Optional as parameter type, please allow
> me
> > >> to
> > >>> quote here:
> > >>>
> > >>> It is often the case that domain objects hang about in memory for a
> > fair
> > >>> while, as processing in the application occurs, making each optional
> > >>> instance rather long-lived (tied to the lifetime of the domain
> object).
> > >> By
> > >>> contrast, the Optionalinstance returned from the getter is likely to
> be
> > >>> very short-lived. The caller will call the getter, interpret the
> > result,
> > >>> and then move on. If you know anything about garbage collection
> you'll
> > >> know
> > >>> that the JVM handl

Re: [DISCUSS][CODE STYLE] Create collections always with initial capacity

[Stephan Ewen]

@Andrey Will you open a PR to add this to the code style?

On Mon, Aug 19, 2019 at 11:51 AM Andrey Zagrebin 
wrote:

> Hi All,
>
> It looks like this proposal has an approval and we can conclude this
> discussion.
> Additionally, I agree with Piotr we should really force the proven good
> reasoning for setting the capacity to avoid confusion, redundancy and other
> already mentioned things while reading and maintaining the code.
> Ideally the need of setting the capacity should be either immediately clear
> (e.g. perf etc) or explained in comments if it is non-trivial.
> Although, it can easily enter a grey zone, so I would not demand strictly
> performance measurement proof e.g. if the size is known and it is "per
> record" code.
> At the end of the day it is a decision of the code developer and reviewer.
>
> The conclusion is then:
> Set the initial capacity only if there is a good proven reason to do it.
> Otherwise do not clutter the code with it.
>
> Best,
> Andrey
>
> On Thu, Aug 1, 2019 at 5:10 PM Piotr Nowojski  wrote:
>
> > Hi,
> >
> > > - a bit more code, increases maintenance burden.
> >
> > I think there is even more to that. It’s almost like a code duplication,
> > albeit expressed in very different way, with all of the drawbacks of
> > duplicated code: initial capacity can drift out of sync, causing
> confusion.
> > Also it’s not “a bit more code”, it might be non trivial
> > reasoning/calculation how to set the initial value. Whenever we change
> > something/refactor the code, "maintenance burden” will mostly come from
> > that.
> >
> > Also I think this just usually falls under a premature optimisation rule.
> >
> > Besides:
> >
> > > The conclusion is the following at the moment:
> > > Only set the initial capacity if you have a good idea about the
> expected
> > size.
> >
> > I would add a clause to set the initial capacity “only for good proven
> > reasons”. It’s not about whether we can set it, but whether it makes
> sense
> > to do so (to avoid the before mentioned "maintenance burden”).
> >
> > Piotrek
> >
> > > On 1 Aug 2019, at 14:41, Xintong Song  wrote:
> > >
> > > +1 on setting initial capacity only when have good expectation on the
> > > collection size.
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Thu, Aug 1, 2019 at 2:32 PM Andrey Zagrebin 
> > wrote:
> > >
> > >> Hi all,
> > >>
> > >> As you probably already noticed, Stephan has triggered a discussion
> > thread
> > >> about code style guide for Flink [1]. Recently we were discussing
> > >> internally some smaller concerns and I would like start separate
> threads
> > >> for them.
> > >>
> > >> This thread is about creating collections always with initial
> capacity.
> > As
> > >> you might have seen, some parts of our code base always initialise
> > >> collections with some non-default capacity. You can even activate a
> > check
> > >> in IntelliJ Idea that can monitor and highlight creation of collection
> > >> without initial capacity.
> > >>
> > >> Pros:
> > >> - performance gain if there is a good reasoning about initial capacity
> > >> - the capacity is always deterministic and does not depend on any
> > changes
> > >> of its default value in Java
> > >> - easy to follow: always initialise, has IDE support for detection
> > >>
> > >> Cons (for initialising w/o good reasoning):
> > >> - We are trying to outsmart JVM. When there is no good reasoning about
> > >> initial capacity, we can rely on JVM default value.
> > >> - It is even confusing e.g. for hash maps as the real size depends on
> > the
> > >> load factor.
> > >> - It would only add minor performance gain.
> > >> - a bit more code, increases maintenance burden.
> > >>
> > >> The conclusion is the following at the moment:
> > >> Only set the initial capacity if you have a good idea about the
> expected
> > >> size.
> > >>
> > >> Please, feel free to share you thoughts.
> > >>
> > >> Best,
> > >> Andrey
> > >>
> > >> [1]
> > >>
> > >>
> >
> http://mail-archives.apache.org/mod_mbox/flink-dev/201906.mbox/%3ced91df4b-7cab-4547-a430-85bc710fd...@apache.org%3E
> > >>
> >
> >
>

Re: [DISCUSS][CODE STYLE] Breaking long function argument lists and chained method calls

[Stephan Ewen]

I personally prefer not to break lines with few parameters.
It just feels unnecessarily clumsy to parse the breaks if there are only
two or three arguments with short names.

So +1
  - for a hard line length limit
  - allowing arguments on the same line if below that limit
  - with consistent argument breaking when that length is exceeded
  - developers can break before that if they feel it helps with readability.

This should be similar to what we have, except for enforcing the line
length limit.

I think our Java guide originally suggested 120 characters line length, but
we can reduce that to 100 if a majority argues for that, but it is separate
discussion.
We uses shorter lines in Scala (100 chars) because Scala code becomes
harder to read faster with long lines.


On Mon, Aug 19, 2019 at 10:45 AM Andrey Zagrebin 
wrote:

> Hi Everybody,
>
> Thanks for your feedback guys and sorry for not getting back to the
> discussion for some time.
>
> @SHI Xiaogang
> About breaking lines for thrown exceptions:
> Indeed that would prevent growing the throw clause indefinitely.
> I am a bit concerned about putting the right parenthesis and/or throw
> clause on the next line
> because in general we do not it and there are a lot of variations of how
> and what to put to the next line so it needs explicit memorising.
> Also, we do not have many checked exceptions and usually avoid them.
> Although I am not a big fan of many function arguments either but this
> seems to be a bigger problem in the code base.
> I would be ok to not enforce anything for exceptions atm.
>
> @Chesnay Schepler 
> Thanks for mentioning automatic checks.
> Indeed, pointing out this kind of style issues during PR reviews is very
> tedious
> and cannot really force it without automated tools.
> I would still consider the outcome of this discussion as a soft
> recommendation atm (which we also have for some other things in the code
> style draft).
> We need more investigation about how to enforce things. I am not so
> knowledgable about code style/IDE checks.
> From the first glance I also do not see a simple way. If somebody has more
> insight please share your experience.
>
> @Biao Liu 
> Line length limitation:
> I do not see anything for Java, only for Scala: 100 (also enforced by build
> AFAIK).
> From what I heard there has been already some discussion about the hard
> limit for the line length.
> Although quite some people are in favour of it (including me) and seems to
> be a nice limitation,
> there are some practical implication about it.
> Historically, Flink did not have any code style checks and huge chunks of
> code base have to be reformatted destroying the commit history.
> Another thing is value for the limit. Nowadays, we have wide screens and do
> not often even need to scroll.
> Nevertheless, we can kick off another discussion about the line length
> limit and enforcing it.
> Atm I see people to adhere to a soft recommendation of 120 line length for
> Java because it is usually a bit more verbose comparing to Scala.
>
> *Question 1*:
> I would be ok to always break line if there is more than one chained call.
> There are a lot of places where this is only one short call I would not
> break line in this case.
> If it is too confusing I would be ok to stick to the rule to break either
> all or none.
> Thanks for pointing out this explicitly: For a chained method calls, the
> new line should be started with the dot.
> I think it should be also part of the rule if forced.
>
> *Question 2:*
> The indent of new line should be 1 tab or 2 tabs? (I assume it matters only
> for function arguments)
> This is a good point which again probably deserves a separate thread.
> We also had an internal discussion about it. I would be also in favour of
> resolving it into one way.
> Atm we indeed have 2 ways in our code base which are again soft
> recommendations.
> The problem is mostly with enforcing it automatically.
> The approach with extra indentation also needs IDE setup otherwise it is
> annoying
> that after every function cut/paste, e.g. Idea changes the format to one
> indentation automatically and often people do not notice it.
>
> I suggest we still finish this discussion to a point of achieving a soft
> recommendation which we can also reconsider
> when there are more ideas about automatically enforcing these things.
>
> Best,
> Andrey
>
> On Sat, Aug 3, 2019 at 7:51 AM SHI Xiaogang 
> wrote:
>
> > Hi Chesnay,
> >
> > Thanks a lot for your reminder.
> >
> > For Intellij settings, the style i proposed can be configured as below
> > * Method declaration parameters: chop down if long
> > * align when multiple: YES
> > * new line after '(': YES
> > * place ')' on new line: YES
> > * Method call arguments: chop down if long
> > * align when multiple: YES
> > * take priority over call chain wrapping: YES
> > * new line after '(': YES
> > * place ')' on new line: YES
> > * Throws list: chop down if long
> > * 

Re: [DISCUSS] FLIP-49: Unified Memory Configuration for TaskExecutors

[Stephan Ewen]

use they are still in use, may lead to
> > overuse
> > > of the total container memory. In that case, if it didn't touch the JVM
> > > default max direct memory limit, we cannot get a direct memory OOM and
> it
> > > will become super hard to understand which part of the configuration
> need
> > > to be updated.
> > >
> > > For option 1.1, it has the similar problem as 1.2, if the exceeded
> direct
> > > memory does not reach the max direct memory limit specified by the
> > > dedicated parameter. I think it is slightly better than 1.2, only
> because
> > > we can tune the parameter.
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Mon, Aug 19, 2019 at 2:53 PM Stephan Ewen  wrote:
> > >
> > > > About the "-XX:MaxDirectMemorySize" discussion, maybe let me
> summarize
> > > it a
> > > > bit differently:
> > > >
> > > > We have the following two options:
> > > >
> > > > (1) We let MemorySegments be de-allocated by the GC. That makes it
> > > segfault
> > > > safe. But then we need a way to trigger GC in case de-allocation and
> > > > re-allocation of a bunch of segments happens quickly, which is often
> > the
> > > > case during batch scheduling or task restart.
> > > >   - The "-XX:MaxDirectMemorySize" (option 1.1) is one way to do this
> > > >   - Another way could be to have a dedicated bookkeeping in the
> > > > MemoryManager (option 1.2), so that this is a number independent of
> the
> > > > "-XX:MaxDirectMemorySize" parameter.
> > > >
> > > > (2) We manually allocate and de-allocate the memory for the
> > > MemorySegments
> > > > (option 2). That way we need not worry about triggering GC by some
> > > > threshold or bookkeeping, but it is harder to prevent segfaults. We
> > need
> > > to
> > > > be very careful about when we release the memory segments (only in
> the
> > > > cleanup phase of the main thread).
> > > >
> > > > If we go with option 1.1, we probably need to set
> > > > "-XX:MaxDirectMemorySize" to "off_heap_managed_memory +
> direct_memory"
> > > and
> > > > have "direct_memory" as a separate reserved memory pool. Because if
> we
> > > just
> > > > set "-XX:MaxDirectMemorySize" to "off_heap_managed_memory +
> > > jvm_overhead",
> > > > then there will be times when that entire memory is allocated by
> direct
> > > > buffers and we have nothing left for the JVM overhead. So we either
> > need
> > > a
> > > > way to compensate for that (again some safety margin cutoff value) or
> > we
> > > > will exceed container memory.
> > > >
> > > > If we go with option 1.2, we need to be aware that it takes elaborate
> > > logic
> > > > to push recycling of direct buffers without always triggering a full
> > GC.
> > > >
> > > >
> > > > My first guess is that the options will be easiest to do in the
> > following
> > > > order:
> > > >
> > > >   - Option 1.1 with a dedicated direct_memory parameter, as discussed
> > > > above. We would need to find a way to set the direct_memory parameter
> > by
> > > > default. We could start with 64 MB and see how it goes in practice.
> One
> > > > danger I see is that setting this loo low can cause a bunch of
> > additional
> > > > GCs compared to before (we need to watch this carefully).
> > > >
> > > >   - Option 2. It is actually quite simple to implement, we could try
> > how
> > > > segfault safe we are at the moment.
> > > >
> > > >   - Option 1.2: We would not touch the "-XX:MaxDirectMemorySize"
> > > parameter
> > > > at all and assume that all the direct memory allocations that the JVM
> > and
> > > > Netty do are infrequent enough to be cleaned up fast enough through
> > > regular
> > > > GC. I am not sure if that is a valid assumption, though.
> > > >
> > > > Best,
> > > > Stephan
> > > >
> > > >
> > > >
> > > > On Fri, Aug 16, 2019 at 2:16 PM Xintong Song 
> > > > wrote:
> > > >
> > > > > T

Re: [DISCUSS][CODE STYLE] Usage of Java Optional

[Stephan Ewen]

I think Dawid raised a very good point here.
One of the outcomes should be that we are consistent in our recommendations
and requests during PR reviews. Otherwise we'll just confuse contributors.

So I would be
  +1 for someone to use Optional in a private method if they believe it is
helpful
  -1 to push contributors during reviews to do that


On Tue, Aug 20, 2019 at 9:42 AM Dawid Wysakowicz 
wrote:

> Hi Andrey,
>
> Just wanted to quickly elaborate on my opinion. I wouldn't say I am -1,
> just -0 for the Optionals in private methods. I am ok with not
> forbidding them there. I just think in all cases there is a better
> solution than passing the Optionals around, even in private methods. I
> just hope the outcome of the discussion won't be that it is no longer
> allowed to suggest those during review.
>
> Best,
>
> Dawid
>
> On 19/08/2019 17:53, Andrey Zagrebin wrote:
> > Hi all,
> >
> > Sorry for not getting back to this discussion for some time.
> > It looks like in general we agree on the initially suggested points:
> >
> >- Always use Optional only to return nullable values in the API/public
> >methods
> >   - Only if you can prove that Optional usage would lead to a
> >   performance degradation in critical code then return nullable value
> >   directly and annotate it with @Nullable
> >- Passing an Optional argument to a method can be allowed if it is
> >within a private helper method and simplifies the code
> >- Optional should not be used for class fields
> >
> > The first point can be also elaborated by explicitly forbiding
> > Optional/Nullable parameters in public methods.
> > In general we can always avoid Optional parameters by either overloading
> > the method or using a pojo with a builder to pass a set of parameters.
> >
> > The third point does not prevent from using @Nullable/@Nonnull for class
> > fields.
> > If we agree to not use Optional for fields then not sure I see any use
> case
> > for SerializableOptional (please comment on that if you have more
> details).
> >
> > @Jingsong Lee
> > Using Optional in Maps.
> > I can see this as a possible use case.
> > I would leave this decision to the developer/reviewer to reason about it
> > and keep the scope of this discussion to the variables/parameters as it
> > seems to be the biggest point of friction atm.
> >
> > Finally, I see a split regarding the second point:  > argument to a method can be allowed if it is within a private helper
> method
> > and simplifies the code>.
> > There are people who have a strong opinion against allowing it.
> > Let's vote then for whether to allow it or not.
> > So far as I see we have the following votes (correct me if wrong and add
> > more if you want):
> > Piotr+1
> > Biao+1
> > Timo   -1
> > Yu   -1
> > Aljoscha -1
> > Till  +1
> > Igal+1
> > Dawid-1
> > Me +1
> >
> > So far: +5 / -4 (Optional arguments in private methods)
> >
> > Best,
> > Andrey
> >
> >
> > On Tue, Aug 6, 2019 at 8:53 AM Piotr Nowojski 
> wrote:
> >
> >> Hi Qi,
> >>
> >>> For example, SingleInputGate is already creating Optional for every
> >> BufferOrEvent in getNextBufferOrEvent(). How much performance gain
> would we
> >> get if it’s replaced by null check?
> >>
> >> When I was introducing it there, I have micro-benchmarked this change,
> and
> >> there was no visible throughput change in a pure network only micro
> >> benchmark (with whole Flink running around it any changes would be even
> >> less visible).
> >>
> >> Piotrek
> >>
> >>> On 5 Aug 2019, at 15:20, Till Rohrmann  wrote:
> >>>
> >>> I'd be in favour of
> >>>
> >>> - Optional for method return values if not performance critical
> >>> - Optional can be used for internal methods if it makes sense
> >>> - No optional fields
> >>>
> >>> Cheers,
> >>> Till
> >>>
> >>> On Mon, Aug 5, 2019 at 12:07 PM Aljoscha Krettek 
> >>> wrote:
> >>>
>  Hi,
> 
>  I’m also in favour of using Optional only for method return values. I
>  could be persuaded to allow them for parameters of internal methods
> but
> >> I’m
>  sceptical about that.
> 
>  Aljoscha
> 
> > On 2. Aug 2019, at 15:35, Yu Li  wrote:
> >
> > TL; DR: I second Timo that we should use Optional only as method
> return
> > type for non-performance critical code.
> >
> > From the example given on our AvroFactory [1] I also noticed that
>  Jetbrains
> > marks the OptionalUsedAsFieldOrParameterType inspection as a warning.
>  It's
> > relatively easy to understand why it's not suggested to use
> (java.util)
> > Optional as a field since it's not serializable. What made me feel
>  curious
> > is that why we shouldn't use it as a parameter type, so I did some
> > investigation and here is what I found:
> >
> > There's a JB blog talking about java8 top tips [2] where we could
> find
>  the
> > advice around Optional, there I found anothe

Re: [DISCUSS] Flink client api enhancement for downstream project

[Stephan Ewen]

gt; >> >> > Thanks tison for the effort. I left a few comments.
> >> >> >
> >> >> >
> >> >> > Zili Chen  于2019年7月31日周三 下午8:24写道：
> >> >> >
> >> >> >> Hi Flavio,
> >> >> >>
> >> >> >> Thanks for your reply.
> >> >> >>
> >> >> >> Either current impl and in the design, ClusterClient
> >> >> >> never takes responsibility for generating JobGraph.
> >> >> >> (what you see in current codebase is several class methods)
> >> >> >>
> >> >> >> Instead, user describes his program in the main method
> >> >> >> with ExecutionEnvironment apis and calls env.compile()
> >> >> >> or env.optimize() to get FlinkPlan and JobGraph respectively.
> >> >> >>
> >> >> >> For listing main classes in a jar and choose one for
> >> >> >> submission, you're now able to customize a CLI to do it.
> >> >> >> Specifically, the path of jar is passed as arguments and
> >> >> >> in the customized CLI you list main classes, choose one
> >> >> >> to submit to the cluster.
> >> >> >>
> >> >> >> Best,
> >> >> >> tison.
> >> >> >>
> >> >> >>
> >> >> >> Flavio Pompermaier  于2019年7月31日周三 下午8:12写道：
> >> >> >>
> >> >> >>> Just one note on my side: it is not clear to me whether the
> client
> >> >> needs
> >> >> >> to
> >> >> >>> be able to generate a job graph or not.
> >> >> >>> In my opinion, the job jar must resides only on the
> >> server/jobManager
> >> >> >> side
> >> >> >>> and the client requires a way to get the job graph.
> >> >> >>> If you really want to access to the job graph, I'd add a
> dedicated
> >> >> method
> >> >> >>> on the ClusterClient. like:
> >> >> >>>
> >> >> >>>   - getJobGraph(jarId, mainClass): JobGraph
> >> >> >>>   - listMainClasses(jarId): List
> >> >> >>>
> >> >> >>> These would require some addition also on the job manager
> endpoint
> >> as
> >> >> >>> well..what do you think?
> >> >> >>>
> >> >> >>> On Wed, Jul 31, 2019 at 12:42 PM Zili Chen  >
> >> >> wrote:
> >> >> >>>
> >> >> >>>> Hi all,
> >> >> >>>>
> >> >> >>>> Here is a document[1] on client api enhancement from our
> >> perspective.
> >> >> >>>> We have investigated current implementations. And we propose
> >> >> >>>>
> >> >> >>>> 1. Unify the implementation of cluster deployment and job
> >> submission
> >> >> in
> >> >> >>>> Flink.
> >> >> >>>> 2. Provide programmatic interfaces to allow flexible job and
> >> cluster
> >> >> >>>> management.
> >> >> >>>>
> >> >> >>>> The first proposal is aimed at reducing code paths of cluster
> >> >> >> deployment
> >> >> >>>> and
> >> >> >>>> job submission so that one can adopt Flink in his usage easily.
> >> The
> >> >> >>> second
> >> >> >>>> proposal is aimed at providing rich interfaces for advanced
> users
> >> >> >>>> who want to make accurate control of these stages.
> >> >> >>>>
> >> >> >>>> Quick reference on open questions:
> >> >> >>>>
> >> >> >>>> 1. Exclude job cluster deployment from client side or redefine
> the
> >> >> >>> semantic
> >> >> >>>> of job cluster? Since it fits in a process quite different from
> >> >> session
> >> >> >>>> cluster deployment and job submission.
> >> >> >>>>
> >> >> >>>> 2. Maintain the codepaths handling class
> o.a.f.api.common.Program
> >

Re: [DISCUSS] Upgrade kinesis connector to Apache 2.0 License and include it in official release

[Stephan Ewen]

Just FYI - Becket, Aljoscha, and me are working on fleshing out the
remaining details of FLIP-27 (new source API).
We will share this as soon as we have made some progress on some of the
details.

The Kinesis connector would be one of the first that we would try to also
implement in that new API, as a validation that it is powerful and flexible
enough.

If the upgrade involved major refactoring, would it make sense combine
these efforts?

Best,
Stephan


On Tue, Aug 20, 2019 at 9:12 AM Dyana Rose  wrote:

> Ahh, brilliant, I had myself on notifications for the streams adapter
> releases, but must have missed it. That's great news.
>
> I've got the branch prepped for moving over to Apache 2.0, but staying on
> KCL 1.x, which requires the least amount of change.
>
> Considering the large amount of change required to update to KCL/SDK 2.x I
> would recommend that be done in a parallel task. Making both connectors
> available then for usage, 1.x and 2.x. If that makes sense.
>
> The branch I push will have the English Language documents updated, but not
> have the Chinese Language documents updated. Is there a process for this?
>
> Thanks,
> Dyana
>
> On Mon, 19 Aug 2019 at 19:08, Bowen Li  wrote:
>
> > Hi all,
> >
> > A while back we discussed upgrading flink-connector-kinesis module to
> > Apache 2.0 license so that its jar can be published as part of official
> > Flink releases. Given we have a large user base using Flink with
> > kinesis/dynamodb streams, it'll free users from building and maintaining
> > the module themselves, and improve user and developer experience. A
> ticket
> > was created [1] but has been idle mainly due to new releases of some aws
> > libs are not available yet then.
> >
> > As of today I see that all flink-connector-kinesis's aws dependencies
> have
> > been updated to Apache 2.0 license and are released. They include:
> >
> > - aws-java-sdk-kinesis
> > - aws-java-sdk-sts
> > - amazon-kinesis-client
> > - amazon-kinesis-producer (Apache 2.0 from 0.13.1, released 18 days ago)
> > [2]
> > - dynamodb-streams-kinesis-adapter (Apache 2.0 from 1.5.0, released 7
> days
> > ago) [3]
> >
> > Therefore, I'd suggest we kick off the initiative and aim for release
> 1.10
> > which is roughly 3 months away, leaving us plenty of time to finish.
> > According to @Dyana 's comment in the ticket [1], seems some large chunks
> > of changes need to be made into multiple parts than simply upgrading lib
> > versions, so we can further break the JIRA down into sub-tasks to limit
> > scope of each change for easier code review.
> >
> > @Dyana would you still be interested in carrying the responsibility and
> > forwarding the effort?
> >
> > Thanks,
> > Bowen
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-12847
> > [2] https://github.com/awslabs/amazon-kinesis-producer/releases
> > [3] https://github.com/awslabs/dynamodb-streams-kinesis-adapter/releases
> >
> >
> >
>
> --
>
> Dyana Rose
> Software Engineer
>
>
> W: www.salecycle.com 
> [image: Airline & Travel Booking Trends - Download Report]
> 
>

Re: [VOTE] Flink Project Bylaws

[Stephan Ewen]

> > > > >>>>> +1 (non-binding)
> > > > >>>>> Thanks Becket.
> > > > >>>>> I've learned a lot from current bylaws.
> > > > >>>>>
> > > > >>>>> Best,
> > > > >>>>> Jingsong Lee
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>
> > --
> > > > >>>>> From:Yu Li 
> > > > >>>>> Send Time:2019年8月13日(星期二) 17:48
> > > > >>>>> To:dev 
> > > > >>>>> Subject:Re: [VOTE] Flink Project Bylaws
> > > > >>>>>
> > > > >>>>> +1 (non-binding)
> > > > >>>>>
> > > > >>>>> Thanks for the efforts Becket!
> > > > >>>>>
> > > > >>>>> Best Regards,
> > > > >>>>> Yu
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> On Tue, 13 Aug 2019 at 16:09, Xintong Song <
> > tonysong...@gmail.com>
> > > > >>>> wrote:
> > > > >>>>>> +1 (non-binding)
> > > > >>>>>>
> > > > >>>>>> Thank you~
> > > > >>>>>>
> > > > >>>>>> Xintong Song
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> On Tue, Aug 13, 2019 at 1:48 PM Robert Metzger <
> > > > >> rmetz...@apache.org>
> > > > >>>>>> wrote:
> > > > >>>>>>
> > > > >>>>>>> +1 (binding)
> > > > >>>>>>>
> > > > >>>>>>> On Tue, Aug 13, 2019 at 1:47 PM Becket Qin <
> > becket@gmail.com
> > > > >>>>> wrote:
> > > > >>>>>>>> Thanks everyone for voting.
> > > > >>>>>>>>
> > > > >>>>>>>> For those who have already voted, just want to bring this up
> > to
> > > > >>>> your
> > > > >>>>>>>> attention that there is a minor clarification to the bylaws
> > > > >> wiki
> > > > >>>> this
> > > > >>>>>>>> morning. The change is in bold format below:
> > > > >>>>>>>>
> > > > >>>>>>>> one +1 from a committer followed by a Lazy approval (not
> > > > >> counting
> > > > >>>> the
> > > > >>>>>>> vote
> > > > >>>>>>>>> of the contributor), moving to lazy majority if a -1 is
> > > > >>> received.
> > > > >>>>>>>>
> > > > >>>>>>>> Note that this implies that committers can +1 their own
> > commits
> > > > >>> and
> > > > >>>>>> merge
> > > > >>>>>>>>> right away. *However, the committe**rs should use their
> best
> > > > >>>>>> judgement
> > > > >>>>>>> to
> > > > >>>>>>>>> respect the components expertise and ongoing development
> > > > >> plan.*
> > > > >>>>>>>>
> > > > >>>>>>>> This addition does not really change anything the bylaws
> meant
> > > > >> to
> > > > >>>>> set.
> > > > >>>>>> It
> > > > >>>>>>>> is simply a clarification. If anyone who have casted the
> vote
> > > > >>>>> objects,
> > > > >>>>>>>> please feel free to withdraw the vote.
> > > > >>>>>>>>
> > > > >>>>>>>> Thanks,
> > > > >>>>>>>>
> > > > >>>>>>>> Jiangjie (Becket) Qin
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>

Re: flink release-1.8.0 Flink-avro unit tests failed

[Stephan Ewen]

Thanks, looks like you diagnosed it correctly. environment specific
encoding settings.

Could you open a ticket (maybe a PR) to set the encoding and make the test
stable across environments?

On Mon, Aug 19, 2019 at 9:46 PM Ethan Li  wrote:

> It’s probably the encoding problem. The environment I ran the unit tests
> on uses ANSI_X3.4-1968
>
> It looks like we have to use "en_US.UTF-8"
>
>
> > On Aug 19, 2019, at 1:44 PM, Ethan Li  wrote:
> >
> > Hello,
> >
> > Not sure if anyone encountered this issue before.  I tried to run “mvn
> clean install” on flink release-1.8, but some unit tests in flink-arvo
> module failed:
> >
> >
> > [ERROR] Tests run: 12, Failures: 4, Errors: 0, Skipped: 0, Time elapsed:
> 4.81 s <<< FAILURE! - in
> org.apache.flink.formats.avro.typeutils.AvroTypeExtractionTest
> > [ERROR] testSimpleAvroRead[Execution mode =
> CLUSTER](org.apache.flink.formats.avro.typeutils.AvroTypeExtractionTest)
> Time elapsed: 0.438 s  <<< FAILURE!
> > java.lang.AssertionError:
> > Different elements in arrays: expected 2 elements and received 2
> > files: [/tmp/junit5386344396421857812/junit6023978980792200274.tmp/4,
> /tmp/junit5386344396421857812/junit6023978980792200274.tmp/2,
> /tmp/junit5386344396421857812/junit6023978980792200274.tmp/1,
> /tmp/junit5386344396421857812/junit6023978980792200274.tmp/3]
> >  expected: [{"name": "Alyssa", "favorite_number": 256, "favorite_color":
> null, "type_long_test": null, "type_double_test": 123.45, "type_null_test":
> null, "type_bool_test": true, "type_array_string": ["ELEMENT 1", "ELEMENT
> 2"], "type_array_boolean": [true, false], "type_nullable_array": null,
> "type_enum": "GREEN", "type_map": {"KEY 2": 17554, "KEY 1": 8546456},
> "type_fixed": null, "type_union": null, "type_nested": {"num": 239,
> "street": "Baker Street", "city": "London", "state": "London", "zip": "NW1
> 6XE"}, "type_bytes": {"bytes":
> "\u\u\u\u\u\u\u\u\u\u"},
> "type_date": 2014-03-01, "type_time_millis": 12:12:12.000,
> "type_time_micros": 123456, "type_timestamp_millis":
> 2014-03-01T12:12:12.321Z, "type_timestamp_micros": 123456,
> "type_decimal_bytes": {"bytes": "\u0007?"}, "type_decimal_fixed": [7,
> -48]}, {"name": "Charlie", "favorite_number": null, "favorite_color":
> "blue", "type_long_test": 1337, "type_double_test": 1.337,
> "type_null_test": null, "type_bool_test": false, "type_array_string": [],
> "type_array_boolean": [], "type_nullable_array": null, "type_enum": "RED",
> "type_map": {}, "type_fixed": null, "type_union": null, "type_nested":
> {"num": 239, "street": "Baker Street", "city": "London", "state": "London",
> "zip": "NW1 6XE"}, "type_bytes": {"bytes":
> "\u\u\u\u\u\u\u\u\u\u"},
> "type_date": 2014-03-01, "type_time_millis": 12:12:12.000,
> "type_time_micros": 123456, "type_timestamp_millis":
> 2014-03-01T12:12:12.321Z, "type_timestamp_micros": 123456,
> "type_decimal_bytes": {"bytes": "\u0007?"}, "type_decimal_fixed": [7, -48]}]
> >  received: [{"name": "Alyssa", "favorite_number": 256, "favorite_color":
> null, "type_long_test": null, "type_double_test": 123.45, "type_null_test":
> null, "type_bool_test": true, "type_array_string": ["ELEMENT 1", "ELEMENT
> 2"], "type_array_boolean": [true, false], "type_nullable_array": null,
> "type_enum": "GREEN", "type_map": {"KEY 2": 17554, "KEY 1": 8546456},
> "type_fixed": null, "type_union": null, "type_nested": {"num": 239,
> "street": "Baker Street", "city": "London", "state": "London", "zip": "NW1
> 6XE"}, "type_bytes": {"bytes":
> "\u\u\u\u\u\u\u\u\u\u"},
> "type_date": 2014-03-01, "type_time_millis": 12:12:12.000,
> "type_time_micros": 123456, "type_timestamp_millis":
> 2014-03-01T12:12:12.321Z, "type_timestamp_micros": 123456,
> "type_decimal_bytes": {"bytes": "\u0007??"}, "type_decimal_fixed": [7,
> -48]}, {"name": "Charlie", "favorite_number": null, "favorite_color":
> "blue", "type_long_test": 1337, "type_double_test": 1.337,
> "type_null_test": null, "type_bool_test": false, "type_array_string": [],
> "type_array_boolean": [], "type_nullable_array": null, "type_enum": "RED",
> "type_map": {}, "type_fixed": null, "type_union": null, "type_nested":
> {"num": 239, "street": "Baker Street", "city": "London", "state": "London",
> "zip": "NW1 6XE"}, "type_bytes": {"bytes":
> "\u\u\u\u\u\u\u\u\u\u"},
> "type_date": 2014-03-01, "type_time_millis": 12:12:12.000,
> "type_time_micros": 123456, "type_timestamp_millis":
> 2014-03-01T12:12:12.321Z, "type_timestamp_micros": 123456,
> "type_decimal_bytes": {"bytes": "\u0007??"}, "type_decimal_fixed": [7,
> -48]}]
> >   at
> org.apache.flink.formats.avro.typeutils.AvroTypeExtractionTest.after(AvroTypeExtractionTest.java:76)
> >
> >
> >
> > Comparing “expected” with “received”, there is really some question mark
> difference.
> >
> > For example, in “expected’, it’s
> >
> > "type_decimal_bytes": {"bytes": "\u0007?”}

Re: [VOTE] Apache Flink 1.9.0, release candidate #3

[Stephan Ewen]

+1 (binding)

 - Downloaded the binary release tarball
 - started a standalone cluster with four nodes
 - ran some examples through the Web UI
 - checked the logs
 - created a project from the Java quickstarts maven archetype
 - ran a multi-stage DataSet job in batch mode
 - killed as TaskManager and verified correct restart behavior, including
failover region backtracking


I found a few issues, and a common theme here is confusing error reporting
and logging.

(1) When testing batch failover and killing a TaskManager, the job reports
as the failure cause "org.apache.flink.util.FlinkException: The assigned
slot 6d0e469d55a2630871f43ad0f89c786c_0 was removed."
I think that is a pretty bad error message, as a user I don't know what
that means. Some internal book keeping thing?
You need to know a lot about Flink to understand that this means
"TaskManager failure".
https://issues.apache.org/jira/browse/FLINK-13805
I would not block the release on this, but think this should get pretty
urgent attention.

(2) The Metric Fetcher floods the log with error messages when a
TaskManager is lost.
 There are many exceptions being logged by the Metrics Fetcher due to
not reaching the TM any more.
 This pollutes the log and drowns out the original exception and the
meaningful logs from the scheduler/execution graph.
 https://issues.apache.org/jira/browse/FLINK-13806
 Again, I would not block the release on this, but think this should
get pretty urgent attention.

(3) If you put "web.submit.enable: false" into the configuration, the web
UI will still display the "SubmitJob" page, but errors will
continuously pop up, stating "Unable to load requested file /jars."
https://issues.apache.org/jira/browse/FLINK-13799

(4) REST endpoint logs ERROR level messages when selecting the
"Checkpoints" tab for batch jobs. That does not seem correct.
 https://issues.apache.org/jira/browse/FLINK-13795

Best,
Stephan




On Tue, Aug 20, 2019 at 11:32 AM Tzu-Li (Gordon) Tai 
wrote:

> +1
>
> Legal checks:
> - verified signatures and hashes
> - New bundled Javascript dependencies for flink-runtime-web are correctly
> reflected under licenses-binary and NOTICE file.
> - locally built from source (Scala 2.12, without Hadoop)
> - No missing artifacts in staging repo
> - No binaries in source release
>
> Functional checks:
> - Quickstart working (both in IDE + job submission)
> - Simple State Processor API program that performs offline key schema
> migration (RocksDB backend). Generated savepoint is valid to restore from.
> - All E2E tests pass locally
> - Didn’t notice any issues with the new WebUI
>
> Cheers,
> Gordon
>
> On Tue, Aug 20, 2019 at 3:53 AM Zili Chen  wrote:
>
> > +1 (non-binding)
> >
> > - build from source: OK(8u212)
> > - check local setup tutorial works as expected
> >
> > Best,
> > tison.
> >
> >
> > Yu Li  于2019年8月20日周二 上午8:24写道：
> >
> > > +1 (non-binding)
> > >
> > > - checked release notes: OK
> > > - checked sums and signatures: OK
> > > - repository appears to contain all expected artifacts
> > > - source release
> > >  - contains no binaries: OK
> > >  - contains no 1.9-SNAPSHOT references: OK
> > >  - build from source: OK (8u102)
> > > - binary release
> > >  - no examples appear to be missing
> > >  - started a cluster; WebUI reachable, example ran successfully
> > > - checked README.md file and found nothing unexpected
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Tue, 20 Aug 2019 at 01:16, Tzu-Li (Gordon) Tai  >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Release candidate #3 for Apache Flink 1.9.0 is now ready for your
> > review.
> > > >
> > > > Please review and vote on release candidate #3 for version 1.9.0, as
> > > > follows:
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > >
> > > > The complete staging area is available for your review, which
> includes:
> > > > * JIRA release notes [1],
> > > > * the official Apache source release and binary convenience releases
> to
> > > be
> > > > deployed to dist.apache.org [2], which are signed with the key with
> > > > fingerprint 1C1E2394D3194E1944613488F320986D35C33D6A [3],
> > > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > > * source code tag “release-1.9.0-rc3” [5].
> > > > * pull requests for the release note documentation [6] and
> announcement
> > > > blog post [7].
> > > >
> > > > As proposed in the RC2 vote thread [8], for RC3 we are only
> > > cherry-picking
> > > > minimal specific changes on top of RC2 to be able to reasonably carry
> > > over
> > > > previous testing efforts and effectively require a shorter voting
> time.
> > > >
> > > > The only extra commits in this RC, compared to RC2, are the
> following:
> > > > - c2d9aeac [FLINK-13231] [pubsub] Replace Max outstanding
> > acknowledgement
> > > > ids limit with a FlinkConnectorRateLimiter
> > > > - d8941711 [FLINK-1369

Re: [VOTE] FLIP-52: Remove legacy Program interface.

[Stephan Ewen]

+1

On Wed, Aug 21, 2019 at 1:07 PM Kostas Kloudas  wrote:

> Hi all,
>
> Following the FLIP process, this is a voting thread dedicated to the
> FLIP-52.
> As shown from the corresponding discussion thread [1], we seem to agree
> that
> the Program interface can be removed, so let's make it also official
> with a vote.
>
> Cheers,
> Kostas
>
>
> [1]
> https://lists.apache.org/thread.html/0dbd0a4adf9ad00d6ad869dffc8820f6ce4c1969e1ea4aafb1dd0aa4@%3Cdev.flink.apache.org%3E
>

Re: CiBot Update

[Stephan Ewen]

Nice, thanks!

On Thu, Aug 22, 2019 at 3:59 AM Zili Chen  wrote:

> Thanks for your announcement. Nice work!
>
> Best,
> tison.
>
>
> vino yang  于2019年8月22日周四 上午8:14写道：
>
> > +1 for "@flinkbot run travis", it is very convenient.
> >
> > Chesnay Schepler  于2019年8月21日周三 下午9:12写道：
> >
> > > Hi everyone,
> > >
> > > this is an update on recent changes to the CI bot.
> > >
> > >
> > > The bot now cancels builds if a new commit was added to a PR, and
> > > cancels all builds if the PR was closed.
> > > (This was implemented a while ago; I'm just mentioning it again for
> > > discoverability)
> > >
> > >
> > > Additionally, starting today you can now re-trigger a Travis run by
> > > writing a comment "@flinkbot run travis"; this means you no longer have
> > > to commit an empty commit or do other shenanigans to get another build
> > > running.
> > > Note that this will /not/ work if the PR was re-opened, until at least
> 1
> > > new build was triggered by a push.
> > >
> >
>

Re: [DISCUSS] Release flink-shaded 8.0

[Stephan Ewen]

+1 to go ahead

at some point we may want to bump the Hadoop versions for which we build
the shaded jars, but that would be a another dedicated effort

On Wed, Aug 21, 2019 at 1:41 PM Chesnay Schepler  wrote:

> Nico has opened a PR for bumping netty; we plan to have this merged by
> tomorrow.
>
> Unless anyone has concerns I will kick off the release on Friday.
>
> On 19/08/2019 12:11, Nico Kruber wrote:
> > I quickly went through all the changelogs for Netty 4.1.32 (which we
> > currently use) to the latest Netty 4.1.39.Final. Below, you will find a
> > list of bug fixes and performance improvements that may affect us. Nice
> > changes we could benefit from, also for the Java > 8 efforts. The most
> > important ones fixing leaks etc are #8921, #9167, #9274, #9394, and the
> > various CompositeByteBuf fixes. The rest are mostly performance
> > improvements.
> >
> > Since we are still early in the dev cycle for Flink 1.10, it would maybe
> > nice to update and verify that the new version works correctly. I'll
> > create a ticket and PR.
> >
> >
> > FYI (1): My own patches to bring dynamically-linked openSSL to more
> > distributions, namely SUSE and Arch, have not made it into a release yet.
> >
> > FYI (2): We are currently using the latest version of netty-tcnative,
> > i.e. 2.0.25.
> >
> >
> > Nico
> >
> > --
> > Netty 4.1.33.Final
> > - Fix ClassCastException and native crash when using kqueue transport
> > (#8665)
> > - Provide a way to cache the internal nioBuffer of the PooledByteBuffer
> > to reduce GC (#8603)
> >
> > Netty 4.1.34.Final
> > - Do not use GetPrimitiveArrayCritical(...) due multiple not-fixed bugs
> > related to GCLocker (#8921)
> > - Correctly monkey-patch id also in whe os / arch is used within library
> > name (#8913)
> > - Further reduce ensureAccessible() overhead (#8895)
> > - Support using an Executor to offload blocking / long-running tasks
> > when processing TLS / SSL via the SslHandler (#8847)
> > - Minimize memory footprint for AbstractChannelHandlerContext for
> > handlers that execute in the EventExecutor (#8786)
> > - Fix three bugs in CompositeByteBuf (#8773)
> >
> > Netty 4.1.35.Final
> > - Fix possible ByteBuf leak when CompositeByteBuf is resized (#8946)
> > - Correctly produce ssl alert when certificate validation fails on the
> > client-side when using native SSL implementation (#8949)
> >
> > Netty 4.1.37.Final
> > - Don't filter out TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (#9274)
> > - Try to mark child channel writable again once the parent channel
> > becomes writable (#9254)
> > - Properly debounce wakeups (#9191)
> > - Don't read from timerfd and eventfd on each EventLoop tick (#9192)
> > - Correctly detect that KeyManagerFactory is not supported when using
> > OpenSSL 1.1.0+ (#9170)
> > - Fix possible unsafe sharing of internal NIO buffer in CompositeByteBuf
> > (#9169)
> > - KQueueEventLoop won't unregister active channels reusing a file
> > descriptor (#9149)
> > - Prefer direct io buffers if direct buffers pooled (#9167)
> >
> > Netty 4.1.38.Final
> > - Prevent ByteToMessageDecoder from overreading when !isAutoRead (#9252)
> > - Correctly take length of ByteBufInputStream into account for
> > readLine() / readByte() (#9310)
> > - availableSharedCapacity will be slowly exhausted (#9394)
> > --
> >
> > On 18/08/2019 16:47, Stephan Ewen wrote:
> >> Are we fine with the current Netty version, or would be want to bump it?
> >>
> >> On Fri, Aug 16, 2019 at 10:30 AM Chesnay Schepler  >> <mailto:ches...@apache.org>> wrote:
> >>
> >>  Hello,
> >>
> >>  I would like to kick off the next flink-shaded release next week.
> There
> >>  are 2 ongoing efforts that are blocked on this release:
> >>
> >>* [FLINK-13467] Java 11 support requires a bump to ASM to
> correctly
> >>  handle Java 11 bytecode
> >>* [FLINK-11767] Reworking the
> typeSerializerSnapshotMigrationTestBase
> >>  requires asm-commons to be added to flink-shaded-asm
> >>
> >>  Are there any other changes on anyone's radar that we will have to
> make
> >>  for 1.10? (will bumping calcite require anything, for example)
> >>
> >>
>
>

[DISCUSS] Use Java's Duration instead of Flink's Time

[Stephan Ewen]

Hi all!

Many parts of the code use Flink's "Time" class. The Time really is a "time
interval" or a "Duration".

Since Java 8, there is a Java class "Duration" that is nice and flexible to
use.
I would suggest we start using Java Duration instead and drop Time as much
as possible in the runtime from now on.

Maybe even drop that class from the API in Flink 2.0.

Best,
Stephan

[DISCUSS] Add ARM CI build to Flink (information-only)

[Stephan Ewen]

Hi all!

As part of the Flink on ARM effort, there is a pull request that triggers a
build on OpenLabs CI for each push and runs tests on ARM machines.

Currently that build is roughly equivalent to what the "core" and "tests"
profiles do on Travis.
The result will be posted to the PR comments, similar to the Flink Bot's
Travis build result.
The build currently passes :-) so Flink seems to be okay on ARM.

My suggestion would be to try and add this and gather some experience with
it.
The Travis build results should be our "ground truth" and the ARM CI
(openlabs CI) would be "informational only" at the beginning, but helping
us understand when we break ARM support.

You can see this in the PR that adds the openlabs CI config:
https://github.com/apache/flink/pull/9416

Any objections?

Best,
Stephan

Re: [DISCUSS] Add ARM CI build to Flink (information-only)

[Stephan Ewen]

t;> > Does it make sense?
>> >
>> > [1]: http://114.115.168.52:8081/#/overview
>> > [1]: https://issues.apache.org/jira/browse/FLINK-13449
>> >https://issues.apache.org/jira/browse/FLINK-13450
>> > [2]: https://issues.apache.org/jira/browse/FLINK-13598
>> > [3]: https://github.com/theopenlab/openlab/issues/new/choose
>> >
>> >
>> >
>> >
>> > Chesnay Schepler  于2019年8月24日周六 上午12:10写道：
>> >
>> >> I'm wondering what we are supposed to do if the build fails?
>> >> We aren't providing and guides on setting up an arm dev environment; so
>> >> reproducing it locally isn't possible.
>> >>
>> >> On 23/08/2019 17:55, Stephan Ewen wrote:
>> >>> Hi all!
>> >>>
>> >>> As part of the Flink on ARM effort, there is a pull request that
>> >> triggers a
>> >>> build on OpenLabs CI for each push and runs tests on ARM machines.
>> >>>
>> >>> Currently that build is roughly equivalent to what the "core" and
>> "tests"
>> >>> profiles do on Travis.
>> >>> The result will be posted to the PR comments, similar to the Flink
>> Bot's
>> >>> Travis build result.
>> >>> The build currently passes :-) so Flink seems to be okay on ARM.
>> >>>
>> >>> My suggestion would be to try and add this and gather some experience
>> >> with
>> >>> it.
>> >>> The Travis build results should be our "ground truth" and the ARM CI
>> >>> (openlabs CI) would be "informational only" at the beginning, but
>> helping
>> >>> us understand when we break ARM support.
>> >>>
>> >>> You can see this in the PR that adds the openlabs CI config:
>> >>> https://github.com/apache/flink/pull/9416
>> >>>
>> >>> Any objections?
>> >>>
>> >>> Best,
>> >>> Stephan
>> >>>
>> >>
>>
>>

Re: [DISCUSS] Use Java's Duration instead of Flink's Time

[Stephan Ewen]

Seems everyone is in favor in principle.

  - For public APIs, I would keep Time for now (to not break the API).
Maybe add a Duration variant and deprecate the Time variant, but not remove
it before Flink 1.0
  - For all runtime Java code, switch to Java's Duration now
  - For all Scala code let's see how much we can switch to Java Durations
without blowing up stuff. After all, like Tison said, we want to get the
runtime Scala free in the future.

On Mon, Aug 26, 2019 at 3:45 AM Jark Wu  wrote:

> +1 to use Java's Duration instead of Flink's Time.
>
> Regarding to the Duration parsing, we have mentioned this in FLIP-54[1] to
> use `org.apache.flink.util.TimeUtils` for the parsing.
>
> Best,
> Jark
>
> [1]:
>
> https://docs.google.com/document/d/1IQ7nwXqmhCy900t2vQLEL3N2HIdMg-JO8vTzo1BtyKU/edit#heading=h.egdwkc93dn1k
>
> On Sat, 24 Aug 2019 at 18:24, Zhu Zhu  wrote:
>
> > +1 since Java Duration is more common and powerful than Flink Time.
> >
> > For whether to drop scala Duration for parsing duration OptionConfig, I
> > think it's another question and should be discussed in another thread.
> >
> > Thanks,
> > Zhu Zhu
> >
> > Becket Qin  于2019年8月24日周六 下午4:16写道：
> >
> > > +1, makes sense. BTW, we probably need a FLIP as this is a public API
> > > change.
> > >
> > > On Sat, Aug 24, 2019 at 8:11 AM SHI Xiaogang 
> > > wrote:
> > >
> > > > +1 to replace Flink's time with Java's Duration.
> > > >
> > > > Besides, i also suggest to use Java's Instant for "point-in-time".
> > > > It can take care of time units when we calculate Duration between
> > > different
> > > > instants.
> > > >
> > > > Regards,
> > > > Xiaogang
> > > >
> > > > Zili Chen  于2019年8月24日周六 上午10:45写道：
> > > >
> > > > > Hi vino,
> > > > >
> > > > > I agree that it introduces extra complexity to replace
> > Duration(Scala)
> > > > > with Duration(Java) *in Scala code*. We could separate the usage
> for
> > > each
> > > > > language and use a bridge when necessary.
> > > > >
> > > > > As a matter of fact, Scala concurrent APIs(including Duration) are
> > used
> > > > > more than necessary at least in flink-runtime. Also we even try to
> > make
> > > > > flink-runtime scala free.
> > > > >
> > > > > Best,
> > > > > tison.
> > > > >
> > > > >
> > > > > vino yang  于2019年8月24日周六 上午10:05写道：
> > > > >
> > > > > > +1 to replace the Time class provided by Flink with Java's
> > Duration:
> > > > > >
> > > > > >
> > > > > >- Java's Duration has better representation than the Flink's
> > Time
> > > > > class;
> > > > > >- As a built-in Java class, Duration class has a clear
> advantage
> > > > over
> > > > > >Java's Time class when interacting with other Java APIs and
> > > > > third-party
> > > > > >libraries;
> > > > > >
> > > > > >
> > > > > > But I have reservations about replacing the Duration and
> > FineDuration
> > > > > > classes in scala with the Duration class in Java. Java and Scala
> > have
> > > > > > different types of systems. Currently, Duration (scala) and
> > > > FineDuration
> > > > > > (scala) work well.  In addition, this work brings additional
> > > complexity
> > > > > and
> > > > > > cost compared to the gains obtained.
> > > > > >
> > > > > > Best,
> > > > > > Vino
> > > > > >
> > > > > > Zili Chen  于2019年8月23日周五 下午11:14写道：
> > > > > >
> > > > > > > Hi Stephan,
> > > > > > >
> > > > > > > I like the idea unify usage of time/duration api. We actually
> > > > > > > use at least five different classes for this purposes(see
> below).
> > > > > > >
> > > > > > > One thing I'd like to pick up is that duration configuration
> > > > > > > in Flink is almost in pattern as "60 s" that fits in the
> pattern
> > > > > > > parsed by scala.concurrent.duration.Dura

Re: [DISCUSS] FLIP-49: Unified Memory Configuration for TaskExecutors

[Stephan Ewen]

One note on the Environment Variables and Configuration discussion.

My understanding is that passed ENV variables are added to the
configuration in the "GlobalConfiguration.loadConfig()" method (or similar).
For all the code inside Flink, it looks like the data was in the config to
start with, just that the scripts that compute the variables can pass the
values to the process without actually needing to write a file.

For example the "GlobalConfiguration.loadConfig()" method would take any
ENV variable prefixed with "flink" and add it as a config key.
"flink_taskmanager_memory_size=2g" would become "taskmanager.memory.size:
2g".


On Tue, Aug 27, 2019 at 4:05 PM Xintong Song  wrote:

> Thanks for the comments, Till.
>
> I've also seen your comments on the wiki page, but let's keep the
> discussion here.
>
> - Regarding 'TaskExecutorSpecifics', how do you think about naming it
> 'TaskExecutorResourceSpecifics'.
> - Regarding passing memory configurations into task executors, I'm in favor
> of do it via environment variables rather than configurations, with the
> following two reasons.
>   - It is easier to keep the memory options once calculate not to be
> changed with environment variables rather than configurations.
>   - I'm not sure whether we should write the configuration in startup
> scripts. Writing changes into the configuration files when running the
> startup scripts does not sounds right to me. Or we could make a copy of
> configuration files per flink cluster, and make the task executor to load
> from the copy, and clean up the copy after the cluster is shutdown, which
> is complicated. (I think this is also what Stephan means in his comment on
> the wiki page?)
> - Regarding reserving memory, I think this change should be included in
> this FLIP. I think a big part of motivations of this FLIP is to unify
> memory configuration for streaming / batch and make it easy for configuring
> rocksdb memory. If we don't support memory reservation, then streaming jobs
> cannot use managed memory (neither on-heap or off-heap), which makes this
> FLIP incomplete.
> - Regarding network memory, I think you are right. I think we probably
> don't need to change network stack from using direct memory to using unsafe
> native memory. Network memory size is deterministic, cannot be reserved as
> managed memory does, and cannot be overused. I think it also works if we
> simply keep using direct memory for network and include it in jvm max
> direct memory size.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Tue, Aug 27, 2019 at 8:12 PM Till Rohrmann 
> wrote:
>
> > Hi Xintong,
> >
> > thanks for addressing the comments and adding a more detailed
> > implementation plan. I have a couple of comments concerning the
> > implementation plan:
> >
> > - The name `TaskExecutorSpecifics` is not really descriptive. Choosing a
> > different name could help here.
> > - I'm not sure whether I would pass the memory configuration to the
> > TaskExecutor via environment variables. I think it would be better to
> write
> > it into the configuration one uses to start the TM process.
> > - If possible, I would exclude the memory reservation from this FLIP and
> > add this as part of a dedicated FLIP.
> > - If possible, then I would exclude changes to the network stack from
> this
> > FLIP. Maybe we can simply say that the direct memory needed by the
> network
> > stack is the framework direct memory requirement. Changing how the memory
> > is allocated can happen in a second step. This would keep the scope of
> this
> > FLIP smaller.
> >
> > Cheers,
> > Till
> >
> > On Thu, Aug 22, 2019 at 2:51 PM Xintong Song 
> > wrote:
> >
> > > Hi everyone,
> > >
> > > I just updated the FLIP document on wiki [1], with the following
> changes.
> > >
> > >- Removed open question regarding MemorySegment allocation. As
> > >discussed, we exclude this topic from the scope of this FLIP.
> > >- Updated content about JVM direct memory parameter according to
> > recent
> > >discussions, and moved the other options to "Rejected Alternatives"
> > for
> > > the
> > >moment.
> > >- Added implementation steps.
> > >
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors
> > >
> > >

Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

[Stephan Ewen]

ore explicit by offering an explicit
> > >> `intType()` method etc. The current design of validators centered
> > >> around Java classes makes it possible to have typical domain
> > >> validators baked by generics as you suggested. If we introduce types
> > >> such as "quantity with measure and unit" we still need to get a class
> > >> out of this option at the end, so why changing a proven concept?
> > >>
> > >> 3. List Options: The `isList` prevents having arbitrary nesting. As
> > >> Dawid mentioned, we kept human readability in mind. For every atomic
> > >> option like "key=12" can be represented by a list "keys=12;13". But
> > >> we don't want to go further; esp. no nesting. A dedicated list option
> > >> would start making this more complicated such as
> > >> "ListOption(ObjectOption(ListOption(IntOption, ...),
> > >> StringOption(...)))", do we want that?
> > >>
> > >> 4. Correlation: The correlation part is one of the suggestions that I
> > >> like least in the document. We can also discuss removing it entirely,
> > >> but I think it solves the use case of relating options with each
> > >> other in a flexible way right next to the actual option. Instead of
> > >> being hidden in some component initialization, we should put it close
> > >> to the option to also perform validation eagerly instead of failing
> > >> at runtime when the option is accessed the first time.
> > >>
> > >> Regards,
> > >> Timo
> > >>
> > >>
> > >> Am 18.08.19 um 23:32 schrieb Stephan Ewen:
> > >>> A "List Type" sounds like a good direction to me.
> > >>>
> > >>> The comment on the type system was a bit brief, I agree. The idea is
> > >>> to see
> > >>> if something like that can ease validation. Especially the
> correlation
> > >>> system seems quite complex (proxies to work around order of
> > >>> initialization).
> > >>>
> > >>> For example, let's assume we don't think primarily about "java
> > >>> types" but
> > >>> would define types as one of the following (just examples, haven't
> > >>> thought
> > >>> all the details through):
> > >>>
> > >>>(a) category type: implies string, and a fix set of possible
> values.
> > >>> Those would be passes and naturally make it into the docs and
> > >>> validation.
> > >>> Maps to a String or Enum in Java.
> > >>>
> > >>>(b) numeric integer type: implies long (or optionally integer, if
> > >>> we want
> > >>> to automatically check overflow / underflow). would take typical
> domain
> > >>> validators, like non-negative, etc.
> > >>>
> > >>>(c) numeric real type: same as above (double or float)
> > >>>
> > >>>(d) numeric interval type: either defined as an interval, or
> > >>> references
> > >>> other parameter by key. validation by valid interval.
> > >>>
> > >>>(e) quantity: a measure and a unit. separately parsable. The
> > >>> measure's
> > >>> type could be any of the numeric types above, with same validation
> > >>> rules.
> > >>>
> > >>> With a system like the above, would we still correlation validators?
> > >>> Are
> > >>> there still cases that we need to catch early (config loading) or
> > >>> are the
> > >>> remaining cases sufficiently rare and runtime or setup specific,
> > >>> that it is
> > >>> fine to handle them in component initialization?
> > >>>
> > >>>
> > >>> On Sun, Aug 18, 2019 at 6:36 PM Dawid Wysakowicz
> > >>> 
> > >>> wrote:
> > >>>
> > >>>> Hi Stephan,
> > >>>>
> > >>>> Thank you for your opinion.
> > >>>>
> > >>>> Actually list/composite types are the topics we spent the most of
> the
> > >>>> time. I understand that from a perspective of a full blown type
> > >>>> system,
> > >>>> a field like isList may look weird. Please let me elaborate a bit
> more
> &

Re: [DISCUSS] FLIP-49: Unified Memory Configuration for TaskExecutors

[Stephan Ewen]

When computing the values in the JVM process after it started, how would
you deal with values like Max Direct Memory, Metaspace size. native memory
reservation (reduce heap size), etc? All the values that are parameters to
the JVM process and that need to be supplied at process startup?

On Wed, Aug 28, 2019 at 4:46 PM Till Rohrmann  wrote:

> Thanks for the clarification. I have some more comments:
>
> - I would actually split the logic to compute the process memory
> requirements and storing the values into two things. E.g. one could name
> the former TaskExecutorProcessUtility and  the latter
> TaskExecutorProcessMemory. But we can discuss this on the PR since it's
> just a naming detail.
>
> - Generally, I'm not opposed to making configuration values overridable by
> ENV variables. I think this is a very good idea and makes the
> configurability of Flink processes easier. However, I think that adding
> this functionality should not be part of this FLIP because it would simply
> widen the scope unnecessarily.
>
> The reasons why I believe it is unnecessary are the following: For Yarn we
> already create write a flink-conf.yaml which could be populated with the
> memory settings. For the other processes it should not make a difference
> whether the loaded Configuration is populated with the memory settings from
> ENV variables or by using TaskExecutorProcessUtility to compute the missing
> values from the loaded configuration. If the latter would not be possible
> (wrong or missing configuration values), then we should not have been able
> to actually start the process in the first place.
>
> - Concerning the memory reservation: I agree with you that we need the
> memory reservation functionality to make streaming jobs work with "managed"
> memory. However, w/o this functionality the whole Flip would already bring
> a good amount of improvements to our users when running batch jobs.
> Moreover, by keeping the scope smaller we can complete the FLIP faster.
> Hence, I would propose to address the memory reservation functionality as a
> follow up FLIP (which Yu is working on if I'm not mistaken).
>
> Cheers,
> Till
>
> On Wed, Aug 28, 2019 at 11:43 AM Yang Wang  wrote:
>
> > Just add my 2 cents.
> >
> > Using environment variables to override the configuration for different
> > taskmanagers is better.
> > We do not need to generate dedicated flink-conf.yaml for all
> taskmanagers.
> > A common flink-conf.yam and different environment variables are enough.
> > By reducing the distributed cached files, it could make launching a
> > taskmanager faster.
> >
> > Stephan gives a good suggestion that we could move the logic into
> > "GlobalConfiguration.loadConfig()" method.
> > Maybe the client could also benefit from this. Different users do not
> have
> > to export FLINK_CONF_DIR to update few config options.
> >
> >
> > Best,
> > Yang
> >
> > Stephan Ewen  于2019年8月28日周三 上午1:21写道：
> >
> > > One note on the Environment Variables and Configuration discussion.
> > >
> > > My understanding is that passed ENV variables are added to the
> > > configuration in the "GlobalConfiguration.loadConfig()" method (or
> > > similar).
> > > For all the code inside Flink, it looks like the data was in the config
> > to
> > > start with, just that the scripts that compute the variables can pass
> the
> > > values to the process without actually needing to write a file.
> > >
> > > For example the "GlobalConfiguration.loadConfig()" method would take
> any
> > > ENV variable prefixed with "flink" and add it as a config key.
> > > "flink_taskmanager_memory_size=2g" would become
> "taskmanager.memory.size:
> > > 2g".
> > >
> > >
> > > On Tue, Aug 27, 2019 at 4:05 PM Xintong Song 
> > > wrote:
> > >
> > > > Thanks for the comments, Till.
> > > >
> > > > I've also seen your comments on the wiki page, but let's keep the
> > > > discussion here.
> > > >
> > > > - Regarding 'TaskExecutorSpecifics', how do you think about naming it
> > > > 'TaskExecutorResourceSpecifics'.
> > > > - Regarding passing memory configurations into task executors, I'm in
> > > favor
> > > > of do it via environment variables rather than configurations, with
> the
> > > > following two reasons.
> > > >   - It is easier to keep the memory options once calculate not to be
> > > > change

Re: [DISCUSS] FLIP-49: Unified Memory Configuration for TaskExecutors

[Stephan Ewen]

I see. Under the assumption of strict determinism that should work.

The original proposal had this point "don't compute inside the TM, compute
outside and supply a full config", because that sounded more intuitive.

On Thu, Aug 29, 2019 at 12:15 PM Till Rohrmann  wrote:

> My understanding was that before starting the Flink process we call a
> utility which calculates these values. I assume that this utility will do
> the calculation based on a set of configured values (process memory, flink
> memory, network memory etc.). Assuming that these values don't differ from
> the values with which the JVM is started, it should be possible to
> recompute them in the Flink process in order to set the values.
>
>
>
> On Thu, Aug 29, 2019 at 11:29 AM Stephan Ewen  wrote:
>
> > When computing the values in the JVM process after it started, how would
> > you deal with values like Max Direct Memory, Metaspace size. native
> memory
> > reservation (reduce heap size), etc? All the values that are parameters
> to
> > the JVM process and that need to be supplied at process startup?
> >
> > On Wed, Aug 28, 2019 at 4:46 PM Till Rohrmann 
> > wrote:
> >
> > > Thanks for the clarification. I have some more comments:
> > >
> > > - I would actually split the logic to compute the process memory
> > > requirements and storing the values into two things. E.g. one could
> name
> > > the former TaskExecutorProcessUtility and  the latter
> > > TaskExecutorProcessMemory. But we can discuss this on the PR since it's
> > > just a naming detail.
> > >
> > > - Generally, I'm not opposed to making configuration values overridable
> > by
> > > ENV variables. I think this is a very good idea and makes the
> > > configurability of Flink processes easier. However, I think that adding
> > > this functionality should not be part of this FLIP because it would
> > simply
> > > widen the scope unnecessarily.
> > >
> > > The reasons why I believe it is unnecessary are the following: For Yarn
> > we
> > > already create write a flink-conf.yaml which could be populated with
> the
> > > memory settings. For the other processes it should not make a
> difference
> > > whether the loaded Configuration is populated with the memory settings
> > from
> > > ENV variables or by using TaskExecutorProcessUtility to compute the
> > missing
> > > values from the loaded configuration. If the latter would not be
> possible
> > > (wrong or missing configuration values), then we should not have been
> > able
> > > to actually start the process in the first place.
> > >
> > > - Concerning the memory reservation: I agree with you that we need the
> > > memory reservation functionality to make streaming jobs work with
> > "managed"
> > > memory. However, w/o this functionality the whole Flip would already
> > bring
> > > a good amount of improvements to our users when running batch jobs.
> > > Moreover, by keeping the scope smaller we can complete the FLIP faster.
> > > Hence, I would propose to address the memory reservation functionality
> > as a
> > > follow up FLIP (which Yu is working on if I'm not mistaken).
> > >
> > > Cheers,
> > > Till
> > >
> > > On Wed, Aug 28, 2019 at 11:43 AM Yang Wang 
> > wrote:
> > >
> > > > Just add my 2 cents.
> > > >
> > > > Using environment variables to override the configuration for
> different
> > > > taskmanagers is better.
> > > > We do not need to generate dedicated flink-conf.yaml for all
> > > taskmanagers.
> > > > A common flink-conf.yam and different environment variables are
> enough.
> > > > By reducing the distributed cached files, it could make launching a
> > > > taskmanager faster.
> > > >
> > > > Stephan gives a good suggestion that we could move the logic into
> > > > "GlobalConfiguration.loadConfig()" method.
> > > > Maybe the client could also benefit from this. Different users do not
> > > have
> > > > to export FLINK_CONF_DIR to update few config options.
> > > >
> > > >
> > > > Best,
> > > > Yang
> > > >
> > > > Stephan Ewen  于2019年8月28日周三 上午1:21写道：
> > > >
> > > > > One note on the Environment Variables and Configuration discussion.
> > > > >
> > > > > My understanding is that 

Re: [DISCUSS] Simplify Flink's cluster level RestartStrategy configuration

[Stephan Ewen]

+1 in general

What is the default in batch, though? No restarts? I always found that
somewhat uncommon.
Should we also change that part, if we are changing the default anyways?


On Fri, Aug 30, 2019 at 2:35 PM Till Rohrmann  wrote:

> Hi everyone,
>
> I wanted to discuss how to simplify Flink's cluster level RestartStrategy
> configuration [1]. Currently, Flink's behaviour with respect to configuring
> the {{RestartStrategies}} is quite complicated and convoluted. The reason
> for this is that we evolved the way it has been configured and wanted to
> keep it backwards compatible. Due to this, we have currently the following
> behaviour:
>
> * If the config option `restart-strategy` is configured, then Flink uses
> this `RestartStrategy` (so far so simple)
> * If the config option `restart-strategy` is not configured, then
> ** If `restart-strategy.fixed-delay.attempts` or
> `restart-strategy.fixed-delay.delay` are defined, then instantiate
> `FixedDelayRestartStrategy(restart-strategy.fixed-delay.attempts,
> restart-strategy.fixed-delay.delay)`
> ** If `restart-strategy.fixed-delay.attempts` and
> `restart-strategy.fixed-delay.delay` are not defined, then
> *** If checkpointing is disabled, then choose `NoRestartStrategy`
> *** If checkpointing is enabled, then choose
> `FixedDelayRestartStrategy(Integer.MAX_VALUE, "0 s")`
>
> I would like to simplify the configuration by removing the "If
> `restart-strategy.fixed-delay.attempts` or
> `restart-strategy.fixed-delay.delay`, then" condition. That way, the logic
> would be the following:
>
> * If the config option `restart-strategy` is configured, then Flink uses
> this `RestartStrategy`
> * If the config option `restart-strategy` is not configured, then
> ** If checkpointing is disabled, then choose `NoRestartStrategy`
> ** If checkpointing is enabled, then choose
> `FixedDelayRestartStrategy(Integer.MAX_VALUE, "0 s")`
>
> That way we retain the user friendliness that jobs restart if the user
> enabled checkpointing and we make it clear that any `
> restart-strategy.fixed-delay.xyz` setting will only be respected if
> `restart-strategy` has been set to `fixed-delay`.
>
> This simplification would, however, change Flink's behaviour and might
> break existing setups. Since we introduced `RestartStrategies` with Flink
> 1.0.0 and deprecated the prior configuration mechanism which enables
> restarting if either the `attempts` or the `delay` has been set, I think
> that the number of broken jobs should be minimal if not non-existent.
>
> I'm sure that one can simplify the way RestartStrategies are
> programmatically configured as well but for the sake of simplicity/scoping
> I'd like to not touch it right away.
>
> What do you think about this behaviour change?
>
> [1] https://issues.apache.org/jira/browse/FLINK-13921
>
> Cheers,
> Till
>

Re: instable checkpointing after migration to flink 1.8

[Stephan Ewen]

Hi all!

A thought would be that this has something to do with timers. Does the task
with that behavior use timers (windows, or process function)?

If that is the case, some theories to check:
  - Could it be a timer firing storm coinciding with a checkpoint?
Currently, that storm synchronously fires, checkpoints cannot preempt that,
which should change in 1.10 with the new mailbox model.
  - Could the timer-async checkpointing changes have something to do with
that? Does some of the usually small "preparation work" (happening
synchronously) lead to an unwanted effect?
  - Are you using TTL for state in that operator?
  - There were some changes made to support timers in RocksDB recently.
Could that have something to do with it?

Best,
Stephan


On Fri, Aug 30, 2019 at 2:45 PM Congxian Qiu  wrote:

> CC flink dev mail list
> Update for those who may be interested in this issue, we'are still
> diagnosing this problem currently.
>
> Best,
> Congxian
>
>
> Congxian Qiu  于2019年8月29日周四 下午8:58写道：
>
> > Hi Bekir
> >
> > Currently, from what we have diagnosed, there is some task complete its
> > checkpoint too late (maybe 15 mins), but we checked the kafka broker log
> > and did not find any interesting things there. could we run another job,
> > that did not commit offset to kafka, this wants to check if it is the
> > "commit offset to kafka" step consumes too much time.
> >
> > Best,
> > Congxian
> >
> >
> > Bekir Oguz  于2019年8月28日周三 下午4:19写道：
> >
> >> Hi Congxian,
> >> sorry for the late reply, but no progress on this issue yet. I checked
> >> also the kafka broker logs, found nothing interesting there.
> >> And we still have 15 min duration checkpoints quite often. Any more
> ideas
> >> on where to look at?
> >>
> >> Regards,
> >> Bekir
> >>
> >> On Fri, 23 Aug 2019 at 12:12, Congxian Qiu 
> >> wrote:
> >>
> >>> Hi Bekir
> >>>
> >>> Do you come back to work now, does there any more findings of this
> >>> problem?
> >>>
> >>> Best,
> >>> Congxian
> >>>
> >>>
> >>> Bekir Oguz  于2019年8月13日周二 下午4:39写道：
> >>>
>  Hi Congxian,
>  Thanks for following up this issue. It is still unresolved and I am on
>  vacation at the moment.  Hopefully my collegues Niels and Vlad can
> spare
>  some time to look into this.
> 
>  @Niels, @Vlad: do you guys also think that this issue might be Kafka
>  related? We could also check kafka broker logs at the time of long
>  checkpointing.
> 
>  Thanks,
>  Bekir
> 
>  Verstuurd vanaf mijn iPhone
> 
>  Op 12 aug. 2019 om 15:18 heeft Congxian Qiu 
>  het volgende geschreven:
> 
>  Hi Bekir
> 
>  Is there any progress about this problem?
> 
>  Best,
>  Congxian
> 
> 
>  Congxian Qiu  于2019年8月8日周四 下午11:17写道：
> 
> > hi Bekir
> > Thanks for the information.
> >
> > - Source's checkpoint was triggered by RPC calls, so it has the
> > "Trigger checkpoint xxx" log,
> > - other task's checkpoint was triggered after received all the
> barrier
> > of upstreams, it didn't log the "Trigger checkpoint XXX" :(
> >
> > Your diagnose makes sense to me, we need to check the Kafka log.
> > I also find out that we always have a log like
> > "org.apache.kafka.clients.consumer.internals.AbstractCoordinator
> Marking
> > the coordinator 172.19.200.73:9092 (id: 2147483646 rack: null) dead
> > for group userprofileaggregator
> > 2019-08-06 13:58:49,872 DEBUG
> > org.apache.flink.streaming.runtime.tasks.StreamTask   -
> Notifica",
> >
> > I checked the doc of kafka[1], only find that the default of `
> > transaction.max.timeout.ms` is 15 min
> >
> > Please let me know there you have any finds. thanks
> >
> > PS: maybe you can also checkpoint the log for task
> > `d0aa98767c852c97ae8faf70a54241e3`, JM received its ack message late
> also.
> >
> > [1] https://kafka.apache.org/documentation/
> > Best,
> > Congxian
> >
> >
> > Bekir Oguz  于2019年8月7日周三 下午6:48写道：
> >
> >> Hi Congxian,
> >> Thanks for checking the logs. What I see from the logs is:
> >>
> >> - For the tasks like "Source:
> >> profileservice-snowplow-clean-events_kafka_source -> Filter” {17,
> 27, 31,
> >> 33, 34} / 70 : We have the ’Triggering checkpoint’ and also ‘Confirm
> >> checkpoint’ log lines, with 15 mins delay in between.
> >> - For the tasks like “KeyedProcess -> (Sink:
> >> profileservice-userprofiles_kafka_sink, Sink:
> >> profileservice-userprofiles_kafka_deletion_marker, Sink:
> >> profileservice-profiledeletion_kafka_sink” {1,2,3,4,5}/70 : We DO
> NOT have
> >> the “Triggering checkpoint” log, but only the ‘Confirm checkpoint’
> lines.
> >>
> >> And as a final point, we ALWAYS have Kafka AbstractCoordinator logs
> >> about lost connection to Kafka at the same time we have the
> checkpoints
> >> confirmed. This 15 minutes delay might be because 

Re: Potential block size issue with S3 binary files

[Stephan Ewen]

Sounds reasonable.

I am adding Arvid to the thread - IIRC he authored that tool in his
Stratosphere days. And my a stroke of luck, he is now working on Flink
again.

@Arvid - what are your thoughts on Ken's suggestions?

On Fri, Aug 30, 2019 at 8:56 PM Ken Krugler 
wrote:

> Hi Stephan (switching to dev list),
>
> On Aug 29, 2019, at 2:52 AM, Stephan Ewen  wrote:
>
> That is a good point.
>
> Which way would you suggest to go? Not relying on the FS block size at
> all, but using a fix (configurable) block size?
>
>
> There’s value to not requiring a fixed block size, as then a file that’s
> moved between different file systems can be read using whatever block size
> is optimal for that system.
>
> Hadoop handles this in sequence files by storing a unique “sync marker”
> value in the file header (essentially a 16 byte UUID), injecting one of
> these every 2K bytes or so (in between records), and then code can scan for
> this to find record boundaries without relying on a block size. The idea is
> that 2^128 is a Big Number, so the odds of finding a false-positive sync
> marker in data is low enough to be ignorable.
>
> But that’s a bigger change. Simpler would be to put a header in each part
> file being written, with some signature bytes to aid in detecting
> old-format files.
>
> Or maybe deprecate SerializedOutputFormat/SerializedInputFormat, and
> provide some wrapper glue to make it easier to write/read Hadoop
> SequenceFiles that have a null key value, and store the POJO as the data
> value. Then you could also leverage Hadoop support for compression at
> either record or block level.
>
> — Ken
>
>
> On Thu, Aug 29, 2019 at 4:49 AM Ken Krugler 
> wrote:
>
>> Hi all,
>>
>> Wondering if anyone else has run into this.
>>
>> We write files to S3 using the SerializedOutputFormat.
>> When we read them back, sometimes we get deserialization errors where the
>> data seems to be corrupt.
>>
>> After a lot of logging, the weathervane of blame pointed towards the
>> block size somehow not being the same between the write (where it’s 64MB)
>> and the read (unknown).
>>
>> When I added a call to SerializedInputFormat.setBlockSize(64MB), the
>> problems went away.
>>
>> It looks like both input and output formats use fs.getDefaultBlockSize()
>> to set this value by default, so maybe the root issue is S3 somehow
>> reporting different values.
>>
>> But it does feel a bit odd that we’re relying on this default setting,
>> versus it being recorded in the file during the write phase.
>>
>> And it’s awkward to try to set the block size on the write, as you need
>> to set it in the environment conf, which means it applies to all output
>> files in the job.
>>
>> — Ken
>>
>
> --
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com
> Custom big data solutions & training
> Flink, Solr, Hadoop, Cascading & Cassandra
>
>

Re: [VOTE] FLIP-49: Unified Memory Configuration for TaskExecutors

[Stephan Ewen]

+1 to the proposal in general

A few things seems to be a bit put of sync with the latest discussions
though.

The section about JVM Parameters states that the
-XX:MaxDirectMemorySize value is set to Task Off-heap Memory, Shuffle
Memory and JVM Overhead.
The way I understand the last discussion conclusion is that it is only the
sum of shuffle memory and user-defined direct memory.

I am someone neutral but unsure about is the separation between
"taskmanager.memory.framework.heap" and "taskmanager.memory.task.heap".
Could that be simply combined under "taskmanager.memory.javaheap"?

It might be good to also expose these values somehow in the web UI so that
users see immediately what amount of memory TMs assume to use for what.

I assume config key names and default values might be adjusted over time as
we get feedback.
  - I would keep the network memory under the name
"taskmanager.memory.network". Because network memory is actually used for
more than shuffling. Also, the old config key seems good, so why change it?

One thing to be aware of is that often, the Java Heap is understood as
"managed memory" as a whole, because it is managed by the GC not explicitly
by the user.
So we need to make sure that we don't confuse users by speaking of managed
heap and unmanaged heap. All heap is managed in Java. Some memory is
explicitly managed by Flink.

Best,
Stephan


On Mon, Sep 2, 2019 at 3:08 PM Xintong Song  wrote:

> Hi everyone,
>
> I'm here to re-start the voting process for FLIP-49 [1], with respect to
> consensus reached in this thread [2] regarding some new comments and
> concerns.
>
> This voting will be open for at least 72 hours. I'll try to close it Sep.
> 5, 14:00 UTC, unless there is an objection or not enough votes.
>
> Thank you~
>
> Xintong Song
>
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors
> [2]
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-49-Unified-Memory-Configuration-for-TaskExecutors-td31436.html
>
> On Tue, Aug 27, 2019 at 9:29 PM Xintong Song 
> wrote:
>
> > Alright, then let's keep the discussion in the DISCUSS mailing thread,
> and
> > see whether we need to restart the vote.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Tue, Aug 27, 2019 at 8:12 PM Till Rohrmann 
> > wrote:
> >
> >> I had a couple of comments concerning the implementation plan. I've
> posted
> >> them to the original discussion thread. Depending on the outcome of this
> >> discussion we might need to restart the vote.
> >>
> >> Cheers,
> >> Till
> >>
> >> On Tue, Aug 27, 2019 at 11:14 AM Xintong Song 
> >> wrote:
> >>
> >> > Hi all,
> >> >
> >> > I would like to start the voting process for FLIP-49 [1], which is
> >> > discussed and reached consensus in this thread [2].
> >> >
> >> > This voting will be open for at least 72 hours. I'll try to close it
> >> Aug.
> >> > 30 10:00 UTC, unless there is an objection or not enough votes.
> >> >
> >> > Thank you~
> >> >
> >> > Xintong Song
> >> >
> >> >
> >> > [1]
> >> >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors
> >> > [2]
> >> >
> >> >
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-49-Unified-Memory-Configuration-for-TaskExecutors-td31436.html
> >> >
> >>
> >
>

Re: [VOTE] FLIP-49: Unified Memory Configuration for TaskExecutors

[Stephan Ewen]

t it to set -XX:MaxDirectMemorySize
> > correctly or to introduce another configuration option.
> >
> > Given all these configuration options, I can see that users will get
> > confused quite easily. Therefore, I would like to emphasise that we need
> a
> > very good documentation about how to properly configure Flink processes
> and
> > which knobs to turn in which cases.
> >
> > Cheers,
> > Till
> >
> > On Tue, Sep 3, 2019 at 2:34 PM Andrey Zagrebin 
> > wrote:
> >
> > > Thanks for starting the vote Xintong
> > >
> > > Also +1 for the proposed FLIP-49.
> > >
> > > @Stephan regarding namings: network vs shuffle.
> > > My understanding so far was that the network memory is what we
> basically
> > > give to Shuffle implementations and default netty implementation uses
> it
> > in
> > > particular mostly for networking.
> > > Are the network pools used for something else outside of the shuffling
> > > scope?
> > >
> > > best,
> > > Andrey
> > >
> > > On Tue, Sep 3, 2019 at 1:01 PM Stephan Ewen  wrote:
> > >
> > > > +1 to the proposal in general
> > > >
> > > > A few things seems to be a bit put of sync with the latest
> discussions
> > > > though.
> > > >
> > > > The section about JVM Parameters states that the
> > > > -XX:MaxDirectMemorySize value is set to Task Off-heap Memory, Shuffle
> > > > Memory and JVM Overhead.
> > > > The way I understand the last discussion conclusion is that it is
> only
> > > the
> > > > sum of shuffle memory and user-defined direct memory.
> > > >
> > > > I am someone neutral but unsure about is the separation between
> > > > "taskmanager.memory.framework.heap" and
> "taskmanager.memory.task.heap".
> > > > Could that be simply combined under "taskmanager.memory.javaheap"?
> > > >
> > > > It might be good to also expose these values somehow in the web UI so
> > > that
> > > > users see immediately what amount of memory TMs assume to use for
> what.
> > > >
> > > > I assume config key names and default values might be adjusted over
> > time
> > > as
> > > > we get feedback.
> > > >   - I would keep the network memory under the name
> > > > "taskmanager.memory.network". Because network memory is actually used
> > for
> > > > more than shuffling. Also, the old config key seems good, so why
> change
> > > it?
> > > >
> > > > One thing to be aware of is that often, the Java Heap is understood
> as
> > > > "managed memory" as a whole, because it is managed by the GC not
> > > explicitly
> > > > by the user.
> > > > So we need to make sure that we don't confuse users by speaking of
> > > managed
> > > > heap and unmanaged heap. All heap is managed in Java. Some memory is
> > > > explicitly managed by Flink.
> > > >
> > > > Best,
> > > > Stephan
> > > >
> > > >
> > > > On Mon, Sep 2, 2019 at 3:08 PM Xintong Song 
> > > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > I'm here to re-start the voting process for FLIP-49 [1], with
> respect
> > > to
> > > > > consensus reached in this thread [2] regarding some new comments
> and
> > > > > concerns.
> > > > >
> > > > > This voting will be open for at least 72 hours. I'll try to close
> it
> > > Sep.
> > > > > 5, 14:00 UTC, unless there is an objection or not enough votes.
> > > > >
> > > > > Thank you~
> > > > >
> > > > > Xintong Song
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors
> > > > > [2]
> > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-49-Unified-Memory-Configuration-for-TaskExecutors-td31436.html
> > > > >
> > > > > On Tue, Aug 27, 2019 at 9:29 PM Xintong Song <
> tonysong...@gmail.com>
> 

Re: [VOTE] FLIP-49: Unified Memory Configuration for TaskExecutors

[Stephan Ewen]

Maybe to clear up confusion about my suggestion:

I would vote to keep the name of the config parameter
"taskmanager.memory.network" because it is the same key as currently (good
to not break things unless good reason) and there currently is no case or
even a concrete follow-up where we would actually differentiate between
different types of network memory.

I would suggest to not prematurely rename this because of something that
might happen in the future. Experience shows that its better to do these
things when the actual change comes.

Side note: I am not sure "shuffle" is a good term in this context. I have
so far only heard that in batch contexts, which is not what we do here. One
more reason for me to not pre-maturely change names.

On Wed, Sep 4, 2019 at 10:56 AM Xintong Song  wrote:

> @till
>
> > Just to clarify Xintong, you suggest that Task off-heap memory represents
> > direct and native memory. Since we don't know how the user will allocate
> > the memory we will add this value to -XX:MaxDirectMemorySize so that the
> > process won't fail if the user allocates only direct memory and no native
> > memory. Is that correct?
> >
> Yes, this is what I mean.
>
>
> Thank you~
>
> Xintong Song
>
>
>
> On Wed, Sep 4, 2019 at 4:25 PM Till Rohrmann  wrote:
>
> > Just to clarify Xintong, you suggest that Task off-heap memory represents
> > direct and native memory. Since we don't know how the user will allocate
> > the memory we will add this value to -XX:MaxDirectMemorySize so that the
> > process won't fail if the user allocates only direct memory and no native
> > memory. Is that correct?
> >
> > Cheers,
> > Till
> >
> > On Wed, Sep 4, 2019 at 10:18 AM Xintong Song 
> > wrote:
> >
> > > @Stephan
> > > Not sure what do you mean by "just having this value". Are you
> suggesting
> > > that having "taskmanager.memory.network" refers to "shuffle" and "other
> > > network memory", or only "shuffle"?
> > >
> > > I guess what you mean is only "shuffle"? Because currently
> > > "taskmanager.network.memory" refers to shuffle buffers only, which is
> > "one
> > > less config value to break".
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Wed, Sep 4, 2019 at 3:42 PM Stephan Ewen  wrote:
> > >
> > > > If we later split the network memory into "shuffle" and "other
> network
> > > > memory", I think it would make sense to split the option then.
> > > >
> > > > In that case "taskmanager.memory.network" would probably refer to the
> > > total
> > > > network memory, which would most likely be what most users actually
> > > > configure.
> > > > My feeling is that for now just having this value is actually easier,
> > and
> > > > it is one less config value to break (which is also good).
> > > >
> > > > On Wed, Sep 4, 2019 at 9:05 AM Xintong Song 
> > > wrote:
> > > >
> > > > > Thanks for the voting and comments.
> > > > >
> > > > > @Stephan
> > > > > - The '-XX:MaxDirectMemorySize' value should not include JVM
> > Overhead.
> > > > > Thanks for correction.
> > > > > - 'taskmanager.memory.framework.heap' it heap memory reserved for
> > task
> > > > > executor framework, which can not be allocated to task slots. I
> think
> > > > users
> > > > > should be able to configure both how many total java heap memory a
> > task
> > > > > executor should have and how many of the total java heap memory can
> > be
> > > > > allocated to task slots.
> > > > > - Regarding network / shuffle memory, I'm with @Andrey. ATM, this
> > > memory
> > > > > pool (derived from "taskmanager.network.memory.[min/max/fraction]")
> > is
> > > > only
> > > > > used inside NettyShuffleEnvironment as network buffers. There might
> > be
> > > > > other network memory usage outside the shuffle component (as
> > @Zhijiang
> > > > also
> > > > > suggested), but that is not accounted by this memory pool. This is
> > > > exactly
> > > > > why I would suggest to change the name from 'network' to 'shuffle'.
> > > > > - I

Re: [VOTE] FLIP-49: Unified Memory Configuration for TaskExecutors

[Stephan Ewen]

Let's not block on config key names, just go ahead and we figure this out
concurrently or on the PR later.


On Wed, Sep 4, 2019 at 3:48 PM Stephan Ewen  wrote:

> Maybe to clear up confusion about my suggestion:
>
> I would vote to keep the name of the config parameter
> "taskmanager.memory.network" because it is the same key as currently (good
> to not break things unless good reason) and there currently is no case or
> even a concrete follow-up where we would actually differentiate between
> different types of network memory.
>
> I would suggest to not prematurely rename this because of something that
> might happen in the future. Experience shows that its better to do these
> things when the actual change comes.
>
> Side note: I am not sure "shuffle" is a good term in this context. I have
> so far only heard that in batch contexts, which is not what we do here. One
> more reason for me to not pre-maturely change names.
>
> On Wed, Sep 4, 2019 at 10:56 AM Xintong Song 
> wrote:
>
>> @till
>>
>> > Just to clarify Xintong, you suggest that Task off-heap memory
>> represents
>> > direct and native memory. Since we don't know how the user will allocate
>> > the memory we will add this value to -XX:MaxDirectMemorySize so that the
>> > process won't fail if the user allocates only direct memory and no
>> native
>> > memory. Is that correct?
>> >
>> Yes, this is what I mean.
>>
>>
>> Thank you~
>>
>> Xintong Song
>>
>>
>>
>> On Wed, Sep 4, 2019 at 4:25 PM Till Rohrmann 
>> wrote:
>>
>> > Just to clarify Xintong, you suggest that Task off-heap memory
>> represents
>> > direct and native memory. Since we don't know how the user will allocate
>> > the memory we will add this value to -XX:MaxDirectMemorySize so that the
>> > process won't fail if the user allocates only direct memory and no
>> native
>> > memory. Is that correct?
>> >
>> > Cheers,
>> > Till
>> >
>> > On Wed, Sep 4, 2019 at 10:18 AM Xintong Song 
>> > wrote:
>> >
>> > > @Stephan
>> > > Not sure what do you mean by "just having this value". Are you
>> suggesting
>> > > that having "taskmanager.memory.network" refers to "shuffle" and
>> "other
>> > > network memory", or only "shuffle"?
>> > >
>> > > I guess what you mean is only "shuffle"? Because currently
>> > > "taskmanager.network.memory" refers to shuffle buffers only, which is
>> > "one
>> > > less config value to break".
>> > >
>> > > Thank you~
>> > >
>> > > Xintong Song
>> > >
>> > >
>> > >
>> > > On Wed, Sep 4, 2019 at 3:42 PM Stephan Ewen  wrote:
>> > >
>> > > > If we later split the network memory into "shuffle" and "other
>> network
>> > > > memory", I think it would make sense to split the option then.
>> > > >
>> > > > In that case "taskmanager.memory.network" would probably refer to
>> the
>> > > total
>> > > > network memory, which would most likely be what most users actually
>> > > > configure.
>> > > > My feeling is that for now just having this value is actually
>> easier,
>> > and
>> > > > it is one less config value to break (which is also good).
>> > > >
>> > > > On Wed, Sep 4, 2019 at 9:05 AM Xintong Song 
>> > > wrote:
>> > > >
>> > > > > Thanks for the voting and comments.
>> > > > >
>> > > > > @Stephan
>> > > > > - The '-XX:MaxDirectMemorySize' value should not include JVM
>> > Overhead.
>> > > > > Thanks for correction.
>> > > > > - 'taskmanager.memory.framework.heap' it heap memory reserved for
>> > task
>> > > > > executor framework, which can not be allocated to task slots. I
>> think
>> > > > users
>> > > > > should be able to configure both how many total java heap memory a
>> > task
>> > > > > executor should have and how many of the total java heap memory
>> can
>> > be
>> > > > > allocated to task slots.
>> > > > > - Regarding network / shuffle memory, I'm with @Andrey. ATM,

Re: [VOTE] FLIP-49: Unified Memory Configuration for TaskExecutors

[Stephan Ewen]

One thing that I just came across: Some of these options should also have a
corresponding value for the JobManager, like JVM overhead, metaspace,
direct memory.

On Fri, Sep 6, 2019 at 4:34 AM Xintong Song  wrote:

> Thanks all for the votes.
> So far, we have
>
>- 4 binding +1 votes (Stephan, Andrey, Till, Zhijiang)
>- 2 un-binding +1 votes (Xintong, Yu)
>- No -1 votes
>
> There are more than 3 binding +1 votes and no -1 votes, and the voting time
> has past. According to the new bylaws, I'm happily to announce that FLIP-49
> is approved to be adopted by Apache Flink.
>
> Regarding the minors mentioned during the voting, if there's no objection,
> I would like to update the FLIP document with the followings
>
>- Exclude JVM Overhead from '-XX:MaxDirectMemorySize'
>- Add a 'Follow-Up' section, with the follow-ups of web ui and
>documentation issues
>- Add a 'Limitation' section, with the Unsafe Java 12 support issue
>
>
> Thank you~
>
> Xintong Song
>
>
>
> On Fri, Sep 6, 2019 at 10:28 AM Xintong Song 
> wrote:
>
> > +1 (non-binding) from my side.
> >
> > @Yu, thanks for the vote.
> > - The current FLIP document already mentioned the issue that Unsafe is
> not
> > supported in Java 12, in the section 'Unifying Explicit and Implicit
> Memory
> > Allocation'. It makes sense to me to emphasize this by adding a separate
> > limitation section.
> > - I think we should also update the FLIP document if we change the config
> > names later in PRs. But I would not consider this as a major change to
> the
> > FLIP that requires another vote, especially when we already agreed during
> > this vote to revisit the config names at implementation stage.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Fri, Sep 6, 2019 at 2:43 AM Yu Li  wrote:
> >
> >> +1 (non-binding)
> >>
> >> Minor:
> >> 1. Is it worth a separate "Limitations" section to contain all relative
> >> information like the Unsafe support issue in Java 12, just like many
> other
> >> FLIPs?
> >> 2. About the config names, if we change them later in PR, does it mean
> we
> >> will need to update the FLIP document? If so, it seems we need another
> >> vote
> >> after the modification according to current bylaw? Or maybe we could
> just
> >> create a subpage under the FLIP and only re-vote on that part later?
> >>
> >> Thanks.
> >>
> >> Best Regards,
> >> Yu
> >>
> >>
> >> On Thu, 5 Sep 2019 at 00:52, Stephan Ewen  wrote:
> >>
> >> > Let's not block on config key names, just go ahead and we figure this
> >> out
> >> > concurrently or on the PR later.
> >> >
> >> >
> >> > On Wed, Sep 4, 2019 at 3:48 PM Stephan Ewen  wrote:
> >> >
> >> > > Maybe to clear up confusion about my suggestion:
> >> > >
> >> > > I would vote to keep the name of the config parameter
> >> > > "taskmanager.memory.network" because it is the same key as currently
> >> > (good
> >> > > to not break things unless good reason) and there currently is no
> >> case or
> >> > > even a concrete follow-up where we would actually differentiate
> >> between
> >> > > different types of network memory.
> >> > >
> >> > > I would suggest to not prematurely rename this because of something
> >> that
> >> > > might happen in the future. Experience shows that its better to do
> >> these
> >> > > things when the actual change comes.
> >> > >
> >> > > Side note: I am not sure "shuffle" is a good term in this context. I
> >> have
> >> > > so far only heard that in batch contexts, which is not what we do
> >> here.
> >> > One
> >> > > more reason for me to not pre-maturely change names.
> >> > >
> >> > > On Wed, Sep 4, 2019 at 10:56 AM Xintong Song  >
> >> > > wrote:
> >> > >
> >> > >> @till
> >> > >>
> >> > >> > Just to clarify Xintong, you suggest that Task off-heap memory
> >> > >> represents
> >> > >> > direct and native memory. Since we don't know how the user will
> >> > allocate
> >> > >> > the memory we will add th

Re: [DISCUSS] Support notifyOnMaster for notifyCheckpointComplete

[Stephan Ewen]

Hi all!

I think it would be time to rethink the Sink API as a whole, like we did
with the Source API in FLIP-27.
It would be nice to have proper design that handles all this consistently,
rather than adding one more hook.

For example:
  - For batch, you can already use the existing "finalize on master" hook
  - For streaming, it is tricky to "commit on end-of-stream" reliably
(tolerating failures)
  - write ahead versus direct writing
  - transactional versus idempotent
  - temp files and renaming versus recoverable writer
==> All these things have special cases currently, rather than a coherent
design.

Best,
Stephan


On Tue, Sep 10, 2019 at 6:40 AM Dian Fu  wrote:

> Hi Jingsong,
>
> Good point!
>
> 1. If it doesn't matter which task performs the finalize work, then I
> think task-0 suggested by Jark is a very good solution.
> 2. If it requires the last finished task to perform the finalize work,
> then we have to consider other solutions.
>   WRT fault-tolerant of StreamingRuntimeContext#getGlobalAggregateManager,
> AFAIK, there is no built-in support.
> 1) Regarding to TM failover, I think it's not a problem. We can use an
> accumulator i.e. finish_count and it is increased by 1 when a sub-task is
> finished(i.e. close() method is called).
>When finish_count == RuntimeContext.getNumberOfParallelSubtasks()
> for some sub-task, then we can know that it's the last finished sub-task.
> This holds true even in case of TM failover.
> 2) Regarding to JM failover, I have no idea how to work around it so
> far. Maybe @Jamie Grier who is the author of this feature could share more
> thoughts. Not sure if there is already solution/plan to support JM failover
> or this feature is not designed for this kind of use case?
>
> Regards,
> Dian
>
> > 在 2019年9月9日，下午3:08，shimin yang  写道：
> >
> > Hi Jingsong,
> >
> > Although it would be nice if the accumulators in GlobalAggregateManager
> is
> > fault-tolerant, we could still take advantage of managed state to
> guarantee
> > the semantic and use the accumulators to implement distributed barrier or
> > lock to solve the distributed access problem.
> >
> > Best,
> > Shimin
> >
> > JingsongLee  于2019年9月9日周一 下午1:33写道：
> >
> >> Thanks jark and dian:
> >> 1.jark's approach: do the work in task-0. Simple way.
> >> 2.dian's approach: use StreamingRuntimeContext#getGlobalAggregateManager
> >> Can do more operation. But these accumulators are not fault-tolerant?
> >>
> >> Best,
> >> Jingsong Lee
> >>
> >>
> >> --
> >> From:shimin yang 
> >> Send Time:2019年9月6日(星期五) 15:21
> >> To:dev 
> >> Subject:Re: [DISCUSS] Support notifyOnMaster for
> notifyCheckpointComplete
> >>
> >> Hi Fu,
> >>
> >> That'll be nice.
> >>
> >> Thanks.
> >>
> >> Best,
> >> Shimin
> >>
> >> Dian Fu  于2019年9月6日周五 下午3:17写道：
> >>
> >>> Hi Shimin,
> >>>
> >>> It can be guaranteed to be an atomic operation. This is ensured by the
> >> RPC
> >>> framework. You could take a look at RpcEndpoint for more details.
> >>>
> >>> Regards,
> >>> Dian
> >>>
>  在 2019年9月6日，下午2:35，shimin yang  写道：
> 
>  Hi Fu,
> 
>  Thank you for the remind. I think it would work in my case as long as
> >>> it's
>  an atomic operation.
> 
>  Dian Fu  于2019年9月6日周五 下午2:22写道：
> 
> > Hi Jingsong,
> >
> > Thanks for bring up this discussion. You can try to look at the
> > GlobalAggregateManager to see if it can meet your requirements. It
> can
> >>> be
> > got via StreamingRuntimeContext#getGlobalAggregateManager().
> >
> > Regards,
> > Dian
> >
> >> 在 2019年9月6日，下午1:39，shimin yang  写道：
> >>
> >> Hi Jingsong,
> >>
> >> Big fan of this idea. We faced the same problem and resolved by
> >> adding
> >>> a
> >> distributed lock. It would be nice to have this feature in
> JobMaster,
> > which
> >> can replace the lock.
> >>
> >> Best,
> >> Shimin
> >>
> >> JingsongLee  于2019年9月6日周五
> >> 下午12:20写道：
> >>
> >>> Hi devs:
> >>>
> >>> I try to implement streaming file sink for table[1] like
> > StreamingFileSink.
> >>> If the underlying is a HiveFormat, or a format that updates
> >> visibility
> >>> through a metaStore, I have to update the metaStore in the
> >>> notifyCheckpointComplete, but this operation occurs on the task
> >> side,
> >>> which will lead to distributed access to the metaStore, which will
> >>> lead to bottleneck.
> >>>
> >>> So I'm curious if we can support notifyOnMaster for
> >>> notifyCheckpointComplete like FinalizeOnMaster.
> >>>
> >>> What do you think?
> >>>
> >>> [1]
> >>>
> >
> >>>
> >>
> https://docs.google.com/document/d/15R3vZ1R_pAHcvJkRx_CWleXgl08WL3k_ZpnWSdzP7GY/edit?usp=sharing
> >>>
> >>> Best,
> >>> Jingsong Lee
> >
> >
> >>>
> >>>
> >>
>
>

Re: [DISCUSS] Contribute Pulsar Flink connector back to Flink

[Stephan Ewen]

Hi all!

Nice to see this lively discussion about the Pulsar connector.
Some thoughts on the open questions:

## Contribute to Flink or maintain as a community package

Looks like the discussion is more going towards contribution. I think that
is good, especially if we think that we want to build a similarly deep
integration with Pulsar as we have for example with Kafka. The connector
already looks like a more thorough connector than many others we have in
the repository.

With either a repo split, or the new build system, I hope that the build
overhead is not a problem.

## Committer Support

Becket offered some help already, I can also help a bit. I hope that
between us, we can cover this.

## Contribute now, or wait for FLIP-27

As Becket said, FLIP-27 is actually making some PoC-ing progress, but will
take 2 more months, I would estimate, before it is fully available.

If we want to be on the safe side with the contribution, we should
contribute the source sooner and adjust it later. That would also help us
in case things get crazy towards the 1.10 feature freeze and it would be
hard to find time to review the new changes.
What would be the overhead of contributing now? Given that the code is
already there, it looks like it would be only review capacity, right?

Best,
Stephan

On Tue, Sep 10, 2019 at 11:04 AM Yijie Shen 
wrote:

> Hi everyone!
>
> Thanks for your attention and the promotion of this work.
>
> We will prepare a FLIP as soon as possible for more specific discussions.
>
> For FLIP-27, it seems that we have not reached a consensus. Therefore,
> I will explain all the functionalities of the existing connector in
> the FLIP (including Source, Sink, and Catalog) to continue our
> discussions in FLIP.
>
> Thanks for your kind help.
>
> Best,
> Yijie
>
> On Tue, Sep 10, 2019 at 9:57 AM Becket Qin  wrote:
> >
> > Hi Sijie,
> >
> > If we agree that the goal is to have Pulsar connector in 1.10, how about
> we
> > do the following:
> >
> > 0. Start a FLIP to add Pulsar connector to Flink main repo as it is a new
> > public interface to Flink main repo.
> > 1. Start to review the Pulsar sink right away as there is no change to
> the
> > sink interface so far.
> > 2. Wait a little bit on FLIP-27. Flink 1.10 is going to be code freeze in
> > late Nov and let's say we give a month to the development and review of
> > Pulsar connector, we need to have FLIP-27 by late Oct. There are still 7
> > weeks. Personally I think it is doable. If FLIP-27 is not ready by late
> > Oct, we can review and check in Pulsar connector with the existing source
> > interface. This means we will have Pulsar connector in Flink 1.10, either
> > with or without FLIP-27.
> >
> > Because we are going to have Pulsar sink and source checked in
> separately,
> > it might make sense to have two FLIPs, one for Pulsar sink and another
> for
> > Pulsar source. And we can start the work on Pulsar sink right away.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Mon, Sep 9, 2019 at 4:13 PM Sijie Guo  wrote:
> >
> > > Thank you Bowen and Becket.
> > >
> > > What's the take from Flink community? Shall we wait for FLIP-27 or
> shall we
> > > proceed to next steps? And what the next steps are? :-)
> > >
> > > Thanks,
> > > Sijie
> > >
> > > On Thu, Sep 5, 2019 at 2:43 PM Bowen Li  wrote:
> > >
> > > > Hi,
> > > >
> > > > I think having a Pulsar connector in Flink can be a good mutual
> benefit
> > > to
> > > > both communities.
> > > >
> > > > Another perspective is that Pulsar connector is the 1st streaming
> > > connector
> > > > that integrates with Flink's metadata management system and Catalog
> APIs.
> > > > It'll be cool to see how the integration turns out and whether we
> need to
> > > > improve Flink Catalog stack, which are currently in Beta, to cater to
> > > > streaming source/sink. Thus I'm in favor of merging Pulsar connector
> into
> > > > Flink 1.10.
> > > >
> > > > I'd suggest to submit smaller sized PRs, e.g. maybe one for basic
> > > > source/sink functionalities and another for schema and catalog
> > > integration,
> > > > just to make them easier to review.
> > > >
> > > > It doesn't seem to hurt to wait for FLIP-27. But I don't think
> FLIP-27
> > > > should be a blocker in cases where it cannot make its way into 1.10
> or
> > > > doesn't leave reasonable amount of time for committers to review or
> for
> > > > Pulsar connector to fully adapt to new interfaces.
> > > >
> > > > Bowen
> > > >
> > > >
> > > >
> > > > On Thu, Sep 5, 2019 at 3:21 AM Becket Qin 
> wrote:
> > > >
> > > > > Hi Till,
> > > > >
> > > > > You are right. It all depends on when the new source interface is
> going
> > > > to
> > > > > be ready. Personally I think it would be there in about a month or
> so.
> > > > But
> > > > > I could be too optimistic. It would also be good to hear what do
> > > Aljoscha
> > > > > and Stephan think as they are also involved in FLIP-27.
> > > > >
> > > > > In general I think we should have Pulsar connector in

[DISCUSS] Drop older versions of Kafka Connectors (0.9, 0.10) for Flink 1.10

[Stephan Ewen]

Hi all!

We still maintain connectors for Kafka 0.8 and 0.9 in Flink.
I would suggest to drop those with Flink 1.10 and start supporting only
Kafka 0.10 onwards.

Are there any concerns about this, or still a significant number of users
of these versions?

Best,
Stephan

Re: [DISCUSS] Contribute Pulsar Flink connector back to Flink

[Stephan Ewen]

Agreed, if we check in the old code, we should make it clear that it will
be removed as soon as the FLIP-27 based version of the connector is there.
We should not commit to maintaining the old version, that would be indeed
too much overhead.

On Thu, Sep 12, 2019 at 3:30 AM Becket Qin  wrote:

> Hi Stephan,
>
> Thanks for the volunteering to help.
>
> Yes, the overhead would just be review capacity. In fact, I am not worrying
> too much about the review capacity. That is just a one time cost. My
> concern is mainly about the long term burden. Assume we have new source
> interface ready in 1.10 with newly added Pulsar connectors in old
> interface. Later on if we migrate Pulsar to new source interface, the old
> Pulsar connector might be deprecated almost immediately after checked in,
> but we may still have to maintain two code bases. For the existing
> connectors, we have to do that anyways. But it would be good to avoid
> introducing a new connector with the same problem.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Tue, Sep 10, 2019 at 6:51 PM Stephan Ewen  wrote:
>
> > Hi all!
> >
> > Nice to see this lively discussion about the Pulsar connector.
> > Some thoughts on the open questions:
> >
> > ## Contribute to Flink or maintain as a community package
> >
> > Looks like the discussion is more going towards contribution. I think
> that
> > is good, especially if we think that we want to build a similarly deep
> > integration with Pulsar as we have for example with Kafka. The connector
> > already looks like a more thorough connector than many others we have in
> > the repository.
> >
> > With either a repo split, or the new build system, I hope that the build
> > overhead is not a problem.
> >
> > ## Committer Support
> >
> > Becket offered some help already, I can also help a bit. I hope that
> > between us, we can cover this.
> >
> > ## Contribute now, or wait for FLIP-27
> >
> > As Becket said, FLIP-27 is actually making some PoC-ing progress, but
> will
> > take 2 more months, I would estimate, before it is fully available.
> >
> > If we want to be on the safe side with the contribution, we should
> > contribute the source sooner and adjust it later. That would also help us
> > in case things get crazy towards the 1.10 feature freeze and it would be
> > hard to find time to review the new changes.
> > What would be the overhead of contributing now? Given that the code is
> > already there, it looks like it would be only review capacity, right?
> >
> > Best,
> > Stephan
> >
> > On Tue, Sep 10, 2019 at 11:04 AM Yijie Shen 
> > wrote:
> >
> > > Hi everyone!
> > >
> > > Thanks for your attention and the promotion of this work.
> > >
> > > We will prepare a FLIP as soon as possible for more specific
> discussions.
> > >
> > > For FLIP-27, it seems that we have not reached a consensus. Therefore,
> > > I will explain all the functionalities of the existing connector in
> > > the FLIP (including Source, Sink, and Catalog) to continue our
> > > discussions in FLIP.
> > >
> > > Thanks for your kind help.
> > >
> > > Best,
> > > Yijie
> > >
> > > On Tue, Sep 10, 2019 at 9:57 AM Becket Qin 
> wrote:
> > > >
> > > > Hi Sijie,
> > > >
> > > > If we agree that the goal is to have Pulsar connector in 1.10, how
> > about
> > > we
> > > > do the following:
> > > >
> > > > 0. Start a FLIP to add Pulsar connector to Flink main repo as it is a
> > new
> > > > public interface to Flink main repo.
> > > > 1. Start to review the Pulsar sink right away as there is no change
> to
> > > the
> > > > sink interface so far.
> > > > 2. Wait a little bit on FLIP-27. Flink 1.10 is going to be code
> freeze
> > in
> > > > late Nov and let's say we give a month to the development and review
> of
> > > > Pulsar connector, we need to have FLIP-27 by late Oct. There are
> still
> > 7
> > > > weeks. Personally I think it is doable. If FLIP-27 is not ready by
> late
> > > > Oct, we can review and check in Pulsar connector with the existing
> > source
> > > > interface. This means we will have Pulsar connector in Flink 1.10,
> > either
> > > > with or without FLIP-27.
> > > >
> > > > Because we are going to have Pulsar sink and source checked in
> > > separately,
> >

Re: [DISCUSS] Use Java's Duration instead of Flink's Time

[Stephan Ewen]

I would deprecate "org.apache.flink.api.common.time.Time" only if we have
alternative methods for the window api.
If users can only write code by using a deprecated class, that would not be
a good experience.

Otherwise sound good.

Best,
Stephan


On Thu, Sep 12, 2019 at 11:23 AM Zili Chen  wrote:

> Hi,
>
> I've given it a try to switch to Java's Duration for all runtime Java code.
> Generally, most of its usages are for @RpcTimeout and testing timeout.
>
> However, do a clean work without touch public interface possibly introduce
> bridge codes convert runtime Java Duration to Flink's Time. I don't think
> it is worth to do the detailed distinguish job, and even we possibly
> introduce
> extra bridge codes.
>
> Given this, I just filed an issue about this topic(we should have done :-))
> FLINK-14068[1] as a subtask of FLINK-3957 "Breaking changes for Flink 2.0".
>
> For what we can do now,
>
> 1. Deprecate org.apache.flink.api.common.time.Time also.
> 2. Stop introduce more usages of Flink's Time, specifically for testing
> timeout. This could be manually checked when review pull request(not
> perfect
> thought :\)
>
> We can do the final removal at once when prepare for 2.0 though.
>
> Best,
> tison.
>
> [1]https://issues.apache.org/jira/browse/FLINK-14068
>
>
> Stephan Ewen  于2019年8月27日周二 上午1:19写道：
>
> > Seems everyone is in favor in principle.
> >
> >   - For public APIs, I would keep Time for now (to not break the API).
> > Maybe add a Duration variant and deprecate the Time variant, but not
> remove
> > it before Flink 1.0
> >   - For all runtime Java code, switch to Java's Duration now
> >   - For all Scala code let's see how much we can switch to Java Durations
> > without blowing up stuff. After all, like Tison said, we want to get the
> > runtime Scala free in the future.
> >
> > On Mon, Aug 26, 2019 at 3:45 AM Jark Wu  wrote:
> >
> > > +1 to use Java's Duration instead of Flink's Time.
> > >
> > > Regarding to the Duration parsing, we have mentioned this in FLIP-54[1]
> > to
> > > use `org.apache.flink.util.TimeUtils` for the parsing.
> > >
> > > Best,
> > > Jark
> > >
> > > [1]:
> > >
> > >
> >
> https://docs.google.com/document/d/1IQ7nwXqmhCy900t2vQLEL3N2HIdMg-JO8vTzo1BtyKU/edit#heading=h.egdwkc93dn1k
> > >
> > > On Sat, 24 Aug 2019 at 18:24, Zhu Zhu  wrote:
> > >
> > > > +1 since Java Duration is more common and powerful than Flink Time.
> > > >
> > > > For whether to drop scala Duration for parsing duration
> OptionConfig, I
> > > > think it's another question and should be discussed in another
> thread.
> > > >
> > > > Thanks,
> > > > Zhu Zhu
> > > >
> > > > Becket Qin  于2019年8月24日周六 下午4:16写道：
> > > >
> > > > > +1, makes sense. BTW, we probably need a FLIP as this is a public
> API
> > > > > change.
> > > > >
> > > > > On Sat, Aug 24, 2019 at 8:11 AM SHI Xiaogang <
> shixiaoga...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > +1 to replace Flink's time with Java's Duration.
> > > > > >
> > > > > > Besides, i also suggest to use Java's Instant for
> "point-in-time".
> > > > > > It can take care of time units when we calculate Duration between
> > > > > different
> > > > > > instants.
> > > > > >
> > > > > > Regards,
> > > > > > Xiaogang
> > > > > >
> > > > > > Zili Chen  于2019年8月24日周六 上午10:45写道：
> > > > > >
> > > > > > > Hi vino,
> > > > > > >
> > > > > > > I agree that it introduces extra complexity to replace
> > > > Duration(Scala)
> > > > > > > with Duration(Java) *in Scala code*. We could separate the
> usage
> > > for
> > > > > each
> > > > > > > language and use a bridge when necessary.
> > > > > > >
> > > > > > > As a matter of fact, Scala concurrent APIs(including Duration)
> > are
> > > > used
> > > > > > > more than necessary at least in flink-runtime. Also we even try
> > to
> > > > make
> > > > > > > flink-runtime scala free.
> > > > > > >
> &

Re: Checkpoint metrics.

[Stephan Ewen]

Hi Jamie!

Did you mean to attach a screenshot? If yes, you need to share that through
a different channel, the mailing list does not support attachments,
unfortunately.

Seth is right how the time is measured.
One important bit to add to the interpretation:
  - For non-source tasks, the time include the "travel of the barriers",
which can take long under back pressure
  - For source tasks, it includes the "time to acquire the checkpoint
lock", which can be long if the source is blocked in trying to emit data
(again, backpressure).

As part of FLIP-27 we will eliminate the checkpoint lock (have a mailbox
instead) which should lead to faster lock acquisition.

The "unaligned checkpoints" discussion is looking at ways to make
checkpoints much less susceptible to back pressure.

https://lists.apache.org/thread.html/fd5b6cceb4bffb635e26e7ec0787a8db454ddd64aadb40a0d08a90a8@%3Cdev.flink.apache.org%3E

Hope that helps understanding what is going on.

Best,
Stephan


On Thu, Sep 12, 2019 at 1:25 AM Seth Wiesman  wrote:

> Great timing, I just debugged this on Monday. E2e time is checkpoint
> coordinator to checkpoint coordinator, so it includes RPC to the source and
> RPC from the operator back for the JM.
>
> Seth
>
> > On Sep 11, 2019, at 6:17 PM, Jamie Grier 
> wrote:
> >
> > Hey all,
> >
> > I need to make sense of this behavior.  Any help would be appreciated.
> >
> > Here’s an example of a set of Flink checkpoint metrics I don’t
> understand.  This is the first operator in a job and as you can see the
> end-to-end time for the checkpoint is long, but it’s not explained by
> either sync, async, or alignment times.  I’m not sure what to make of
> this.  It makes me think I don’t understand the meaning of the metrics
> themselves.  In my interpretation the end-to-end time should always be,
> roughly, the sum of the other components — certainly in the case of a
> source task such as this.
> >
> > Any thoughts or clarifications anyone can provide on this?  We have many
> jobs with slow checkpoints that suffer from this sort of thing with metrics
> that look similar.
> >
> > -Jamie
> >
>

Re: How stable is FlinkSQL.

[Stephan Ewen]

Can you share some more details?

  - are you running batch SQL or streaming SQL
  - are you running the original Flink SQL engine or the new Blink SQL
engine (since 1.9)

Best,
Stephan


On Fri, Sep 13, 2019 at 3:24 PM srikanth flink  wrote:

> Hi there,
>
> I'm trying to get some hands on with FlinkSQL and take to production, if
> works. Would like to know if someone deployed FlinkSQL in production?
>
> I'm facing issues while running FlinkSQL queries.
>
> Thanks
> Srikanth
>

Re: [DISCUSS] FLIP-60: Restructure the Table API & SQL documentation

[Stephan Ewen]

There are also some other efforts to restructure the docs, which have
resulted until now in more quickstarts and more concepts.

IIRC there is the goal to have a big section on concepts for the whole
system: streaming concepts, time, order, etc.
The API docs would be really more about an API specific reference guide
then.

Should the table API concepts be a section in the overall concepts then?


On Tue, Sep 3, 2019 at 5:11 AM Jark Wu  wrote:

> big +1 to the idea of restructuring the docs. We got a lot of complaints
> from users about the Table & SQL docs.
>
> In general, I think the new structure is very nice.
>
> Regarding to moving "User-defined Extensions" to corresponding broader
> topics, I would prefer current "User-defined Extensions".
> Because it is a more advanced topic than "Connect to external systems" and
> "Builtin Functions", and we can mention the common points (e.g. pom
> dependency) in the overview of the Extensions section.
> Besides that, I would like to keep Builtin Functions as a top-level to make
> it have more exposure and may further split the page.
>
> I have some other suggestions:
>
> 1) Having subpages under "Built-in Functions". For example:
>
> Built-in Functions
>  - Mathematical Functions
>  - Bit Functions
>  - Date and Time Functions
>  - Conditional Functions
>  - String Functions
>  - Aggregate Functions
>  - ...
>
> Currently, all the functions are squeezed in one page. It make the
> page bloated.
> Meanwhile, I think it would be great to enrich the built-in functions with
> argument explanation and more clear examples like MySQL[1] and other
> DataBase docs.
>
> 2) +1 to the "Architecture & Internals" chapter.
> We already have a pull request[2] to add "Streaming Aggregation Performance
> Tuning" page which talks about the performance tuning tips around streaming
> aggregation and the internals.
> Maybe we can put it under the internal chapter or a "Performance Tuning"
> chapter.
>
> 3) How about restructure SQL chapter a bit like this?
>
> SQL
>  - Overview
>  - Data Manipulation Statements (all operations available in SQL)
>  - Data Definition Statements (DDL syntaxes)
>  - Pattern Matching
>
> It renames "Full Reference" to "Data Manipulation Statements" which is more
> align with "Data Definition Statements".
>
>
> Regards,
> Jark
>
> [1]:
>
> https://dev.mysql.com/doc/refman/8.0/en/date-and-time-functions.html#function_adddate
> [2]: https://github.com/apache/flink/pull/9525
>
>
>
>
>
> On Mon, 2 Sep 2019 at 17:29, Kurt Young  wrote:
>
> > +1 to the general idea and thanks for driving this. I think the new
> > structure is
> > more clear than the old one, and i have some suggestions:
> >
> > 1. How about adding a "Architecture & Internals" chapter? This can help
> > developers
> > or users who want to contribute more to have a better understanding about
> > Table.
> > Essentially with blink planner, we merged a lots of codes and features
> but
> > lack of
> > proper user and design documents.
> >
> > 2. Add a dedicated "Hive Integration" chapter. We spend lots of effort on
> > integrating
> > hive, and hive integration is happened in different areas, like catalog,
> > function and
> > maybe ddl in the future. I think a dedicated chapter can make users who
> are
> > interested
> > in this topic easier to find the information they need.
> >
> > 3. Add a chapter about how to manage, monitor or tune the Table & SQL
> jobs,
> > and
> > might adding something like how to migrate old version jobs to new
> version
> > in the future.
> >
> > Best,
> > Kurt
> >
> >
> > On Mon, Sep 2, 2019 at 4:17 PM vino yang  wrote:
> >
> > > Agree with Dawid's suggestion about function.
> > >
> > > Having a Functions section to unify the built-in function and UDF would
> > be
> > > better.
> > >
> > > Dawid Wysakowicz  于2019年8月30日周五 下午7:43写道：
> > >
> > > > +1 to the idea of restructuring the docs.
> > > >
> > > > My only suggestion to consider is how about moving the
> > > > User-Defined-Extensions subpages to corresponding broader topics?
> > > >
> > > > Sources & Sinks >> Connect to external systems
> > > >
> > > > Catalogs >> Connect to external systems
> > > >
> > > > and then have a Functions sections with subsections:
> > > >
> > > > functions
> > > >
> > > > |- built in functions
> > > >
> > > > |- user defined functions
> > > >
> > > >
> > > > Best,
> > > >
> > > > Dawid
> > > >
> > > > On 30/08/2019 10:59, Timo Walther wrote:
> > > > > Hi everyone,
> > > > >
> > > > > the Table API & SQL documentation was already in a very good shape
> in
> > > > > Flink 1.8. However, in the past it was mostly presented as an
> > addition
> > > > > to DataStream API. As the Table and SQL world is growing quickly,
> > > > > stabilizes in its concepts, and is considered as another top-level
> > API
> > > > > and closed ecosystem, it is time to restructure the docs a little
> bit
> > > > > to represent the vision of FLIP-32.
> > > > >
> > > > > Current state:
> > > > > https://ci.apache.o

Re: [DISCUSS] Contribute Pulsar Flink connector back to Flink

[Stephan Ewen]

Some quick thoughts on the connector contribution process. I basically
reiterate here what Thomas mentioned in another thread about the Kinesis
connector.

For connectors, we should favor a low-overhead contribution process, and
accept user code and changes more readily than in the core system.
That is because connectors have both a big variety of scenarios they get
used in (only through use and many small contributions do they become
really useful over time) and at the same time, and committers do not use
the connector themselves and usually cannot foresee too well what is needed.

Further more, a missing connector (or connector feature) is often a bigger
show stopper for users than a missing API or system feature.

Along these lines of thougt, the conclusion would be to take the Pulsar
connector now, focus the review on legal/dependencies/rough code style and
conventions, label it as "beta" (in the sense of "new code" that is "not
yet tested through longer use") and go ahead. And then evolve it quickly
without putting formal blockers in the way, meaning also adding a new FLIP
27 version when it is there.

Best,
Stephan



On Thu, Sep 19, 2019 at 3:47 AM Becket Qin  wrote:

> Hi Yijie,
>
> Could you please follow the FLIP process to start a new FLIP [DISCUSSION]
> thread in the mailing list?
>
>
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process
>
> I see two FLIP-69 discussion in the mailing list now. So there is a FLIP
> number collision. Can you change the FLIP number to 72?
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Thu, Sep 19, 2019 at 12:23 AM Rong Rong  wrote:
>
> > Hi Yijie,
> >
> > Thanks for sharing the pulsar FLIP.
> > Would you mind enabling comments/suggestions on the google doc link? This
> > way the contributors from the community can comment on the doc.
> >
> > Best,
> > Rong
> >
> > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen 
> > wrote:
> >
> > > Hello everyone,
> > >
> > > I've drafted a FLIP that describes the current design of the Pulsar
> > > connector:
> > >
> > >
> > >
> >
> https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#
> > >
> > > Please take a look and let me know what you think.
> > >
> > > Thanks,
> > > Yijie
> > >
> > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong 
> wrote:
> > > >
> > > > Hi All,
> > > >
> > > > Sorry for joining the discussion late and thanks Yijie & Sijie for
> > > driving
> > > > the discussion.
> > > > I also think the Pulsar connector would be a very valuable addition
> to
> > > > Flink. I can also help out a bit on the review side :-)
> > > >
> > > > Regarding the timeline, I also share concerns with Becket on the
> > > > relationship between the new Pulsar connector and FLIP-27.
> > > > There's also another discussion just started by Stephan on dropping
> > Kafka
> > > > 9/10 support on next Flink release [1].  Although the situation is
> > > somewhat
> > > > different, and Kafka 9/10 connector has been in Flink for almost 3-4
> > > years,
> > > > based on the discussion I am not sure if a major version release is a
> > > > requirement for removing old connector supports.
> > > >
> > > > I think there shouldn't be a blocker if we agree the old connector
> will
> > > be
> > > > removed once FLIP-27 based Pulsar connector is there. As Stephan
> > stated,
> > > it
> > > > is easier to contribute the source sooner and adjust it later.
> > > > We should also ensure we clearly communicate the message: for
> example,
> > > > putting an experimental flag on the pre-FLIP27 connector page of the
> > > > website, documentations, etc. Any other thoughts?
> > > >
> > > > --
> > > > Rong
> > > >
> > > > [1]
> > > >
> > >
> >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html
> > > >
> > > >
> > > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin 
> > wrote:
> > > >
> > > > > Technically speaking, removing the old connector code is a
> backwards
> > > > > incompatible change which requires a major version bump, i.e. Flink
> > > 2.x.
> > > > >

Re: [DISCUSS] have separate Flink distributions with built-in Hive dependencies

[Stephan Ewen]

Some thoughts about other options we have:

  - Put fat/shaded jars for the common versions into "flink-shaded" and
offer them for download on the website, similar to pre-bundles Hadoop
versions.

  - Look at the Presto code (Metastore protocol) and see if we can reuse
that

  - Have a setup helper script that takes the versions and pulls the
required dependencies.

Can you share how can a "built-in" dependency could work, if there are so
many different conflicting versions?

Thanks,
Stephan


On Tue, Feb 4, 2020 at 12:59 PM Rui Li  wrote:

> Hi Stephan,
>
> As Jingsong stated, in our documentation the recommended way to add Hive
> deps is to use exactly what users have installed. It's just we ask users to
> manually add those jars, instead of automatically find them based on env
> variables. I prefer to keep it this way for a while, and see if there're
> real concerns/complaints from user feedbacks.
>
> Please also note the Hive jars are not the only ones needed to integrate
> with Hive, users have to make sure flink-connector-hive and Hadoop jars are
> in classpath too. So I'm afraid a single "HIVE" env variable wouldn't save
> all the manual work for our users.
>
> On Tue, Feb 4, 2020 at 5:54 PM Jingsong Li  wrote:
>
> > Hi all,
> >
> > For your information, we have document the dependencies detailed
> > information [1]. I think it's a lot clearer than before, but it's worse
> > than presto and spark (they avoid or have built-in hive dependency).
> >
> > I thought about Stephan's suggestion:
> > - The hive/lib has 200+ jars, but we only need hive-exec.jar or plus two
> > or three jars, if so many jars are introduced, maybe will there be a big
> > conflict.
> > - And hive/lib is not available on every machine. We need to upload so
> > many jars.
> > - A separate classloader maybe hard to work too, our flink-connector-hive
> > need hive jars, we may need to deal with flink-connector-hive jar spacial
> > too.
> > CC: Rui Li
> >
> > I think the best system to integrate with hive is presto, which only
> > connects hive metastore through thrift protocol. But I understand that it
> > costs a lot to rewrite the code.
> >
> > [1]
> >
> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/hive/#dependencies
> >
> > Best,
> > Jingsong Lee
> >
> > On Tue, Feb 4, 2020 at 1:44 AM Stephan Ewen  wrote:
> >
> >> We have had much trouble in the past from "too deep too custom"
> >> integrations that everyone got out of the box, i.e., Hadoop.
> >> Flink has has such a broad spectrum of use cases, if we have custom
> build
> >> for every other framework in that spectrum, we'll be in trouble.
> >>
> >> So I would also be -1 for custom builds.
> >>
> >> Couldn't we do something similar as we started doing for Hadoop? Moving
> >> away from convenience downloads to allowing users to "export" their
> setup
> >> for Flink?
> >>
> >>   - We can have a "hive module (loader)" in flink/lib by default
> >>   - The module loader would look for an environment variable like
> >> "HIVE_CLASSPATH" and load these classes (ideally in a separate
> >> classloader).
> >>   - The loader can search for certain classes and instantiate catalog /
> >> functions / etc. when finding them instantiates the hive module
> >> referencing
> >> them
> >>   - That way, we use exactly what users have installed, without needing
> to
> >> build our own bundles.
> >>
> >> Could that work?
> >>
> >> Best,
> >> Stephan
> >>
> >>
> >> On Wed, Dec 18, 2019 at 9:43 AM Till Rohrmann 
> >> wrote:
> >>
> >> > Couldn't it simply be documented which jars are in the convenience
> jars
> >> > which are pre built and can be downloaded from the website? Then
> people
> >> who
> >> > need a custom version know which jars they need to provide to Flink?
> >> >
> >> > Cheers,
> >> > Till
> >> >
> >> > On Tue, Dec 17, 2019 at 6:49 PM Bowen Li  wrote:
> >> >
> >> > > I'm not sure providing an uber jar would be possible.
> >> > >
> >> > > Different from kafka and elasticsearch connector who have
> dependencies
> >> > for
> >> > > a specific kafka/elastic version, or the kafka universal connector
> >> that
> >> > > provides good c

Re: [VOTE] Release 1.10.0, release candidate #1

[Stephan Ewen]

Should we make these a blocker? I am not sure - we could also clearly state
in the release notes how to restore the old behavior, if your setup assumes
that behavior.

Release candidates for this release have been out since mid December, it is
a bit unfortunate that these things have been raised so late.
Having these rather open ended tickets (how to re-define the existing
metrics in the new scheduler/failover handling) now as release blockers
would mean that 1.10 is indefinitely delayed.

Are we sure we want to do that?

On Wed, Feb 5, 2020 at 6:53 PM Thomas Weise  wrote:

> Hi Gary,
>
> Thanks for the clarification!
>
> When we upgrade to a new Flink release, we don't start with a default
> flink-conf.yaml but upgrade our existing tooling and configuration.
> Therefore we notice this issue as part of the upgrade to 1.10, and not when
> we upgraded to 1.9.
>
> I would expect many other users to be in the same camp, and therefore
> consider making these regressions a blocker for 1.10?
>
> Thanks,
> Thomas
>
>
> On Wed, Feb 5, 2020 at 4:53 AM Gary Yao  wrote:
>
> > > also notice that the exception causing a restart is no longer displayed
> > > in the UI, which is probably related?
> >
> > Yes, this is also related to the new scheduler. I created FLINK-15917 [1]
> > to
> > track this. Moreover, I created a ticket about the uptime metric not
> > resetting
> > [2]. Both issues already exist in 1.9 if
> > "jobmanager.execution.failover-strategy" is set to "region", which is the
> > case
> > in the default flink-conf.yaml.
> >
> > In 1.9, unsetting "jobmanager.execution.failover-strategy" was enough to
> > fall
> > back to the previous behavior.
> >
> > In 1.10, you can still fall back to the previous behavior by setting
> > "jobmanager.scheduler: legacy" and unsetting
> > "jobmanager.execution.failover-strategy" in your flink-conf.yaml
> >
> > I would not consider these issues blockers since there is a workaround
> for
> > them, but of course we would like to see the new scheduler getting some
> > production exposure. More detailed release notes about the caveats of the
> > new
> > scheduler will be added to the user documentation.
> >
> >
> > > The watermark issue was
> > https://issues.apache.org/jira/browse/FLINK-14470
> >
> > This should be fixed now [3].
> >
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-15917
> > [2] https://issues.apache.org/jira/browse/FLINK-15918
> > [3] https://issues.apache.org/jira/browse/FLINK-8949
> >
> > On Wed, Feb 5, 2020 at 7:04 AM Thomas Weise  wrote:
> >
> >> Hi Gary,
> >>
> >> Thanks for the reply.
> >>
> >> -->
> >>
> >> On Tue, Feb 4, 2020 at 5:20 AM Gary Yao  wrote:
> >>
> >> > Hi Thomas,
> >> >
> >> > > 2) Was there a change in how job recovery reflects in the uptime
> >> metric?
> >> > > Didn't uptime previously reset to 0 on recovery (now it just keeps
> >> > > increasing)?
> >> >
> >> > The uptime is the difference between the current time and the time
> when
> >> the
> >> > job transitioned to RUNNING state. By default we no longer transition
> >> the
> >> > job
> >> > out of the RUNNING state when restarting. This has something to do
> with
> >> the
> >> > new scheduler which enables pipelined region failover by default [1].
> >> > Actually
> >> > we enabled pipelined region failover already in the binary
> distribution
> >> of
> >> > Flink 1.9 by setting:
> >> >
> >> > jobmanager.execution.failover-strategy: region
> >> >
> >> > in the default flink-conf.yaml. Unless you have removed this config
> >> option
> >> > or
> >> > you are using a custom yaml, you should be seeing this behavior in
> Flink
> >> > 1.9.
> >> > If you do not want region failover, set
> >> >
> >> > jobmanager.execution.failover-strategy: full
> >> >
> >> >
> >> We are using the default (the jobmanager.execution.failover-strategy
> >> setting is not present in our flink config).
> >>
> >> The change in behavior I see is between the 1.9 based deployment and the
> >> 1.10 RC.
> >>
> >> Our 1.9 branch is here:
> >> https://github.com/lyft/flink/tree/release-1.9-lyft
> >>
> >> I also notice that the exception causing a restart is no longer
> displayed
> >> in the UI, which is probably related?
> >>
> >>
> >> >
> >> > > 1) Is the low watermark display in the UI still broken?
> >> >
> >> > I was not aware that this is broken. Is there an issue tracking this
> >> bug?
> >> >
> >>
> >> The watermark issue was
> https://issues.apache.org/jira/browse/FLINK-14470
> >>
> >> (I don't have a good way to verify it is fixed at the moment.)
> >>
> >> Another problem with this 1.10 RC is that the checkpointAlignmentTime
> >> metric is missing. (I have not been able to investigate this further
> yet.)
> >>
> >>
> >> >
> >> > Best,
> >> > Gary
> >> >
> >> > [1] https://issues.apache.org/jira/browse/FLINK-14651
> >> >
> >> > On Tue, Feb 4, 2020 at 2:56 AM Thomas Weise  wrote:
> >> >
> >> >> I opened a PR for FLINK-15868
> >> >> :
> >> >> https://gi

Re: [DISCUSS] have separate Flink distributions with built-in Hive dependencies

[Stephan Ewen]

Hi Jingsong!

This sounds that with two pre-bundled versions (hive 1.2.1 and hive 2.3.6)
you can cover a lot of versions.

Would it make sense to add these to flink-shaded (with proper dependency
exclusions of unnecessary dependencies) and offer them as a download,
similar as we offer pre-shaded Hadoop downloads?

Best,
Stephan


On Thu, Feb 6, 2020 at 10:26 AM Jingsong Li  wrote:

> Hi Stephan,
>
> The hive/lib/ has many jars, this lib is for execution, metastore, hive
> client and all things.
> What we really depend on is hive-exec.jar. (hive-metastore.jar is also
> required in the low version hive)
> And hive-exec.jar is a uber jar. We just want half classes of it. These
> half classes are not so clean, but it is OK to have them.
>
> Our solution now:
> - exclude hive jars from build
> - provide 8 versions dependencies way, user choose by his hive version.[1]
>
> Spark's solution:
> - build-in hive 1.2.1 dependencies to support hive 0.12.0 through 2.3.3.
> [2]
> - hive-exec.jar is hive-exec.spark.jar, Spark has modified the
> hive-exec build pom to exclude unnecessary classes including Orc and
> parquet.
> - build-in orc and parquet dependencies to optimizer performance.
> - support hive version 2.3.3 upper by "mvn install -Phive-2.3", to
> built-in hive-exec-2.3.6.jar. It seems that since this version, hive's API
> has been seriously incompatible.
> Most of the versions used by users are hive 0.12.0 through 2.3.3. So the
> default build of Spark is good to most of users.
>
> Presto's solution:
> - Built-in presto's hive.[3] Shade hive classes instead of thrift classes.
> - Rewrite some client related code to solve kinds of issues.
> This approach is the heaviest, but also the cleanest. It can support all
> kinds of hive versions with one build.
>
> So I think we can do:
>
> - The eight versions we now maintain are too many. I think we can move
> forward in the direction of Presto/Spark and try to reduce dependencies
> versions.
>
> - As your said, about provide fat/uber jars or helper script, I prefer
> uber jars, user can download one jar to their startup. Just like Kafka.
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/hive/#dependencies
> [2]
> https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html#interacting-with-different-versions-of-hive-metastore
> [3] https://github.com/prestodb/presto-hive-apache
>
> Best,
> Jingsong Lee
>
> On Wed, Feb 5, 2020 at 10:15 PM Stephan Ewen  wrote:
>
>> Some thoughts about other options we have:
>>
>>   - Put fat/shaded jars for the common versions into "flink-shaded" and
>> offer them for download on the website, similar to pre-bundles Hadoop
>> versions.
>>
>>   - Look at the Presto code (Metastore protocol) and see if we can reuse
>> that
>>
>>   - Have a setup helper script that takes the versions and pulls the
>> required dependencies.
>>
>> Can you share how can a "built-in" dependency could work, if there are so
>> many different conflicting versions?
>>
>> Thanks,
>> Stephan
>>
>>
>> On Tue, Feb 4, 2020 at 12:59 PM Rui Li  wrote:
>>
>>> Hi Stephan,
>>>
>>> As Jingsong stated, in our documentation the recommended way to add Hive
>>> deps is to use exactly what users have installed. It's just we ask users
>>> to
>>> manually add those jars, instead of automatically find them based on env
>>> variables. I prefer to keep it this way for a while, and see if there're
>>> real concerns/complaints from user feedbacks.
>>>
>>> Please also note the Hive jars are not the only ones needed to integrate
>>> with Hive, users have to make sure flink-connector-hive and Hadoop jars
>>> are
>>> in classpath too. So I'm afraid a single "HIVE" env variable wouldn't
>>> save
>>> all the manual work for our users.
>>>
>>> On Tue, Feb 4, 2020 at 5:54 PM Jingsong Li 
>>> wrote:
>>>
>>> > Hi all,
>>> >
>>> > For your information, we have document the dependencies detailed
>>> > information [1]. I think it's a lot clearer than before, but it's worse
>>> > than presto and spark (they avoid or have built-in hive dependency).
>>> >
>>> > I thought about Stephan's suggestion:
>>> > - The hive/lib has 200+ jars, but we only need hive-exec.jar or plus
>>> two
>>> > or three jars, if so many jars are introduced, maybe will there be a
>>> big
>>> > conflict.
>&

Re: [VOTE] FLIP-27 - Refactor Source Interface

[Stephan Ewen]

+1 (binding) (belated)

Quick addendum to clarify some questions from recent discussions in other
threads:

  - The core interfaces (Source, SourceReader, Enumerator) and the core
architecture (Enumerator as coordinators on the JobManager, SourceReaders
in Tasks) seem to have no open questions

  - Some open questions remained about whether the proposed base
implementation (SplitReader) fits all possible sources (like block-based
file readers).
  - Even if that turns out to not be the case, this FLIP does not prevent
the creation of another base implementation. That would be completely in
line with this FLIP and exactly the reason why the FLIP proposes the
separation between the simpler core interfaces (SourceReader) and the base
implementations, of which the first one (so that we can start to
immediately build some connectors) is the SplitReader.

On Fri, Feb 7, 2020 at 9:47 AM Becket Qin  wrote:

> Thanks everyone for voting. The voting result is following:
>
> +1 (Binding): 5 (Yu, Jark, Zhijiang, Piotr, Becket)
>
> +1 (Non-binding): 4 (Jingsong, Danny, Wei, Guowei)
>
> -1: 0
>
> FLIP-27 has passed.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Tue, Feb 4, 2020 at 3:42 PM Piotr Nowojski  wrote:
>
> > +1 (binding)
> >
> > Piotrek
> >
> > > On 4 Feb 2020, at 05:39, Zhijiang 
> > wrote:
> > >
> > > +1 (binding), we are waiting too long for it. :)
> > >
> > > Best,
> > > Zhijiang
> > >
> > >
> > > --
> > > From:Guowei Ma 
> > > Send Time:2020 Feb. 4 (Tue.) 12:34
> > > To:dev 
> > > Subject:Re: [VOTE] FLIP-27 - Refactor Source Interface
> > >
> > > +1 (non-binding), thanks for driving.
> > >
> > > Best,
> > > Guowei
> > >
> > >
> > > Jingsong Li  于2020年2月4日周二 上午11:20写道：
> > >
> > >> +1 (non-binding), thanks for driving.
> > >> FLIP-27 is the basis of a lot of follow-up work.
> > >>
> > >> Best,
> > >> Jingsong Lee
> > >>
> > >> On Tue, Feb 4, 2020 at 10:26 AM Jark Wu  wrote:
> > >>
> > >>> Thanks for driving this Becket!
> > >>>
> > >>> +1 from my side.
> > >>>
> > >>> Cheers,
> > >>> Jark
> > >>>
> > >>> On Mon, 3 Feb 2020 at 18:06, Yu Li  wrote:
> > >>>
> >  +1, thanks for the efforts Becket!
> > 
> >  Best Regards,
> >  Yu
> > 
> > 
> >  On Mon, 3 Feb 2020 at 17:52, Becket Qin 
> wrote:
> > 
> > > Bump up the thread.
> > >
> > > On Tue, Jan 21, 2020 at 10:43 AM Becket Qin 
> >  wrote:
> > >
> > >> Hi Folks,
> > >>
> > >> I'd like to resume the voting thread for FlIP-27.
> > >>
> > >> Please note that the FLIP wiki has been updated to reflect the
> > >> latest
> > >> discussions in the discussion thread.
> > >>
> > >> To avoid confusion, I'll only count the votes casted after this
> > >>> point.
> > >>
> > >> FLIP wiki:
> > >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27
> > >> %3A+Refactor+Source+Interface
> > >>
> > >> Discussion thread:
> > >>
> > >>
> > >
> > 
> > >>>
> > >>
> >
> https://lists.apache.org/thread.html/70484d6aa4b8e7121181ed8d5857a94bfb7d5a76334b9c8fcc59700c%40%3Cdev.flink.apache.org%3E
> > >>
> > >> The vote will last for at least 72 hours, following the consensus
> >  voting
> > >> process.
> > >>
> > >> Thanks,
> > >>
> > >> Jiangjie (Becket) Qin
> > >>
> > >> On Thu, Dec 5, 2019 at 10:31 AM jincheng sun <
> > >>> sunjincheng...@gmail.com
> > >
> > >> wrote:
> > >>
> > >>> +1 (binding), and looking forward to seeing the new interface in
> > >> the
> > >>> master.
> > >>>
> > >>> Best,
> > >>> Jincheng
> > >>>
> > >>> Becket Qin  于2019年12月5日周四 上午8:05写道：
> > >>>
> >  Hi all,
> > 
> >  I would like to start the vote for FLIP-27 which proposes to
> > > introduce a
> >  new Source connector interface to address a few problems in the
> > > existing
> >  source connector. The main goals of the the FLIP are following:
> > 
> >  1. Unify the Source interface in Flink for batch and stream.
> >  2. Significantly reduce the work for developers to develop new
> >  source
> >  connectors.
> >  3. Provide a common abstraction for all the sources, as well as
> > >> a
> > >>> mechanism
> >  to allow source subtasks to coordinate among themselves.
> > 
> >  The vote will last for at least 72 hours, following the
> > >> consensus
> > > voting
> >  process.
> > 
> >  FLIP wiki:
> > 
> > 
> > >>>
> > >
> > 
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
> > 
> >  Discussion thread:
> > 
> > 
> > >>>
> > >
> > 
> > >>>
> > >>
> >
> https://lists.apache.org/thread.html/70484d6aa4b8e7121181ed8d5857a94bfb7d5a76334b9c8fcc59700c@%3Cdev.flink.apache.org%3E
> 

Re: [DISCUSS] Does removing deprecated interfaces needs another FLIP

[Stephan Ewen]

I would also agree with the above.

Changing a stable API and deprecating stable methods would need a FLIP in
my opinion. But then executing the removal of previously deprecated methods
would be fine in my understanding.


On Fri, Feb 7, 2020 at 11:17 AM Kurt Young  wrote:

> Thanks for the clarification, that make sense to me.
>
> Best,
> Kurt
>
>
> On Fri, Feb 7, 2020 at 4:56 PM Timo Walther  wrote:
>
> > Hi Kurt,
> >
> > I agree with Aljoscha. We don't need to introduce a big process or do
> > voting but we should ensure that all stakeholders are notified and have
> > a chance to raise doubts.
> >
> > Regards,
> > Timo
> >
> >
> > On 07.02.20 09:51, Aljoscha Krettek wrote:
> > > I would say a ML discussion or even a Jira issue is enough because
> > >
> > > a) the methods are already deprecated
> > > b) the methods are @PublicEvolving, which I don't consider a super
> > > strong guarantee to users (we still shouldn't remove them lightly, but
> > > we can if we have to...)
> > >
> > > Best,
> > > Aljoscha
> > >
> > > On 07.02.20 04:40, Kurt Young wrote:
> > >> Hi dev,
> > >>
> > >> Currently I want to remove some already deprecated methods from
> > >> TableEnvironment which annotated with @PublicEnvolving. And I also
> > >> created
> > >> a discussion thread [1] to both dev and user mailing lists to gather
> > >> feedback on that. But I didn't find any matching rule in Flink bylaw
> > >> [2] to
> > >> follow. Since this is definitely a API breaking change, but we already
> > >> voted for that back in the FLIP which deprecated these methods.
> > >>
> > >> I'm not sure about how to proceed for now. Looks like I have 2
> choices:
> > >>
> > >> 1. If no one raise any objections in discuss thread in like 72 hours,
> I
> > >> will create a jira to start working on it.
> > >> 2. Since this is a API breaking change, I need to open another FLIP to
> > >> tell
> > >> that I want to remove these deprecated methods. This seems a little
> > >> redundant with the first FLIP which deprecate the methods.
> > >>
> > >> What do you think?
> > >>
> > >> Best,
> > >> Kurt
> > >>
> > >> [1]
> > >>
> >
> https://lists.apache.org/thread.html/r98af66feb531ce9e6b94914e44391609cad857e16ea84db5357c1980%40%3Cdev.flink.apache.org%3E
> > >>
> > >> [2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws
> > >>
> >
> >
>

Re: [DISCUSS] Drop connectors for Elasticsearch 2.x and 5.x

[Stephan Ewen]

+1 to drop ES 2.x - unsure about 5.x (makes sense to get more user input
for that one).

@Itamar - if you would be interested in contributing a "universal" or
"cross version" ES connector, that could be very interesting. Do you know
if there are known performance issues or feature restrictions with that
approach?
@dawid what do you think about that?


On Tue, Feb 11, 2020 at 6:28 AM Danny Chan  wrote:

> 5.x seems to have a lot of users, is the 6.x completely compatible with
> 5.x ~
>
> Best,
> Danny Chan
> 在 2020年2月10日 +0800 PM9:45，Dawid Wysakowicz ，写道：
> > Hi all,
> >
> > As described in this https://issues.apache.org/jira/browse/FLINK-11720
> > ticket our elasticsearch 5.x connector does not work out of the box on
> > some systems and requires a version bump. This also happens for our e2e.
> > We cannot bump the version in es 5.x connector, because 5.x connector
> > shares a common class with 2.x that uses an API that was replaced in 5.2.
> >
> > Both versions are already long eol: https://www.elastic.co/support/eol
> >
> > I suggest to drop both connectors 5.x and 2.x. If it is too much to drop
> > both of them, I would strongly suggest dropping at least 2.x connector
> > and update the 5.x line to a working es client module.
> >
> > What do you think? Should we drop both versions? Drop only the 2.x
> > connector? Or keep them both?
> >
> > Best,
> >
> > Dawid
> >
> >
>

Re: [DISCUSS] have separate Flink distributions with built-in Hive dependencies

[Stephan Ewen]

IIRC, Guowei wants to work on supporting Table API connectors in Plugins.
With that, we could have the Hive dependency as a plugin, avoiding
dependency conflicts.

On Thu, Feb 6, 2020 at 1:11 PM Jingsong Li  wrote:

> Hi Stephan,
>
> Good idea. Just like hadoop, we can have flink-shaded-hive-uber.
> Then the startup of hive integration will be very simple with one or two
> pre-bundled, user just add these dependencies:
> - flink-connector-hive.jar
> - flink-shaded-hive-uber-.jar
>
> Some changes are needed, but I think it should work.
>
> Another thing is can we put flink-connector-hive.jar into flink/lib, it
> should clean and no dependencies.
>
> Best,
> Jingsong Lee
>
> On Thu, Feb 6, 2020 at 7:13 PM Stephan Ewen  wrote:
>
>> Hi Jingsong!
>>
>> This sounds that with two pre-bundled versions (hive 1.2.1 and hive
>> 2.3.6) you can cover a lot of versions.
>>
>> Would it make sense to add these to flink-shaded (with proper dependency
>> exclusions of unnecessary dependencies) and offer them as a download,
>> similar as we offer pre-shaded Hadoop downloads?
>>
>> Best,
>> Stephan
>>
>>
>> On Thu, Feb 6, 2020 at 10:26 AM Jingsong Li 
>> wrote:
>>
>>> Hi Stephan,
>>>
>>> The hive/lib/ has many jars, this lib is for execution, metastore, hive
>>> client and all things.
>>> What we really depend on is hive-exec.jar. (hive-metastore.jar is also
>>> required in the low version hive)
>>> And hive-exec.jar is a uber jar. We just want half classes of it. These
>>> half classes are not so clean, but it is OK to have them.
>>>
>>> Our solution now:
>>> - exclude hive jars from build
>>> - provide 8 versions dependencies way, user choose by his hive
>>> version.[1]
>>>
>>> Spark's solution:
>>> - build-in hive 1.2.1 dependencies to support hive 0.12.0 through 2.3.3.
>>> [2]
>>> - hive-exec.jar is hive-exec.spark.jar, Spark has modified the
>>> hive-exec build pom to exclude unnecessary classes including Orc and
>>> parquet.
>>> - build-in orc and parquet dependencies to optimizer performance.
>>> - support hive version 2.3.3 upper by "mvn install -Phive-2.3", to
>>> built-in hive-exec-2.3.6.jar. It seems that since this version, hive's API
>>> has been seriously incompatible.
>>> Most of the versions used by users are hive 0.12.0 through 2.3.3. So the
>>> default build of Spark is good to most of users.
>>>
>>> Presto's solution:
>>> - Built-in presto's hive.[3] Shade hive classes instead of thrift
>>> classes.
>>> - Rewrite some client related code to solve kinds of issues.
>>> This approach is the heaviest, but also the cleanest. It can support all
>>> kinds of hive versions with one build.
>>>
>>> So I think we can do:
>>>
>>> - The eight versions we now maintain are too many. I think we can move
>>> forward in the direction of Presto/Spark and try to reduce dependencies
>>> versions.
>>>
>>> - As your said, about provide fat/uber jars or helper script, I prefer
>>> uber jars, user can download one jar to their startup. Just like Kafka.
>>>
>>> [1]
>>> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/hive/#dependencies
>>> [2]
>>> https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html#interacting-with-different-versions-of-hive-metastore
>>> [3] https://github.com/prestodb/presto-hive-apache
>>>
>>> Best,
>>> Jingsong Lee
>>>
>>> On Wed, Feb 5, 2020 at 10:15 PM Stephan Ewen  wrote:
>>>
>>>> Some thoughts about other options we have:
>>>>
>>>>   - Put fat/shaded jars for the common versions into "flink-shaded" and
>>>> offer them for download on the website, similar to pre-bundles Hadoop
>>>> versions.
>>>>
>>>>   - Look at the Presto code (Metastore protocol) and see if we can
>>>> reuse that
>>>>
>>>>   - Have a setup helper script that takes the versions and pulls the
>>>> required dependencies.
>>>>
>>>> Can you share how can a "built-in" dependency could work, if there are
>>>> so many different conflicting versions?
>>>>
>>>> Thanks,
>>>> Stephan
>>>>
>>>>
>>>> On Tue, Feb 4, 2020 at 12:59 PM Rui Li  wrote:
>>>>
>>>>> Hi Stephan,