[jira] [Created] (FLINK-19870) Fix special case when the reuse of exchange causes the deadlock

[Caizhi Weng (Jira)]

Caizhi Weng created FLINK-19870:
---

 Summary: Fix special case when the reuse of exchange causes the 
deadlock
 Key: FLINK-19870
 URL: https://issues.apache.org/jira/browse/FLINK-19870
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Reporter: Caizhi Weng
 Fix For: 1.12.0


Currently the reuse of exchange is not considered to be a deadlock because 
although the exec node of an exchange is reused, its underlying transformation 
is not reused. However if this behavior changes a deadlock may occur.

For example, consider the following SQL and its plan:

{code:sql}
WITH T1 AS (SELECT a FROM x)
SELECT * FROM T1
  INNER JOIN T1 AS T2 ON T1.a = T2.a
{code}

{code}

HashJoin(joinType=[InnerJoin], where=[=(a, a0)], select=[a, a0], build=[right])
:- Exchange(distribution=[hash[a]], shuffle_mode=[BATCH], reuse_id=[1])
:  +- Calc(select=[a])
: +- LegacyTableSourceScan(table=[[default_catalog, default_database, x, 
source: [TestTableSource(a, b, c)]]], fields=[a, b, c])
+- Reused(reference_id=[1])
{code}

The reuse of exchange may cause a deadlock on hash join.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19869) Support java.util.UUID as RAW type in PostgresCatalog and handle it in PostgresRowConverter

[Jingwei Zhang (Jira)]

Jingwei Zhang created FLINK-19869:
-

 Summary: Support java.util.UUID as RAW type in PostgresCatalog and 
handle it in PostgresRowConverter
 Key: FLINK-19869
 URL: https://issues.apache.org/jira/browse/FLINK-19869
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / JDBC
Affects Versions: 1.11.2
Reporter: Jingwei Zhang


Problem:
UUID is not a SQL standard type. But it is supported by Postgres as an internal 
type which postgres client will help to convert to java.util.UUID when queried. 
However, it seems flink doesn't support it as a sql extension in 
PostgresCatalog. There is no uuid serializer provided for it, either.  So if I 
want to filter a result set and put the filter result back to a table with same 
schema with uuid column type in it. I will have problem. 

Proposal:
Handle UUID in postgres as a SQL extension. Use the DataTypes.RAW to wrap its 
original class info and its serializer so as to expose it as a RAW 
JDBCType(only in postgres) in PostgresCatalog.
And meanwhile, for PostgresRowConverter, we could use BinaryRawValueData to 
handle those SQL extension for byte serialization/deserialization. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19868) Csv Serialization schema contains line delimiter

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-19868:


 Summary: Csv Serialization schema contains line delimiter
 Key: FLINK-19868
 URL: https://issues.apache.org/jira/browse/FLINK-19868
 Project: Flink
  Issue Type: Bug
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Reporter: Jingsong Lee


Csv Serialization schema is for one line, why contains line delimiter?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19867) Validation fails for UDF that accepts var-args

[Rui Li (Jira)]

Rui Li created FLINK-19867:
--

 Summary: Validation fails for UDF that accepts var-args
 Key: FLINK-19867
 URL: https://issues.apache.org/jira/browse/FLINK-19867
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API
Reporter: Rui Li






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19866) FunctionsStateBootstrapOperator.createStateAccessor fails due to uninitialized runtimeContext

[wang (Jira)]

wang created FLINK-19866:


 Summary: FunctionsStateBootstrapOperator.createStateAccessor fails 
due to uninitialized runtimeContext
 Key: FLINK-19866
 URL: https://issues.apache.org/jira/browse/FLINK-19866
 Project: Flink
  Issue Type: Bug
  Components: Stateful Functions
Affects Versions: 1.11.2, statefun-2.2.0
Reporter: wang


It has bugs similar to 
[FLINK-19330|https://issues.apache.org/jira/browse/FLINK-19330]

In Flink 1.11.2, statefun-flink-state-processor 2.2.0, the 
AbstractStreamOperator's runtimeContext is not fully initialized when executing
 AbstractStreamOperator#intializeState()
in particular KeyedStateStore is set after intializeState was finished.
See:
[https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/AbstractStreamOperator.java#L258,L259]
This behaviour was changed from Flink 1.10->Flink 1.11.

StateFun's FunctionsStateBootstrapOperator performs its initialization logic at 
initalizeState, and it requires an already initialized runtimeContext to create 
stateAccessor.

This situation causes the following failure: 
{code:java}
Caused by: java.lang.NullPointerException: Keyed state can only be used on a 
'keyed stream', i.e., after a 'keyBy()' operation.Caused by: 
java.lang.NullPointerException: Keyed state can only be used on a 'keyed 
stream', i.e., after a 'keyBy()' operation. at 
org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:75) at 
org.apache.flink.streaming.api.operators.StreamingRuntimeContext.checkPreconditionsAndGetKeyedStateStore(StreamingRuntimeContext.java:223)
 at 
org.apache.flink.streaming.api.operators.StreamingRuntimeContext.getState(StreamingRuntimeContext.java:188)
 at 
org.apache.flink.statefun.flink.core.state.FlinkState.createFlinkStateAccessor(FlinkState.java:69)
 at 
org.apache.flink.statefun.flink.core.state.FlinkStateBinder.bindValue(FlinkStateBinder.java:48)
 at org.apache.flink.statefun.sdk.state.StateBinder.bind(StateBinder.java:30) 
at 
org.apache.flink.statefun.flink.core.state.PersistedStates.findReflectivelyAndBind(PersistedStates.java:46)
 at 
org.apache.flink.statefun.flink.state.processor.operator.StateBootstrapFunctionRegistry.bindState(StateBootstrapFunctionRegistry.java:120)
 at 
org.apache.flink.statefun.flink.state.processor.operator.StateBootstrapFunctionRegistry.initialize(StateBootstrapFunctionRegistry.java:103)
 at 
org.apache.flink.statefun.flink.state.processor.operator.StateBootstrapper.(StateBootstrapper.java:39)
 at 
org.apache.flink.statefun.flink.state.processor.operator.FunctionsStateBootstrapOperator.initializeState(FunctionsStateBootstrapOperator.java:67)
 at 
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:106)
 at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:258)
 at 
org.apache.flink.state.api.output.BoundedStreamTask.init(BoundedStreamTask.java:85)
 at 
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:457)
 at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:522) 
at 
org.apache.flink.state.api.output.BoundedOneInputStreamTaskRunner.mapPartition(BoundedOneInputStreamTaskRunner.java:76)
 at 
org.apache.flink.runtime.operators.MapPartitionDriver.run(MapPartitionDriver.java:103)
 at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:504) at 
org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:369) at 
org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) at 
org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) at 
java.lang.Thread.run(Thread.java:748){code}
 

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] 1.12 feature freeze date += one week ?

[Yuan Mei]

Hey Stephan,

Thanks for bringing this up!

+1 to one week extension for 1.12 feature freeze.

Things I am working on are under review and one week would definitely ease
my (and reviewers') life a lot.

Best,

Yuan

On Thu, Oct 29, 2020 at 2:13 AM Stephan Ewen  wrote:

> Hi all!
>
> We are approaching the feature freeze date for the 1.12 release that was
> discussed earlier.
>
> From my side and the developments I am involved with, we are close and in
> good shape, but could really use one more week to round things off. It
> would help both the code quality and our mental health a lot :-)
>
> From some personal conversations I heard at least from some other
> committers a similar sentiment.
>
> My proposal would hence be to set the 1.12 cutoff date to the weekend of
> the 7th/8th and fork the release-1.12 branch on Monday Nov. 9th.
>
> What do you think?
>
> Best,
> Stephan
>

Re: [DISCUSS] Remove flink-connector-filesystem module.

[Jingsong Li]

+1 to remove the Bucketing Sink.

Thanks for the effort on ORC and `HadoopPathBasedBulkFormatBuilder`, I
think it's safe to get rid of the old Bucketing API with them.

Best,
Jingsong

On Thu, Oct 29, 2020 at 3:06 AM Kostas Kloudas  wrote:

> Thanks for the discussion!
>
> From this thread I do not see any objection with moving forward with
> removing the sink.
> Given this I will open a voting thread tomorrow.
>
> Cheers,
> Kostas
>
> On Wed, Oct 28, 2020 at 6:50 PM Stephan Ewen  wrote:
> >
> > +1 to remove the Bucketing Sink.
> >
> > It has been very common in the past to remove code that was deprecated
> for multiple releases in favor of reducing baggage.
> > Also in cases that had no perfect drop-in replacement, but needed users
> to forward fit the code.
> > I am not sure I understand why this case is so different.
> >
> > Why the Bucketing Sink should be thrown out, in my opinion:
> >
> > The Bucketing sink makes it easier for users to add general Hadoop
> writes.
> > But the price is that it easily leads to dataloss, because it assumes
> flush()/sync() work reliably on Hadoop relicably, which they don't (HDFS
> works somewhat, S3 works not at all).
> > I think the Bucketing sink is a trap for users, that's why it was
> deprecated long ago.
> >
> > The StreamingFileSink covers the majority of cases from the Bucketing
> Sink.
> > It does have some friction when adding/wrapping some general Hadoop
> writers. Parts will be solved with the transactional sink work.
> > If something is missing and blocking users, we can prioritize adding it
> to the Streaming File Sink. Also that is something we did before and it
> helped being pragmatic with moving forward, rather than being held back by
> "maybe there is something we don't know".
> >
> >
> >
> >
> > On Wed, Oct 28, 2020 at 12:36 PM Chesnay Schepler 
> wrote:
> >>
> >> Then we can't remove it, because there is no way for us to ascertain
> >> whether anyone is still using it.
> >>
> >> Sure, the user ML is the best we got, but you can't argue that we don't
> >> want any users to be affected and then use an imperfect mean to find
> users.
> >> If you are fine with relying on the user ML, then you _are_ fine with
> >> removing it at the cost of friction for some users.
> >>
> >> To be clear, I, personally, don't have a problem with removing it (we
> >> have removed other connectors in the past that did not have a migration
> >> plan), I just reject he argumentation.
> >>
> >> On 10/28/2020 12:21 PM, Kostas Kloudas wrote:
> >> > No, I do not think that "we are fine with removing it at the cost of
> >> > friction for some users".
> >> >
> >> > I believe that this can be another discussion that we should have as
> >> > soon as we establish that someone is actually using it. The point I am
> >> > trying to make is that if no user is using it, we should remove it and
> >> > not leave unmaintained code around.
> >> >
> >> > On Wed, Oct 28, 2020 at 12:11 PM Chesnay Schepler 
> wrote:
> >> >> The alternative could also be to use a different argument than "no
> one
> >> >> uses it", e.g., we are fine with removing it at the cost of friction
> for
> >> >> some users because there are better alternatives.
> >> >>
> >> >> On 10/28/2020 10:46 AM, Kostas Kloudas wrote:
> >> >>> I think that the mailing lists is the best we can do and I would say
> >> >>> that they seem to be working pretty well (e.g. the recent Mesos
> >> >>> discussion).
> >> >>> Of course they are not perfect but the alternative would be to never
> >> >>> remove anything user facing until the next major release, which I
> find
> >> >>> pretty strict.
> >> >>>
> >> >>> On Wed, Oct 28, 2020 at 10:04 AM Chesnay Schepler <
> ches...@apache.org> wrote:
> >>  If the conclusion is that we shouldn't remove it if _anyone_ is
> using
> >>  it, then we cannot remove it because the user ML obviously does not
> >>  reach all users.
> >> 
> >>  On 10/28/2020 9:28 AM, Kostas Kloudas wrote:
> >> > Hi all,
> >> >
> >> > I am bringing the up again to see if there are any users actively
> >> > using the BucketingSink.
> >> > So far, if I am not mistaken (and really sorry if I forgot
> anything),
> >> > it is only a discussion between devs about the potential problems
> of
> >> > removing it. I totally understand Chesnay's concern about not
> >> > providing compatibility with the StreamingFileSink (SFS) and if
> there
> >> > are any users, then we should not remove it without trying to
> find a
> >> > solution for them.
> >> >
> >> > But if there are no users then I would still propose to remove the
> >> > module, given that I am not aware of any efforts to provide
> >> > compatibility with the SFS any time soon.
> >> > The reasons for removing it also include the facts that we do not
> >> > actively maintain it and we do not add new features. As for
> potential
> >> > missing features in the SFS compared to the BucketingSink that was

Re: [DISCUSS] Releasing Apache Flink 1.11.3

[Jingsong Li]

+1 to backport the FLIP-27 adjustments to 1.11.x.

If possible, that would be great. Many people are looking forward to the
FLIP-27 interface, but they don't want to take the risk to upgrade to 1.12
(And wait 1.12). After all, 1.11 is a relatively stable version.

Best,
Jingsong

On Thu, Oct 29, 2020 at 1:24 AM Stephan Ewen  wrote:

> Thanks for starting this.
>
> +1 form my side to backport the FLIP-27 adjustments to 1.11.x.
>
> There were quite a few changes, and I think we need to cherry-pick them all
> to not get some inconsistent mix of changes and many merge conflicts.
> I made a list below of what we need to add to "release-1.11".
>
> * Core Source API Changes to backport (in REVERSE order)*
>
>   (Use: "git log
> flink-core/src/main/java/org/apache/flink/api/connector/source")
>
> commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> [FLINK-19492][core] Consolidate Source Events between Source API and Split
> Reader API
>
> commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
> [FLINK-19205][core] Add access to configuration and hostname in the
> SourceReaderContext
>
> commit 723e1790744ddc78d19c7c978442af1383f38d33
> [hotfix][core] Add to Source Enumerator convenience methods to assign
> single split
>
> * Connector Base Changes to Backport (in REVERSE order)*
>
>   (Use: "git log flink-connectors/flink-connector-base")
>
> commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
> [FLINK-19427][FLINK-19489][tests] Fix test conditions for
> 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
>
> commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
> [FLINK-19448][connector base] Explicitly check for un-expected condition
> that would leave an inconsistent state
>
> commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
> [FLINK-19492][core] Consolidate Source Events between Source API and Split
> Reader API
>
> commit c1ca7a4c7c21ec8868c14c43c559625b794c
> [refactor][tests] Move some source test utils from flink-connector-base to
> flink-core
>
> commit ee5c4c211c35c70d28252363bbc8400453609977
> [FLINK-19251][connectors] Avoid confusing queue handling in
> "SplitReader.handleSplitsChanges()"
>
> commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
> [FLINK-19250][connectors] Fix error propagation in connector base
> (SplitFetcherManager).
>
> commit 8fcca837c55a9216595ee4c03038b52747098dbb
> [hotfix][connectors] Improve JavaDocs for SingleThreadFetcherManager
>
> commit 4700bb5dde3303cbe98882f6beb7379425717b01
> [FLINK-19225][connectors] Various small improvements to SourceReaderBase
> (part 2)
>
> commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
> [FLINK-17393][connectors] (follow-up) Wakeup the SplitFetchers more
> elegantly.
>
> commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
> [hotfix][tests] Extend test coverage for FutureCompletingBlockingQueue
>
> commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
> [FLINK-19245][connectors] Set default capacity for
> FutureCompletingBlockingQueue.
>
> commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
> [FLINK-19223][connectors] Simplify Availability Future Model in Base
> Connector
>
> commit 511857049ba30c8ff0ee56da551fa4a479dc583e
> [FLINK-18128][connectors] Ensure idle split fetchers lead to availability
> notifications.
>
> commit a8206467af0830dcb89623ea068b5ca3b3450c92
> [refactor][core] Eagerly initialize the FetchTask to support proper unit
> testing
>
> commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
> [hotfix][tests] Move constants in SplitFetcherTest relevant to only one
> test into test method
>
> commit d7625760a75a508bf05bcddc380bb4d62ee1743e
> [FLINK-19225][connectors] Various small improvements to SourceReaderBase
>
> commit a5b0d3297748c1be47ad579a88f24df2255a8df1
> [FLINK-17393][connectors] Wakeup the SplitFetchers more elegantly.
>
> commit f42a3ebc3e81a034b7221a803c153636fef34903
> [FLINK-18680][connectors] Make connector base RecordsWithSplitIds more
> lightweight.
>
> commit e3d273de822b085183d09b275a445879ff94b350
> [FLINK-19162][connectors] Add 'recycle()' to the RecordsWithSplitIds to
> support reuse of heavy objects.
>
> commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
> [hotfix][testing] Add a set of parameterizable testing mocks for the Split
> Reader API
>
> commit 930a07438be1185388d7150640f294dfe2a5d378
> [hotfix][connectors] Add RequestSplitEvent to 'flink-connector-base'
>
> commit e945ce8a933bc378844782f784ca473c767ca159
> [refactor][connectors] Backport of the connector-base exception handling
> from the Kafka Connector Pull Request
>
>
> On Wed, Oct 28, 2020 at 5:49 PM Steven Wu  wrote:
>
> > I would love to see this FLIP-27 source interface improvement [1] made to
> > 1.11.3.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-19698
> >
> > On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai <
> tzuli...@apache.org>
> > wrote:
> >
> > > Thanks for the replies so far!
> > >
> > > Just to provide a brief update on the status of blockers for 1.11.3 so
> > far:
> > >
> > >
> > > *PR opened, pending reviewer*- [FLINK-19717] 

[jira] [Created] (FLINK-19865) YARN tests failed with "java.lang.NumberFormatException: For input string: "${env:MAX_LOG_FILE_NUMBER}" java.lang.NumberFormatException: For input string: "${env:MAX_LOG

[Dian Fu (Jira)]

Dian Fu created FLINK-19865:
---

 Summary: YARN tests failed with "java.lang.NumberFormatException: 
For input string: "${env:MAX_LOG_FILE_NUMBER}" java.lang.NumberFormatException: 
For input string: "${env:MAX_LOG_FILE_NUMBER}""
 Key: FLINK-19865
 URL: https://issues.apache.org/jira/browse/FLINK-19865
 Project: Flink
  Issue Type: Bug
  Components: Deployment / YARN
Affects Versions: 1.12.0
Reporter: Dian Fu


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=8541=logs=245e1f2e-ba5b-5570-d689-25ae21e5302f=e7f339b2-a7c3-57d9-00af-3712d4b15354

{code}
2020-10-28T22:58:39.4927767Z 2020-10-28 22:57:33,866 main ERROR Could not 
create plugin of type class 
org.apache.logging.log4j.core.appender.rolling.DefaultRolloverStrategy for 
element DefaultRolloverStrategy: java.lang.NumberFormatException: For input 
string: "${env:MAX_LOG_FILE_NUMBER}" java.lang.NumberFormatException: For input 
string: "${env:MAX_LOG_FILE_NUMBER}"
2020-10-28T22:58:39.4929252Zat 
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
2020-10-28T22:58:39.4929823Zat java.lang.Integer.parseInt(Integer.java:569)
2020-10-28T22:58:39.4930327Zat java.lang.Integer.parseInt(Integer.java:615)
2020-10-28T22:58:39.4931047Zat 
org.apache.logging.log4j.core.appender.rolling.DefaultRolloverStrategy$Builder.build(DefaultRolloverStrategy.java:137)
2020-10-28T22:58:39.4931866Zat 
org.apache.logging.log4j.core.appender.rolling.DefaultRolloverStrategy$Builder.build(DefaultRolloverStrategy.java:90)
2020-10-28T22:58:39.4932720Zat 
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:122)
2020-10-28T22:58:39.4933446Zat 
org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
2020-10-28T22:58:39.4934275Zat 
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
2020-10-28T22:58:39.4935029Zat 
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
2020-10-28T22:58:39.4935837Zat 
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
2020-10-28T22:58:39.4936605Zat 
org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
2020-10-28T22:58:39.4937573Zat 
org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
2020-10-28T22:58:39.4938429Zat 
org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
2020-10-28T22:58:39.4939206Zat 
org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
2020-10-28T22:58:39.4939885Zat 
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
2020-10-28T22:58:39.4940490Zat 
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
2020-10-28T22:58:39.4941087Zat 
org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
2020-10-28T22:58:39.4941733Zat 
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
2020-10-28T22:58:39.4942534Zat 
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
2020-10-28T22:58:39.4943154Zat 
org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
2020-10-28T22:58:39.4943820Zat 
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
2020-10-28T22:58:39.4944540Zat 
org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
2020-10-28T22:58:39.4945199Zat 
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
2020-10-28T22:58:39.4945858Zat 
org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
2020-10-28T22:58:39.4946426Zat 
org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
2020-10-28T22:58:39.4946965Zat 
org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
2020-10-28T22:58:39.4947698Zat 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.(ClusterEntrypoint.java:108)
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19864) TwoInputStreamTaskTest.testWatermarkMetrics failed with "expected:<1> but was:<-9223372036854775808>"

[Dian Fu (Jira)]

Dian Fu created FLINK-19864:
---

 Summary: TwoInputStreamTaskTest.testWatermarkMetrics failed with 
"expected:<1> but was:<-9223372036854775808>"
 Key: FLINK-19864
 URL: https://issues.apache.org/jira/browse/FLINK-19864
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Metrics
Affects Versions: 1.12.0
Reporter: Dian Fu


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=8541=logs=77a9d8e1-d610-59b3-fc2a-4766541e0e33=7c61167f-30b3-5893-cc38-a9e3d057e392

{code}
2020-10-28T22:40:44.2528420Z [ERROR] 
testWatermarkMetrics(org.apache.flink.streaming.runtime.tasks.TwoInputStreamTaskTest)
 Time elapsed: 1.528 s <<< FAILURE! 2020-10-28T22:40:44.2529225Z 
java.lang.AssertionError: expected:<1> but was:<-9223372036854775808> 
2020-10-28T22:40:44.2541228Z at org.junit.Assert.fail(Assert.java:88) 
2020-10-28T22:40:44.2542157Z at org.junit.Assert.failNotEquals(Assert.java:834) 
2020-10-28T22:40:44.2542954Z at org.junit.Assert.assertEquals(Assert.java:645) 
2020-10-28T22:40:44.2543456Z at org.junit.Assert.assertEquals(Assert.java:631) 
2020-10-28T22:40:44.2544002Z at 
org.apache.flink.streaming.runtime.tasks.TwoInputStreamTaskTest.testWatermarkMetrics(TwoInputStreamTaskTest.java:540)


{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19863) SQLClientHBaseITCase.testHBase failed with "java.io.IOException: Process failed due to timeout"

[Dian Fu (Jira)]

Dian Fu created FLINK-19863:
---

 Summary: SQLClientHBaseITCase.testHBase failed with 
"java.io.IOException: Process failed due to timeout"
 Key: FLINK-19863
 URL: https://issues.apache.org/jira/browse/FLINK-19863
 Project: Flink
  Issue Type: Bug
  Components: Connectors / HBase
Affects Versions: 1.12.0
Reporter: Dian Fu


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=8541=logs=91bf6583-3fb2-592f-e4d4-d79d79c3230a=3425d8ba-5f03-540a-c64b-51b8481bf7d6

{code}
00:50:02,589 [main] INFO  
org.apache.flink.tests.util.flink.FlinkDistribution  [] - Stopping 
Flink cluster.
00:50:04,106 [main] INFO  
org.apache.flink.tests.util.flink.FlinkDistribution  [] - Stopping 
Flink cluster.
00:50:04,741 [main] INFO  
org.apache.flink.tests.util.flink.LocalStandaloneFlinkResource [] - Backed up 
logs to 
/home/vsts/work/1/s/flink-end-to-end-tests/artifacts/flink-b3924665-1ac9-4309-8255-20f0dc94e7b9.
00:50:04,788 [main] INFO  
org.apache.flink.tests.util.hbase.LocalStandaloneHBaseResource [] - Stopping 
HBase Cluster
00:50:16,243 [main] ERROR 
org.apache.flink.tests.util.hbase.SQLClientHBaseITCase   [] - 

Test testHBase[0: 
hbase-version:1.4.3](org.apache.flink.tests.util.hbase.SQLClientHBaseITCase) 
failed with:
java.io.IOException: Process failed due to timeout.
at 
org.apache.flink.tests.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlocking(AutoClosableProcess.java:130)
at 
org.apache.flink.tests.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlocking(AutoClosableProcess.java:108)
at 
org.apache.flink.tests.util.flink.FlinkDistribution.submitSQLJob(FlinkDistribution.java:221)
at 
org.apache.flink.tests.util.flink.LocalStandaloneFlinkResource$StandaloneClusterController.submitSQLJob(LocalStandaloneFlinkResource.java:196)
at 
org.apache.flink.tests.util.hbase.SQLClientHBaseITCase.executeSqlStatements(SQLClientHBaseITCase.java:215)
at 
org.apache.flink.tests.util.hbase.SQLClientHBaseITCase.testHBase(SQLClientHBaseITCase.java:152)
{code}





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Remove flink-connector-filesystem module.

[Kostas Kloudas]

Thanks for the discussion!

>From this thread I do not see any objection with moving forward with
removing the sink.
Given this I will open a voting thread tomorrow.

Cheers,
Kostas

On Wed, Oct 28, 2020 at 6:50 PM Stephan Ewen  wrote:
>
> +1 to remove the Bucketing Sink.
>
> It has been very common in the past to remove code that was deprecated for 
> multiple releases in favor of reducing baggage.
> Also in cases that had no perfect drop-in replacement, but needed users to 
> forward fit the code.
> I am not sure I understand why this case is so different.
>
> Why the Bucketing Sink should be thrown out, in my opinion:
>
> The Bucketing sink makes it easier for users to add general Hadoop writes.
> But the price is that it easily leads to dataloss, because it assumes 
> flush()/sync() work reliably on Hadoop relicably, which they don't (HDFS 
> works somewhat, S3 works not at all).
> I think the Bucketing sink is a trap for users, that's why it was deprecated 
> long ago.
>
> The StreamingFileSink covers the majority of cases from the Bucketing Sink.
> It does have some friction when adding/wrapping some general Hadoop writers. 
> Parts will be solved with the transactional sink work.
> If something is missing and blocking users, we can prioritize adding it to 
> the Streaming File Sink. Also that is something we did before and it helped 
> being pragmatic with moving forward, rather than being held back by "maybe 
> there is something we don't know".
>
>
>
>
> On Wed, Oct 28, 2020 at 12:36 PM Chesnay Schepler  wrote:
>>
>> Then we can't remove it, because there is no way for us to ascertain
>> whether anyone is still using it.
>>
>> Sure, the user ML is the best we got, but you can't argue that we don't
>> want any users to be affected and then use an imperfect mean to find users.
>> If you are fine with relying on the user ML, then you _are_ fine with
>> removing it at the cost of friction for some users.
>>
>> To be clear, I, personally, don't have a problem with removing it (we
>> have removed other connectors in the past that did not have a migration
>> plan), I just reject he argumentation.
>>
>> On 10/28/2020 12:21 PM, Kostas Kloudas wrote:
>> > No, I do not think that "we are fine with removing it at the cost of
>> > friction for some users".
>> >
>> > I believe that this can be another discussion that we should have as
>> > soon as we establish that someone is actually using it. The point I am
>> > trying to make is that if no user is using it, we should remove it and
>> > not leave unmaintained code around.
>> >
>> > On Wed, Oct 28, 2020 at 12:11 PM Chesnay Schepler  
>> > wrote:
>> >> The alternative could also be to use a different argument than "no one
>> >> uses it", e.g., we are fine with removing it at the cost of friction for
>> >> some users because there are better alternatives.
>> >>
>> >> On 10/28/2020 10:46 AM, Kostas Kloudas wrote:
>> >>> I think that the mailing lists is the best we can do and I would say
>> >>> that they seem to be working pretty well (e.g. the recent Mesos
>> >>> discussion).
>> >>> Of course they are not perfect but the alternative would be to never
>> >>> remove anything user facing until the next major release, which I find
>> >>> pretty strict.
>> >>>
>> >>> On Wed, Oct 28, 2020 at 10:04 AM Chesnay Schepler  
>> >>> wrote:
>>  If the conclusion is that we shouldn't remove it if _anyone_ is using
>>  it, then we cannot remove it because the user ML obviously does not
>>  reach all users.
>> 
>>  On 10/28/2020 9:28 AM, Kostas Kloudas wrote:
>> > Hi all,
>> >
>> > I am bringing the up again to see if there are any users actively
>> > using the BucketingSink.
>> > So far, if I am not mistaken (and really sorry if I forgot anything),
>> > it is only a discussion between devs about the potential problems of
>> > removing it. I totally understand Chesnay's concern about not
>> > providing compatibility with the StreamingFileSink (SFS) and if there
>> > are any users, then we should not remove it without trying to find a
>> > solution for them.
>> >
>> > But if there are no users then I would still propose to remove the
>> > module, given that I am not aware of any efforts to provide
>> > compatibility with the SFS any time soon.
>> > The reasons for removing it also include the facts that we do not
>> > actively maintain it and we do not add new features. As for potential
>> > missing features in the SFS compared to the BucketingSink that was
>> > mentioned before, I am not aware of any fundamental limitations and
>> > even if there are, I would assume that the solution is not to direct
>> > the users to a deprecated sink but rather try to increase the
>> > functionality of the actively maintained one.
>> >
>> > Please keep in mind that the BucketingSink is deprecated since FLINK
>> > 1.9 and there is a new File Sink that is coming as part of 

[jira] [Created] (FLINK-19862) DeclarativeSlotManager#suspend can fail with NPE

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-19862:


 Summary: DeclarativeSlotManager#suspend can fail with NPE
 Key: FLINK-19862
 URL: https://issues.apache.org/jira/browse/FLINK-19862
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.12.0


{{DeclarativeSlotManager#suspend}} can fail with an NPE if a suspended slot 
manager is subsequently closed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[DISCUSS] 1.12 feature freeze date += one week ?

[Stephan Ewen]

Hi all!

We are approaching the feature freeze date for the 1.12 release that was
discussed earlier.

>From my side and the developments I am involved with, we are close and in
good shape, but could really use one more week to round things off. It
would help both the code quality and our mental health a lot :-)

>From some personal conversations I heard at least from some other
committers a similar sentiment.

My proposal would hence be to set the 1.12 cutoff date to the weekend of
the 7th/8th and fork the release-1.12 branch on Monday Nov. 9th.

What do you think?

Best,
Stephan

Is there any way to change operator level parallelism dynamically?

[DEEP NARAYAN Singh]

Hi Team,

I just want quick help here. How to achieve the dynamic nature of
operator level parallelism for the flink job running in AWS EMR cluster
during runtime to avoid downtime and backpressure based on the incoming
load. As I am very new to flink and currently using flink 1.9 version.Is
there any way to solve this problem during runtime?.

Any lead will be highly appreciated.

Thanks & Regards,
-Deep

Re: [DISCUSS] Remove flink-connector-filesystem module.

[Stephan Ewen]

+1 to remove the Bucketing Sink.

It has been very common in the past to remove code that was deprecated for
multiple releases in favor of reducing baggage.
Also in cases that had no perfect drop-in replacement, but needed users to
forward fit the code.
I am not sure I understand why this case is so different.

Why the Bucketing Sink should be thrown out, in my opinion:

The Bucketing sink makes it easier for users to add general Hadoop writes.
But the price is that it easily leads to dataloss, because it assumes
flush()/sync() work reliably on Hadoop relicably, which they don't (HDFS
works somewhat, S3 works not at all).
I think the Bucketing sink is a trap for users, that's why it was
deprecated long ago.

The StreamingFileSink covers the majority of cases from the Bucketing Sink.
It does have some friction when adding/wrapping some general Hadoop
writers. Parts will be solved with the transactional sink work.
If something is missing and blocking users, we can prioritize adding it to
the Streaming File Sink. Also that is something we did before and it helped
being pragmatic with moving forward, rather than being held back by "maybe
there is something we don't know".




On Wed, Oct 28, 2020 at 12:36 PM Chesnay Schepler 
wrote:

> Then we can't remove it, because there is no way for us to ascertain
> whether anyone is still using it.
>
> Sure, the user ML is the best we got, but you can't argue that we don't
> want any users to be affected and then use an imperfect mean to find users.
> If you are fine with relying on the user ML, then you _are_ fine with
> removing it at the cost of friction for some users.
>
> To be clear, I, personally, don't have a problem with removing it (we
> have removed other connectors in the past that did not have a migration
> plan), I just reject he argumentation.
>
> On 10/28/2020 12:21 PM, Kostas Kloudas wrote:
> > No, I do not think that "we are fine with removing it at the cost of
> > friction for some users".
> >
> > I believe that this can be another discussion that we should have as
> > soon as we establish that someone is actually using it. The point I am
> > trying to make is that if no user is using it, we should remove it and
> > not leave unmaintained code around.
> >
> > On Wed, Oct 28, 2020 at 12:11 PM Chesnay Schepler 
> wrote:
> >> The alternative could also be to use a different argument than "no one
> >> uses it", e.g., we are fine with removing it at the cost of friction for
> >> some users because there are better alternatives.
> >>
> >> On 10/28/2020 10:46 AM, Kostas Kloudas wrote:
> >>> I think that the mailing lists is the best we can do and I would say
> >>> that they seem to be working pretty well (e.g. the recent Mesos
> >>> discussion).
> >>> Of course they are not perfect but the alternative would be to never
> >>> remove anything user facing until the next major release, which I find
> >>> pretty strict.
> >>>
> >>> On Wed, Oct 28, 2020 at 10:04 AM Chesnay Schepler 
> wrote:
>  If the conclusion is that we shouldn't remove it if _anyone_ is using
>  it, then we cannot remove it because the user ML obviously does not
>  reach all users.
> 
>  On 10/28/2020 9:28 AM, Kostas Kloudas wrote:
> > Hi all,
> >
> > I am bringing the up again to see if there are any users actively
> > using the BucketingSink.
> > So far, if I am not mistaken (and really sorry if I forgot anything),
> > it is only a discussion between devs about the potential problems of
> > removing it. I totally understand Chesnay's concern about not
> > providing compatibility with the StreamingFileSink (SFS) and if there
> > are any users, then we should not remove it without trying to find a
> > solution for them.
> >
> > But if there are no users then I would still propose to remove the
> > module, given that I am not aware of any efforts to provide
> > compatibility with the SFS any time soon.
> > The reasons for removing it also include the facts that we do not
> > actively maintain it and we do not add new features. As for potential
> > missing features in the SFS compared to the BucketingSink that was
> > mentioned before, I am not aware of any fundamental limitations and
> > even if there are, I would assume that the solution is not to direct
> > the users to a deprecated sink but rather try to increase the
> > functionality of the actively maintained one.
> >
> > Please keep in mind that the BucketingSink is deprecated since FLINK
> > 1.9 and there is a new File Sink that is coming as part of FLIP-143
> > [1].
> > Again, if there are any active users who cannot migrate easily, then
> > we cannot remove it before trying to provide a smooth migration path.
> >
> > Thanks,
> > Kostas
> >
> > [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API
> >
> > On Fri, Oct 16, 2020 at 4:36 PM Chesnay 

Re: [DISCUSS] Releasing Apache Flink 1.11.3

[Stephan Ewen]

Thanks for starting this.

+1 form my side to backport the FLIP-27 adjustments to 1.11.x.

There were quite a few changes, and I think we need to cherry-pick them all
to not get some inconsistent mix of changes and many merge conflicts.
I made a list below of what we need to add to "release-1.11".

* Core Source API Changes to backport (in REVERSE order)*

  (Use: "git log
flink-core/src/main/java/org/apache/flink/api/connector/source")

commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
[FLINK-19492][core] Consolidate Source Events between Source API and Split
Reader API

commit 428fbbad85379f71a9fbf1a1ab3e3dcfa9006436
[FLINK-19205][core] Add access to configuration and hostname in the
SourceReaderContext

commit 723e1790744ddc78d19c7c978442af1383f38d33
[hotfix][core] Add to Source Enumerator convenience methods to assign
single split

* Connector Base Changes to Backport (in REVERSE order)*

  (Use: "git log flink-connectors/flink-connector-base")

commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d
[FLINK-19427][FLINK-19489][tests] Fix test conditions for
'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'

commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1
[FLINK-19448][connector base] Explicitly check for un-expected condition
that would leave an inconsistent state

commit 162c072e9265a7b6dd9d6f5459eb7974091c4c4e
[FLINK-19492][core] Consolidate Source Events between Source API and Split
Reader API

commit c1ca7a4c7c21ec8868c14c43c559625b794c
[refactor][tests] Move some source test utils from flink-connector-base to
flink-core

commit ee5c4c211c35c70d28252363bbc8400453609977
[FLINK-19251][connectors] Avoid confusing queue handling in
"SplitReader.handleSplitsChanges()"

commit 5abef56b2bf85bcac786f6b16b6899b6cced7176
[FLINK-19250][connectors] Fix error propagation in connector base
(SplitFetcherManager).

commit 8fcca837c55a9216595ee4c03038b52747098dbb
[hotfix][connectors] Improve JavaDocs for SingleThreadFetcherManager

commit 4700bb5dde3303cbe98882f6beb7379425717b01
[FLINK-19225][connectors] Various small improvements to SourceReaderBase
(part 2)

commit 12261c6b7ed6478a9b9f6a69cb58246b83cab9b7
[FLINK-17393][connectors] (follow-up) Wakeup the SplitFetchers more
elegantly.

commit c60aaff0249bfd6b5871b7f82e03efc487a54d6b
[hotfix][tests] Extend test coverage for FutureCompletingBlockingQueue

commit cef8a587d7fd2fe64cc644da5ed095d82e46f631
[FLINK-19245][connectors] Set default capacity for
FutureCompletingBlockingQueue.

commit 4ea95782b4c6a2538153d4d16ad3f4839c7de0fb
[FLINK-19223][connectors] Simplify Availability Future Model in Base
Connector

commit 511857049ba30c8ff0ee56da551fa4a479dc583e
[FLINK-18128][connectors] Ensure idle split fetchers lead to availability
notifications.

commit a8206467af0830dcb89623ea068b5ca3b3450c92
[refactor][core] Eagerly initialize the FetchTask to support proper unit
testing

commit 3b2f54bcb437f98e6137c904045cc51072b5c06b
[hotfix][tests] Move constants in SplitFetcherTest relevant to only one
test into test method

commit d7625760a75a508bf05bcddc380bb4d62ee1743e
[FLINK-19225][connectors] Various small improvements to SourceReaderBase

commit a5b0d3297748c1be47ad579a88f24df2255a8df1
[FLINK-17393][connectors] Wakeup the SplitFetchers more elegantly.

commit f42a3ebc3e81a034b7221a803c153636fef34903
[FLINK-18680][connectors] Make connector base RecordsWithSplitIds more
lightweight.

commit e3d273de822b085183d09b275a445879ff94b350
[FLINK-19162][connectors] Add 'recycle()' to the RecordsWithSplitIds to
support reuse of heavy objects.

commit 8ebc464c2520453a70001cd712abc8dee6ee89e0
[hotfix][testing] Add a set of parameterizable testing mocks for the Split
Reader API

commit 930a07438be1185388d7150640f294dfe2a5d378
[hotfix][connectors] Add RequestSplitEvent to 'flink-connector-base'

commit e945ce8a933bc378844782f784ca473c767ca159
[refactor][connectors] Backport of the connector-base exception handling
from the Kafka Connector Pull Request


On Wed, Oct 28, 2020 at 5:49 PM Steven Wu  wrote:

> I would love to see this FLIP-27 source interface improvement [1] made to
> 1.11.3.
>
> [1] https://issues.apache.org/jira/browse/FLINK-19698
>
> On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai 
> wrote:
>
> > Thanks for the replies so far!
> >
> > Just to provide a brief update on the status of blockers for 1.11.3 so
> far:
> >
> >
> > *PR opened, pending reviewer*- [FLINK-19717] SourceReaderBase.pollNext
> may
> > return END_OF_INPUT if SplitReader.fetch throws (
> > https://github.com/apache/flink/pull/13776)
> >
> > *PR opened, reviewed + close to being merged*
> > - [FLINK-19741] Timer service should skip restoring from raw keyed stream
> > if it isn't the writer (https://github.com/apache/flink/pull/13761)
> > - [FLINK-19748] Raw keyed stream key group iterator should be skipping
> > unwritten key groups (https://github.com/apache/flink/pull/13772)
> >
> > *Merged*
> > - [FLINK-19154] Application mode deletes HA data in case of suspended
> > ZooKeeper connection
> > 

[jira] [Created] (FLINK-19861) Improve the document of now() function

[hailong wang (Jira)]

hailong wang created FLINK-19861:


 Summary: Improve the document of now() function
 Key: FLINK-19861
 URL: https://issues.apache.org/jira/browse/FLINK-19861
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Affects Versions: 1.11.0
Reporter: hailong wang
 Fix For: 1.12.0


`NOW()` function is deterministic in batch mode, but not deterministic in 
streaming.

 We should Improve the description of `NOW()` in 
/dev/table/functions/systemFunctions.md.

See the email thread, 

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-understand-NOW-in-SQL-when-using-Table-amp-SQL-API-to-develop-a-streaming-app-tc38881.html#none



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Releasing Apache Flink 1.11.3

[Steven Wu]

I would love to see this FLIP-27 source interface improvement [1] made to
1.11.3.

[1] https://issues.apache.org/jira/browse/FLINK-19698

On Wed, Oct 28, 2020 at 12:32 AM Tzu-Li (Gordon) Tai 
wrote:

> Thanks for the replies so far!
>
> Just to provide a brief update on the status of blockers for 1.11.3 so far:
>
>
> *PR opened, pending reviewer*- [FLINK-19717] SourceReaderBase.pollNext may
> return END_OF_INPUT if SplitReader.fetch throws (
> https://github.com/apache/flink/pull/13776)
>
> *PR opened, reviewed + close to being merged*
> - [FLINK-19741] Timer service should skip restoring from raw keyed stream
> if it isn't the writer (https://github.com/apache/flink/pull/13761)
> - [FLINK-19748] Raw keyed stream key group iterator should be skipping
> unwritten key groups (https://github.com/apache/flink/pull/13772)
>
> *Merged*
> - [FLINK-19154] Application mode deletes HA data in case of suspended
> ZooKeeper connection
> - [FLINK-19569] Upgrade ICU4J to 67.1+
>
> Right now as it seems, progress is mainly blocked on a reviewer for
> FLINK-19717.
> Meanwhile, Xintong is keeping an eye on test instabilities [1] to see if
> there are any fixes that should be applied to `release-1.11`.
>
> This is also a reminder, that if there are other blockers that we need to
> be aware of, or a need to re-establish estimated time for getting fixes in
> and delay the RC for 1.11.3, please do let us know!
>
> Cheers,
> Gordon
>
> [1]
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability
>
> On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas 
> wrote:
>
> > +1 for releasing Flink 1.11.3 as it contains a number of important
> > fixes and thanks Gordon and Xintong for volunteering.
> >
> > Cheers,
> > Kostas
> >
> > On Mon, Oct 26, 2020 at 4:37 AM Yu Li  wrote:
> > >
> > > +1 for releasing Flink 1.11.3, and thanks Gordon and Xintong for
> > > volunteering as our release managers.
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Mon, 26 Oct 2020 at 09:45, Xintong Song 
> > wrote:
> > >
> > > > Thanks Gordan for starting this discussion.
> > > > My pleasure to help with the release process.
> > > >
> > > > Thank you~
> > > >
> > > > Xintong Song
> > > >
> > > >
> > > >
> > > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann  >
> > > > wrote:
> > > >
> > > > > Thanks for starting this discussion Gordon. There are over 100
> issues
> > > > > which are fixed for 1.11.3. Hence +1 for a soonish 1.11.3 release.
> > Thanks
> > > > > for volunteering as our release managers Gordon and Xintong!
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <
> > tzuli...@apache.org
> > > > >
> > > > > wrote:
> > > > >
> > > > >> Hi,
> > > > >>
> > > > >> Xintong and I would like to start a discussion for releasing Flink
> > > > 1.11.3
> > > > >> soon.
> > > > >>
> > > > >> It seems like we already have a few pressing issues that needs to
> be
> > > > >> included in a new hotfix release:
> > > > >>
> > > > >>- Heap-based timers’ restore behaviour is causing a critical
> > recovery
> > > > >>issue for StateFun [1] [2] [3].
> > > > >>- There are several robustness issues for the FLIP-27 new
> source
> > API,
> > > > >>such as [4]. We already have some users using the FLIP-27 API
> > with
> > > > >> 1.11.x,
> > > > >>so it would be important to get those fixes in for 1.11.x as
> > well.
> > > > >>
> > > > >> Apart from the issues that are already marked as blocker for
> 1.11.3
> > in
> > > > our
> > > > >> JIRA [5], please let us know in this thread if there is already
> > ongoing
> > > > >> work for other important fixes that we should try to include.
> > > > >>
> > > > >> Xintong and I would like to volunteer for managing this release,
> and
> > > > will
> > > > >> try to communicate the priority of pending blockers over the next
> > few
> > > > >> days.
> > > > >> Since the aforementioned issues are quite critical, we’d like to
> aim
> > > > >> for a *feature
> > > > >> freeze by the end of next week (Oct. 30th)* and start the release
> > voting
> > > > >> process the week after.
> > > > >> If that is too short of a notice and you might need more time,
> > please
> > > > let
> > > > >> us know!
> > > > >>
> > > > >> Cheers,
> > > > >> Gordon
> > > > >>
> > > > >> [1] https://issues.apache.org/jira/browse/FLINK-19692
> > > > >> [2] https://issues.apache.org/jira/browse/FLINK-19741
> > > > >> [3] https://issues.apache.org/jira/browse/FLINK-19748
> > > > >> [4] https://issues.apache.org/jira/browse/FLINK-19717
> > > > >> [5]
> > > > >>
> > > > >>
> > > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > > > >>
> > > > >
> > > >
> >
>

[jira] [Created] (FLINK-19860) Consider skipping restart and traverse regions which are already being restarted in RestartPipelinedRegionFailoverStrategy

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-19860:
---

 Summary: Consider skipping restart and traverse regions which are 
already being restarted in RestartPipelinedRegionFailoverStrategy
 Key: FLINK-19860
 URL: https://issues.apache.org/jira/browse/FLINK-19860
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Reporter: Andrey Zagrebin


Original 
[discussion|https://github.com/apache/flink/pull/13749#pullrequestreview-516385846].
 Follow-up for FLINK-19712.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] NEW FLIP-104: Add More Metrics to JobManager

[Till Rohrmann]

+1

Cheers,
Till

On Wed, Oct 28, 2020 at 9:07 AM Kurt Young  wrote:

> +1
>
> Best,
> Kurt
>
>
> On Wed, Oct 28, 2020 at 2:44 PM Robert Metzger 
> wrote:
>
> > Thank you for your UI work Yadong!
> >
> > +1
> >
> > On Tue, Oct 27, 2020 at 6:33 PM Matthias Pohl 
> > wrote:
> >
> > > Thanks for restarting the vote, Yadong. I really like your UI
> proposals.
> > > +1 for adding the changes of FLIP-104.
> > >
> > > Matthias
> > >
> > > On Tue, Oct 27, 2020 at 10:29 AM Xintong Song 
> > > wrote:
> > >
> > > > Thanks for reviving this FLIP, Yandong.
> > > >
> > > > The proposed changes look good to me.
> > > > +1 for accepting this FLIP.
> > > >
> > > > Thank you~
> > > >
> > > > Xintong Song
> > > >
> > > >
> > > >
> > > > On Tue, Oct 27, 2020 at 4:02 PM Yadong Xie 
> > wrote:
> > > >
> > > > > Hi all
> > > > >
> > > > > I want to start a new vote for FLIP-104, which proposes to add more
> > > > metrics
> > > > > to the job manager in web UI.
> > > > >
> > > > > The new FLIP-104 was revisited and adapted following the old ML
> > > > discussion
> > > > > <
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-104-Add-More-Metrics-to-Jobmanager-td37901.html
> > > > > >
> > > > > .
> > > > >
> > > > > The vote will last for at least 72 hours, following the consensus
> > > voting.
> > > > >
> > > > >
> > > > > FLIP-104 wiki:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-104%3A+Add+More+Metrics+to+Jobmanager
> > > > >
> > > > >
> > > > > Discussion thread:
> > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-104-Add-More-Metrics-to-Jobmanager-td37901.html
> > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Yadong
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (FLINK-19859) Add documentation for the upsert-kafka connector

[Jark Wu (Jira)]

Jark Wu created FLINK-19859:
---

 Summary: Add documentation for the upsert-kafka connector
 Key: FLINK-19859
 URL: https://issues.apache.org/jira/browse/FLINK-19859
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Kafka, Documentation
Reporter: Jark Wu
 Fix For: 1.12.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19858) Add the new table factory for upsert-kafka connector

[Jark Wu (Jira)]

Jark Wu created FLINK-19858:
---

 Summary: Add the new table factory for upsert-kafka connector
 Key: FLINK-19858
 URL: https://issues.apache.org/jira/browse/FLINK-19858
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Kafka, Table SQL / Ecosystem
Reporter: Jark Wu
Assignee: Shengkai Fang
 Fix For: 1.12.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19857) FLIP-149: Introduce the upsert-kafka Connector

[Jark Wu (Jira)]

Jark Wu created FLINK-19857:
---

 Summary: FLIP-149: Introduce the upsert-kafka Connector
 Key: FLINK-19857
 URL: https://issues.apache.org/jira/browse/FLINK-19857
 Project: Flink
  Issue Type: New Feature
  Components: Connectors / Kafka, Table SQL / API, Table SQL / Ecosystem
Reporter: Jark Wu


This is the umbrella issue for FLIP-149: Introduce the upsert-kafka Connector

FLIP-149: 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-149%3A+Introduce+the+upsert-kafka+Connector



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE]FLIP-149: Introduce the upsert-kafka connector

[Shengkai Fang]

Hi all,

The vote has been opened for more than 72 hours. We have already received 5
votes (4 binding) and no -1 votes.  I am happy to announce the FLIP-149 is
approved.

* Jark (binding)
* Konstantin (binding)
* Jingsong (binding)
* Timo (binding)
* Danny (non-binding)

Thanks everyone for participating!

Best,
Shengkai

Danny Chan  于2020年10月27日周二 下午3:43写道：

> +1
>
> Timo Walther  于2020年10月23日周五 下午5:22写道：
>
> > +1
> >
> > Thanks,
> > Timo
> >
> > On 23.10.20 10:21, Jingsong Li wrote:
> > > +1
> > >
> > > On Fri, Oct 23, 2020 at 3:52 PM Konstantin Knauf 
> > wrote:
> > >
> > >> +1
> > >>
> > >> On Fri, Oct 23, 2020 at 9:36 AM Jark Wu  wrote:
> > >>
> > >>> +1
> > >>>
> > >>> On Fri, 23 Oct 2020 at 15:25, Shengkai Fang 
> wrote:
> > >>>
> >  Hi, all,
> > 
> >  I would like to start the vote for FLIP-149[1], which is discussed
> and
> >  reached a consensus in the discussion thread[2]. The vote will be
> open
> >  until 16:00(UTC+8) 28th Oct. (72h, exclude weekends), unless there
> is
> > >> an
> >  objection or not enough votes.
> > 
> >  [1]
> > 
> > 
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-149%3A+Introduce+the+upsert-kafka+Connector
> >  [2]
> > 
> > 
> > >>>
> > >>
> >
> https://lists.apache.org/thread.html/r83e3153377594276b2066e49e399ec05d127b58bb4ce0fde33309da2%40%3Cdev.flink.apache.org%3E
> > 
> > >>>
> > >>
> > >>
> > >> --
> > >>
> > >> Konstantin Knauf
> > >>
> > >> https://twitter.com/snntrable
> > >>
> > >> https://github.com/knaufk
> > >>
> > >
> > >
> >
> >
>

Re: [DISCUSS] Remove flink-connector-filesystem module.

[Chesnay Schepler]

Then we can't remove it, because there is no way for us to ascertain 
whether anyone is still using it.


Sure, the user ML is the best we got, but you can't argue that we don't 
want any users to be affected and then use an imperfect mean to find users.
If you are fine with relying on the user ML, then you _are_ fine with 
removing it at the cost of friction for some users.


To be clear, I, personally, don't have a problem with removing it (we 
have removed other connectors in the past that did not have a migration 
plan), I just reject he argumentation.


On 10/28/2020 12:21 PM, Kostas Kloudas wrote:

No, I do not think that "we are fine with removing it at the cost of
friction for some users".

I believe that this can be another discussion that we should have as
soon as we establish that someone is actually using it. The point I am
trying to make is that if no user is using it, we should remove it and
not leave unmaintained code around.

On Wed, Oct 28, 2020 at 12:11 PM Chesnay Schepler  wrote:

The alternative could also be to use a different argument than "no one
uses it", e.g., we are fine with removing it at the cost of friction for
some users because there are better alternatives.

On 10/28/2020 10:46 AM, Kostas Kloudas wrote:

I think that the mailing lists is the best we can do and I would say
that they seem to be working pretty well (e.g. the recent Mesos
discussion).
Of course they are not perfect but the alternative would be to never
remove anything user facing until the next major release, which I find
pretty strict.

On Wed, Oct 28, 2020 at 10:04 AM Chesnay Schepler  wrote:

If the conclusion is that we shouldn't remove it if _anyone_ is using
it, then we cannot remove it because the user ML obviously does not
reach all users.

On 10/28/2020 9:28 AM, Kostas Kloudas wrote:

Hi all,

I am bringing the up again to see if there are any users actively
using the BucketingSink.
So far, if I am not mistaken (and really sorry if I forgot anything),
it is only a discussion between devs about the potential problems of
removing it. I totally understand Chesnay's concern about not
providing compatibility with the StreamingFileSink (SFS) and if there
are any users, then we should not remove it without trying to find a
solution for them.

But if there are no users then I would still propose to remove the
module, given that I am not aware of any efforts to provide
compatibility with the SFS any time soon.
The reasons for removing it also include the facts that we do not
actively maintain it and we do not add new features. As for potential
missing features in the SFS compared to the BucketingSink that was
mentioned before, I am not aware of any fundamental limitations and
even if there are, I would assume that the solution is not to direct
the users to a deprecated sink but rather try to increase the
functionality of the actively maintained one.

Please keep in mind that the BucketingSink is deprecated since FLINK
1.9 and there is a new File Sink that is coming as part of FLIP-143
[1].
Again, if there are any active users who cannot migrate easily, then
we cannot remove it before trying to provide a smooth migration path.

Thanks,
Kostas

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API

On Fri, Oct 16, 2020 at 4:36 PM Chesnay Schepler  wrote:

@Seth: Earlier in this discussion it was said that the BucketingSink
would not be usable in 1.12 .

On 10/16/2020 4:25 PM, Seth Wiesman wrote:

+1 It has been deprecated for some time and the StreamingFileSink has
stabalized with a large number of formats and features.

Plus, the bucketing sink only implements a small number of stable
interfaces[1]. I would expect users to continue to use the bucketing sink
from the 1.11 release with future versions for some time.

Seth

https://github.com/apache/flink/blob/2ff3b771cbb091e1f43686dd8e176cea6d435501/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java#L170-L172

On Thu, Oct 15, 2020 at 2:57 PM Kostas Kloudas  wrote:


@Arvid Heise I also do not remember exactly what were all the
problems. The fact that we added some more bulk formats to the
streaming file sink definitely reduced the non-supported features. In
addition, the latest discussion I found on the topic was [1] and the
conclusion of that discussion seems to be to remove it.

Currently, I cannot find any obvious reason why keeping the
BucketingSink, apart from the fact that we do not have a migration
plan unfortunately. This is why I posted this to dev@ and user@.

Cheers,
Kostas

[1]
https://lists.apache.org/thread.html/r799be74658bc7e169238cc8c1e479e961a9e85ccea19089290940ff0%40%3Cdev.flink.apache.org%3E

On Wed, Oct 14, 2020 at 8:03 AM Arvid Heise  wrote:

I remember this conversation popping up a few times already and I'm in
general a big fan of removing BucketingSink.

However, until now there were a few features lacking in StreamingFileSink
that 

Re: [DISCUSS] Remove flink-connector-filesystem module.

[Kostas Kloudas]

No, I do not think that "we are fine with removing it at the cost of
friction for some users".

I believe that this can be another discussion that we should have as
soon as we establish that someone is actually using it. The point I am
trying to make is that if no user is using it, we should remove it and
not leave unmaintained code around.

On Wed, Oct 28, 2020 at 12:11 PM Chesnay Schepler  wrote:
>
> The alternative could also be to use a different argument than "no one
> uses it", e.g., we are fine with removing it at the cost of friction for
> some users because there are better alternatives.
>
> On 10/28/2020 10:46 AM, Kostas Kloudas wrote:
> > I think that the mailing lists is the best we can do and I would say
> > that they seem to be working pretty well (e.g. the recent Mesos
> > discussion).
> > Of course they are not perfect but the alternative would be to never
> > remove anything user facing until the next major release, which I find
> > pretty strict.
> >
> > On Wed, Oct 28, 2020 at 10:04 AM Chesnay Schepler  
> > wrote:
> >> If the conclusion is that we shouldn't remove it if _anyone_ is using
> >> it, then we cannot remove it because the user ML obviously does not
> >> reach all users.
> >>
> >> On 10/28/2020 9:28 AM, Kostas Kloudas wrote:
> >>> Hi all,
> >>>
> >>> I am bringing the up again to see if there are any users actively
> >>> using the BucketingSink.
> >>> So far, if I am not mistaken (and really sorry if I forgot anything),
> >>> it is only a discussion between devs about the potential problems of
> >>> removing it. I totally understand Chesnay's concern about not
> >>> providing compatibility with the StreamingFileSink (SFS) and if there
> >>> are any users, then we should not remove it without trying to find a
> >>> solution for them.
> >>>
> >>> But if there are no users then I would still propose to remove the
> >>> module, given that I am not aware of any efforts to provide
> >>> compatibility with the SFS any time soon.
> >>> The reasons for removing it also include the facts that we do not
> >>> actively maintain it and we do not add new features. As for potential
> >>> missing features in the SFS compared to the BucketingSink that was
> >>> mentioned before, I am not aware of any fundamental limitations and
> >>> even if there are, I would assume that the solution is not to direct
> >>> the users to a deprecated sink but rather try to increase the
> >>> functionality of the actively maintained one.
> >>>
> >>> Please keep in mind that the BucketingSink is deprecated since FLINK
> >>> 1.9 and there is a new File Sink that is coming as part of FLIP-143
> >>> [1].
> >>> Again, if there are any active users who cannot migrate easily, then
> >>> we cannot remove it before trying to provide a smooth migration path.
> >>>
> >>> Thanks,
> >>> Kostas
> >>>
> >>> [1] 
> >>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API
> >>>
> >>> On Fri, Oct 16, 2020 at 4:36 PM Chesnay Schepler  
> >>> wrote:
>  @Seth: Earlier in this discussion it was said that the BucketingSink
>  would not be usable in 1.12 .
> 
>  On 10/16/2020 4:25 PM, Seth Wiesman wrote:
> > +1 It has been deprecated for some time and the StreamingFileSink has
> > stabalized with a large number of formats and features.
> >
> > Plus, the bucketing sink only implements a small number of stable
> > interfaces[1]. I would expect users to continue to use the bucketing 
> > sink
> > from the 1.11 release with future versions for some time.
> >
> > Seth
> >
> > https://github.com/apache/flink/blob/2ff3b771cbb091e1f43686dd8e176cea6d435501/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java#L170-L172
> >
> > On Thu, Oct 15, 2020 at 2:57 PM Kostas Kloudas  
> > wrote:
> >
> >> @Arvid Heise I also do not remember exactly what were all the
> >> problems. The fact that we added some more bulk formats to the
> >> streaming file sink definitely reduced the non-supported features. In
> >> addition, the latest discussion I found on the topic was [1] and the
> >> conclusion of that discussion seems to be to remove it.
> >>
> >> Currently, I cannot find any obvious reason why keeping the
> >> BucketingSink, apart from the fact that we do not have a migration
> >> plan unfortunately. This is why I posted this to dev@ and user@.
> >>
> >> Cheers,
> >> Kostas
> >>
> >> [1]
> >> https://lists.apache.org/thread.html/r799be74658bc7e169238cc8c1e479e961a9e85ccea19089290940ff0%40%3Cdev.flink.apache.org%3E
> >>
> >> On Wed, Oct 14, 2020 at 8:03 AM Arvid Heise  
> >> wrote:
> >>> I remember this conversation popping up a few times already and I'm in
> >>> general a big fan of removing BucketingSink.
> >>>
> >>> However, until now there were a few features lacking in 
> >>> 

[jira] [Created] (FLINK-19856) Add EndOfChannelRecovery rescaling epoch

[Roman Khachatryan (Jira)]

Roman Khachatryan created FLINK-19856:
-

 Summary: Add EndOfChannelRecovery rescaling epoch
 Key: FLINK-19856
 URL: https://issues.apache.org/jira/browse/FLINK-19856
 Project: Flink
  Issue Type: New Feature
  Components: Runtime / Network
Affects Versions: 1.12.0
Reporter: Roman Khachatryan
Assignee: Roman Khachatryan
 Fix For: 1.12.0


This event would allow to tear down "virtual channels"     This event would 
allow to tear down "virtual channels"     used to read channel state on 
recovery with unaligned checkpoints and     rescaling.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19855) Incompatible semantics of channelIndex in UnionInputGate.resumeConsumption and its clients

[Roman Khachatryan (Jira)]

Roman Khachatryan created FLINK-19855:
-

 Summary: Incompatible semantics of channelIndex in 
UnionInputGate.resumeConsumption and its clients
 Key: FLINK-19855
 URL: https://issues.apache.org/jira/browse/FLINK-19855
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Network
Affects Versions: 1.11.2, 1.12.0
Reporter: Roman Khachatryan
Assignee: Roman Khachatryan
 Fix For: 1.12.0


Given only channelIndex for resumeConsumption, UnionInputGate has to guess 
which wrapped input gate this channel belongs to.
 For that, UnionInputGate expects channelIndex with an inputGate offset.
However, some clients (e.g. AlignedController) pass channel index without 
offset.
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Remove flink-connector-filesystem module.

[Chesnay Schepler]

The alternative could also be to use a different argument than "no one 
uses it", e.g., we are fine with removing it at the cost of friction for 
some users because there are better alternatives.


On 10/28/2020 10:46 AM, Kostas Kloudas wrote:

I think that the mailing lists is the best we can do and I would say
that they seem to be working pretty well (e.g. the recent Mesos
discussion).
Of course they are not perfect but the alternative would be to never
remove anything user facing until the next major release, which I find
pretty strict.

On Wed, Oct 28, 2020 at 10:04 AM Chesnay Schepler  wrote:

If the conclusion is that we shouldn't remove it if _anyone_ is using
it, then we cannot remove it because the user ML obviously does not
reach all users.

On 10/28/2020 9:28 AM, Kostas Kloudas wrote:

Hi all,

I am bringing the up again to see if there are any users actively
using the BucketingSink.
So far, if I am not mistaken (and really sorry if I forgot anything),
it is only a discussion between devs about the potential problems of
removing it. I totally understand Chesnay's concern about not
providing compatibility with the StreamingFileSink (SFS) and if there
are any users, then we should not remove it without trying to find a
solution for them.

But if there are no users then I would still propose to remove the
module, given that I am not aware of any efforts to provide
compatibility with the SFS any time soon.
The reasons for removing it also include the facts that we do not
actively maintain it and we do not add new features. As for potential
missing features in the SFS compared to the BucketingSink that was
mentioned before, I am not aware of any fundamental limitations and
even if there are, I would assume that the solution is not to direct
the users to a deprecated sink but rather try to increase the
functionality of the actively maintained one.

Please keep in mind that the BucketingSink is deprecated since FLINK
1.9 and there is a new File Sink that is coming as part of FLIP-143
[1].
Again, if there are any active users who cannot migrate easily, then
we cannot remove it before trying to provide a smooth migration path.

Thanks,
Kostas

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API

On Fri, Oct 16, 2020 at 4:36 PM Chesnay Schepler  wrote:

@Seth: Earlier in this discussion it was said that the BucketingSink
would not be usable in 1.12 .

On 10/16/2020 4:25 PM, Seth Wiesman wrote:

+1 It has been deprecated for some time and the StreamingFileSink has
stabalized with a large number of formats and features.

Plus, the bucketing sink only implements a small number of stable
interfaces[1]. I would expect users to continue to use the bucketing sink
from the 1.11 release with future versions for some time.

Seth

https://github.com/apache/flink/blob/2ff3b771cbb091e1f43686dd8e176cea6d435501/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java#L170-L172

On Thu, Oct 15, 2020 at 2:57 PM Kostas Kloudas  wrote:


@Arvid Heise I also do not remember exactly what were all the
problems. The fact that we added some more bulk formats to the
streaming file sink definitely reduced the non-supported features. In
addition, the latest discussion I found on the topic was [1] and the
conclusion of that discussion seems to be to remove it.

Currently, I cannot find any obvious reason why keeping the
BucketingSink, apart from the fact that we do not have a migration
plan unfortunately. This is why I posted this to dev@ and user@.

Cheers,
Kostas

[1]
https://lists.apache.org/thread.html/r799be74658bc7e169238cc8c1e479e961a9e85ccea19089290940ff0%40%3Cdev.flink.apache.org%3E

On Wed, Oct 14, 2020 at 8:03 AM Arvid Heise  wrote:

I remember this conversation popping up a few times already and I'm in
general a big fan of removing BucketingSink.

However, until now there were a few features lacking in StreamingFileSink
that are present in BucketingSink and that are being actively used (I

can't

exactly remember them now, but I can look it up if everyone else is also
suffering from bad memory). Did we manage to add them in the meantime? If
not, then it feels rushed to remove it at this point.

On Tue, Oct 13, 2020 at 2:33 PM Kostas Kloudas 

wrote:

@Chesnay Schepler  Off the top of my head, I cannot find an easy way
to migrate from the BucketingSink to the StreamingFileSink. It may be
possible but it will require some effort because the logic would be
"read the old state, commit it, and start fresh with the
StreamingFileSink."

On Tue, Oct 13, 2020 at 2:09 PM Aljoscha Krettek 
wrote:

On 13.10.20 14:01, David Anderson wrote:

I thought this was waiting on FLIP-46 -- Graceful Shutdown

Handling --

and

in fact, the StreamingFileSink is mentioned in that FLIP as a

motivating

use case.

Ah yes, I see FLIP-147 as a more general replacement for FLIP-46.

Thanks

for the reminder, we should close FLIP-46 now with an 

[jira] [Created] (FLINK-19854) TableScanTest.testTemporalJoinOnUpsertSource fails

[Robert Metzger (Jira)]

Robert Metzger created FLINK-19854:
--

 Summary: TableScanTest.testTemporalJoinOnUpsertSource fails
 Key: FLINK-19854
 URL: https://issues.apache.org/jira/browse/FLINK-19854
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.12.0
Reporter: Robert Metzger
 Fix For: 1.12.0


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=8482=logs=e25d5e7e-2a9c-5589-4940-0b638d75a414=a6e0f756-5bb9-5ea8-a468-5f60db442a29

org.junit.ComparisonFailure: planAfter expected:<...me, 
__TEMPORAL_JOIN_[]LEFT_KEY(currency), ...> but was:<...me, 
__TEMPORAL_JOIN_[CONDITION_PRIMARY_KEY(currency0), 
__TEMPORAL_JOIN_]LEFT_KEY(currency), ...>
at org.junit.Assert.assertEquals(Assert.java:115)
at 
org.apache.flink.table.planner.utils.DiffRepository.assertEquals(DiffRepository.java:436)
at 
org.apache.flink.table.planner.utils.TableTestUtilBase.assertEqualsOrExpand(TableTestBase.scala:478)
at 
org.apache.flink.table.planner.utils.TableTestUtilBase.doVerifyPlan(TableTestBase.scala:362)
at 
org.apache.flink.table.planner.utils.TableTestUtilBase.verifyPlan(TableTestBase.scala:275)
at 
org.apache.flink.table.planner.plan.stream.sql.TableScanTest.testTemporalJoinOnUpsertSource(TableScanTest.scala:515)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19853) BulkSlotProviderImplTest.testFailedBulkSlotAllocationReleasesAllocatedSlot failed in azure

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-19853:


 Summary: 
BulkSlotProviderImplTest.testFailedBulkSlotAllocationReleasesAllocatedSlot 
failed in azure
 Key: FLINK-19853
 URL: https://issues.apache.org/jira/browse/FLINK-19853
 Project: Flink
  Issue Type: Test
  Components: Runtime / Coordination
Reporter: Jingsong Lee


Instance: 
[https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_apis/build/builds/8486/logs/80]
{code:java}
2020-10-28T10:08:41.3922048Z java.lang.AssertionError: 
2020-10-28T10:08:41.3923050Z 
2020-10-28T10:08:41.3923362Z Expected: a collection with size <0>
2020-10-28T10:08:41.3926321Z  but: collection size was <1>
2020-10-28T10:08:41.3926732Zat 
org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
2020-10-28T10:08:41.3927463Zat org.junit.Assert.assertThat(Assert.java:956)
2020-10-28T10:08:41.3928006Zat org.junit.Assert.assertThat(Assert.java:923)
2020-10-28T10:08:41.3928622Zat 
org.apache.flink.runtime.jobmaster.slotpool.BulkSlotProviderImplTest.testFailedBulkSlotAllocationReleasesAllocatedSlot(BulkSlotProviderImplTest.java:154)
2020-10-28T10:08:41.3929157Zat 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-10-28T10:08:41.3929530Zat 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-10-28T10:08:41.3929981Zat 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19852) Managed memory released check can block IterativeTask

[shaomeng.wang (Jira)]

shaomeng.wang created FLINK-19852:
-

 Summary: Managed memory released check can block IterativeTask
 Key: FLINK-19852
 URL: https://issues.apache.org/jira/browse/FLINK-19852
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Task
Affects Versions: 1.11.2, 1.11.1, 1.10.2, 1.11.0
Reporter: shaomeng.wang
 Attachments: image-2020-10-28-17-48-28-395.png, 
image-2020-10-28-17-48-48-583.png

UnsafeMemoryBudget#reserveMemory, called on TempBarrier, needs time to wait on 
GC of all allocated/released managed memory at every iteration.

 

stack:

!image-2020-10-28-17-48-48-583.png!

new TempBarrier in BatchTask

!image-2020-10-28-17-48-28-395.png!

 

These will be very slow than before.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19851) flink sql client connector type jdbc exception

[hulingchan (Jira)]

hulingchan created FLINK-19851:
--

 Summary: flink sql client connector type jdbc exception
 Key: FLINK-19851
 URL: https://issues.apache.org/jira/browse/FLINK-19851
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Client
Affects Versions: 1.11.1
 Environment: hardware: Mac Pro

software: MacOS

 
Reporter: hulingchan


When I want to experience the SQL client using jdbc as the source, there is a 
problem.

*run command*：

./sql-client.sh embedded -e ../conf/sql-client-demo.yaml

*sql-client-demo.yaml conttent*：
{code:java}
tables:
  - name: mysql_test
type: source-table
connector:
type: jdbc
property-version: 1
url: 
jdbc:mysql://127.0.0.1:3306/test?useUnicode=true=utf8=true=Asia/Shanghai
table: book_info
driver: com.mysql.jdbc.Driver
username: lloo
password: dsfsdf
{code}
*log below*：
{code:java}
No default environment specified.
 Searching for 'flink-1.11.1/conf/sql-client-defaults.yaml'...found.
 Reading default environment from: 
file:flink-1.11.1/conf/sql-client-defaults.yaml
 Reading session environment from: 
file:flink-1.11.1/bin/../conf/sql-client-demo.yaml
Exception in thread "main" org.apache.flink.table.client.SqlClientException: 
Unexpected exception. This is a bug. Please consider filing an issue.
 at org.apache.flink.table.client.SqlClient.main(SqlClient.java:213)
 Caused by: org.apache.flink.table.client.gateway.SqlExecutionException: Could 
not create execution context.
 at 
org.apache.flink.table.client.gateway.local.ExecutionContext$Builder.build(ExecutionContext.java:870)
 at 
org.apache.flink.table.client.gateway.local.LocalExecutor.openSession(LocalExecutor.java:227)
 at org.apache.flink.table.client.SqlClient.start(SqlClient.java:108)
 at org.apache.flink.table.client.SqlClient.main(SqlClient.java:201)
 Caused by: org.apache.flink.table.api.NoMatchingTableFactoryException: Could 
not find a suitable table factory for 
'org.apache.flink.table.factories.TableSourceFactory' in
 the classpath.
Reason: Required context properties mismatch.
The following properties are requested:
 connector.driver=com.mysql.jdbc.Driver
 connector.password=123456
 connector.property-version=1
 connector.table=durotar_wx_user_info
 connector.type=jdbc
 
connector.url=jdbc:mysql://qa.vm.com:3306/zh_portal?useUnicode=true=utf8=true=Asia/Shanghai
 connector.username=root
The following factories have been considered:
 org.apache.flink.table.sources.CsvBatchTableSourceFactory
 org.apache.flink.table.sources.CsvAppendTableSourceFactory
 org.apache.flink.table.filesystem.FileSystemTableFactory
 at 
org.apache.flink.table.factories.TableFactoryService.filterByContext(TableFactoryService.java:322)
 at 
org.apache.flink.table.factories.TableFactoryService.filter(TableFactoryService.java:190)
 at 
org.apache.flink.table.factories.TableFactoryService.findSingleInternal(TableFactoryService.java:143)
 at 
org.apache.flink.table.factories.TableFactoryService.find(TableFactoryService.java:113)
 at 
org.apache.flink.table.client.gateway.local.ExecutionContext.createTableSource(ExecutionContext.java:384)
 at 
org.apache.flink.table.client.gateway.local.ExecutionContext.lambda$initializeCatalogs$7(ExecutionContext.java:638)
 at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
 at 
org.apache.flink.table.client.gateway.local.ExecutionContext.initializeCatalogs(ExecutionContext.java:636)
 at 
org.apache.flink.table.client.gateway.local.ExecutionContext.initializeTableEnvironment(ExecutionContext.java:523)
 at 
org.apache.flink.table.client.gateway.local.ExecutionContext.(ExecutionContext.java:183)
 at 
org.apache.flink.table.client.gateway.local.ExecutionContext.(ExecutionContext.java:136)
 at 
org.apache.flink.table.client.gateway.local.ExecutionContext$Builder.build(ExecutionContext.java:859)
 ... 3 more
  
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Remove flink-connector-filesystem module.

[Kostas Kloudas]

I think that the mailing lists is the best we can do and I would say
that they seem to be working pretty well (e.g. the recent Mesos
discussion).
Of course they are not perfect but the alternative would be to never
remove anything user facing until the next major release, which I find
pretty strict.

On Wed, Oct 28, 2020 at 10:04 AM Chesnay Schepler  wrote:
>
> If the conclusion is that we shouldn't remove it if _anyone_ is using
> it, then we cannot remove it because the user ML obviously does not
> reach all users.
>
> On 10/28/2020 9:28 AM, Kostas Kloudas wrote:
> > Hi all,
> >
> > I am bringing the up again to see if there are any users actively
> > using the BucketingSink.
> > So far, if I am not mistaken (and really sorry if I forgot anything),
> > it is only a discussion between devs about the potential problems of
> > removing it. I totally understand Chesnay's concern about not
> > providing compatibility with the StreamingFileSink (SFS) and if there
> > are any users, then we should not remove it without trying to find a
> > solution for them.
> >
> > But if there are no users then I would still propose to remove the
> > module, given that I am not aware of any efforts to provide
> > compatibility with the SFS any time soon.
> > The reasons for removing it also include the facts that we do not
> > actively maintain it and we do not add new features. As for potential
> > missing features in the SFS compared to the BucketingSink that was
> > mentioned before, I am not aware of any fundamental limitations and
> > even if there are, I would assume that the solution is not to direct
> > the users to a deprecated sink but rather try to increase the
> > functionality of the actively maintained one.
> >
> > Please keep in mind that the BucketingSink is deprecated since FLINK
> > 1.9 and there is a new File Sink that is coming as part of FLIP-143
> > [1].
> > Again, if there are any active users who cannot migrate easily, then
> > we cannot remove it before trying to provide a smooth migration path.
> >
> > Thanks,
> > Kostas
> >
> > [1] 
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API
> >
> > On Fri, Oct 16, 2020 at 4:36 PM Chesnay Schepler  wrote:
> >> @Seth: Earlier in this discussion it was said that the BucketingSink
> >> would not be usable in 1.12 .
> >>
> >> On 10/16/2020 4:25 PM, Seth Wiesman wrote:
> >>> +1 It has been deprecated for some time and the StreamingFileSink has
> >>> stabalized with a large number of formats and features.
> >>>
> >>> Plus, the bucketing sink only implements a small number of stable
> >>> interfaces[1]. I would expect users to continue to use the bucketing sink
> >>> from the 1.11 release with future versions for some time.
> >>>
> >>> Seth
> >>>
> >>> https://github.com/apache/flink/blob/2ff3b771cbb091e1f43686dd8e176cea6d435501/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java#L170-L172
> >>>
> >>> On Thu, Oct 15, 2020 at 2:57 PM Kostas Kloudas  wrote:
> >>>
>  @Arvid Heise I also do not remember exactly what were all the
>  problems. The fact that we added some more bulk formats to the
>  streaming file sink definitely reduced the non-supported features. In
>  addition, the latest discussion I found on the topic was [1] and the
>  conclusion of that discussion seems to be to remove it.
> 
>  Currently, I cannot find any obvious reason why keeping the
>  BucketingSink, apart from the fact that we do not have a migration
>  plan unfortunately. This is why I posted this to dev@ and user@.
> 
>  Cheers,
>  Kostas
> 
>  [1]
>  https://lists.apache.org/thread.html/r799be74658bc7e169238cc8c1e479e961a9e85ccea19089290940ff0%40%3Cdev.flink.apache.org%3E
> 
>  On Wed, Oct 14, 2020 at 8:03 AM Arvid Heise  wrote:
> > I remember this conversation popping up a few times already and I'm in
> > general a big fan of removing BucketingSink.
> >
> > However, until now there were a few features lacking in 
> > StreamingFileSink
> > that are present in BucketingSink and that are being actively used (I
>  can't
> > exactly remember them now, but I can look it up if everyone else is also
> > suffering from bad memory). Did we manage to add them in the meantime? 
> > If
> > not, then it feels rushed to remove it at this point.
> >
> > On Tue, Oct 13, 2020 at 2:33 PM Kostas Kloudas 
>  wrote:
> >> @Chesnay Schepler  Off the top of my head, I cannot find an easy way
> >> to migrate from the BucketingSink to the StreamingFileSink. It may be
> >> possible but it will require some effort because the logic would be
> >> "read the old state, commit it, and start fresh with the
> >> StreamingFileSink."
> >>
> >> On Tue, Oct 13, 2020 at 2:09 PM Aljoscha Krettek 
> >> wrote:
> >>> On 13.10.20 14:01, David Anderson wrote:
>  

[jira] [Created] (FLINK-19850) Add E2E tests for the new streaming case of the new FileSink

[Yun Gao (Jira)]

Yun Gao created FLINK-19850:
---

 Summary: Add E2E tests for the new streaming case of the new 
FileSink
 Key: FLINK-19850
 URL: https://issues.apache.org/jira/browse/FLINK-19850
 Project: Flink
  Issue Type: Sub-task
Reporter: Yun Gao






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Remove flink-connector-filesystem module.

[Chesnay Schepler]

If the conclusion is that we shouldn't remove it if _anyone_ is using 
it, then we cannot remove it because the user ML obviously does not 
reach all users.


On 10/28/2020 9:28 AM, Kostas Kloudas wrote:

Hi all,

I am bringing the up again to see if there are any users actively
using the BucketingSink.
So far, if I am not mistaken (and really sorry if I forgot anything),
it is only a discussion between devs about the potential problems of
removing it. I totally understand Chesnay's concern about not
providing compatibility with the StreamingFileSink (SFS) and if there
are any users, then we should not remove it without trying to find a
solution for them.

But if there are no users then I would still propose to remove the
module, given that I am not aware of any efforts to provide
compatibility with the SFS any time soon.
The reasons for removing it also include the facts that we do not
actively maintain it and we do not add new features. As for potential
missing features in the SFS compared to the BucketingSink that was
mentioned before, I am not aware of any fundamental limitations and
even if there are, I would assume that the solution is not to direct
the users to a deprecated sink but rather try to increase the
functionality of the actively maintained one.

Please keep in mind that the BucketingSink is deprecated since FLINK
1.9 and there is a new File Sink that is coming as part of FLIP-143
[1].
Again, if there are any active users who cannot migrate easily, then
we cannot remove it before trying to provide a smooth migration path.

Thanks,
Kostas

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API

On Fri, Oct 16, 2020 at 4:36 PM Chesnay Schepler  wrote:

@Seth: Earlier in this discussion it was said that the BucketingSink
would not be usable in 1.12 .

On 10/16/2020 4:25 PM, Seth Wiesman wrote:

+1 It has been deprecated for some time and the StreamingFileSink has
stabalized with a large number of formats and features.

Plus, the bucketing sink only implements a small number of stable
interfaces[1]. I would expect users to continue to use the bucketing sink
from the 1.11 release with future versions for some time.

Seth

https://github.com/apache/flink/blob/2ff3b771cbb091e1f43686dd8e176cea6d435501/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java#L170-L172

On Thu, Oct 15, 2020 at 2:57 PM Kostas Kloudas  wrote:


@Arvid Heise I also do not remember exactly what were all the
problems. The fact that we added some more bulk formats to the
streaming file sink definitely reduced the non-supported features. In
addition, the latest discussion I found on the topic was [1] and the
conclusion of that discussion seems to be to remove it.

Currently, I cannot find any obvious reason why keeping the
BucketingSink, apart from the fact that we do not have a migration
plan unfortunately. This is why I posted this to dev@ and user@.

Cheers,
Kostas

[1]
https://lists.apache.org/thread.html/r799be74658bc7e169238cc8c1e479e961a9e85ccea19089290940ff0%40%3Cdev.flink.apache.org%3E

On Wed, Oct 14, 2020 at 8:03 AM Arvid Heise  wrote:

I remember this conversation popping up a few times already and I'm in
general a big fan of removing BucketingSink.

However, until now there were a few features lacking in StreamingFileSink
that are present in BucketingSink and that are being actively used (I

can't

exactly remember them now, but I can look it up if everyone else is also
suffering from bad memory). Did we manage to add them in the meantime? If
not, then it feels rushed to remove it at this point.

On Tue, Oct 13, 2020 at 2:33 PM Kostas Kloudas 

wrote:

@Chesnay Schepler  Off the top of my head, I cannot find an easy way
to migrate from the BucketingSink to the StreamingFileSink. It may be
possible but it will require some effort because the logic would be
"read the old state, commit it, and start fresh with the
StreamingFileSink."

On Tue, Oct 13, 2020 at 2:09 PM Aljoscha Krettek 
wrote:

On 13.10.20 14:01, David Anderson wrote:

I thought this was waiting on FLIP-46 -- Graceful Shutdown

Handling --

and

in fact, the StreamingFileSink is mentioned in that FLIP as a

motivating

use case.

Ah yes, I see FLIP-147 as a more general replacement for FLIP-46.

Thanks

for the reminder, we should close FLIP-46 now with an explanatory
message to avoid confusion.

--

Arvid Heise | Senior Java Developer



Follow us @VervericaData

--

Join Flink Forward  - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
(Toni) Cheng

Re: [BULK]Re: [SURVEY] Remove Mesos support

[Till Rohrmann]

Hi Oleksandr,

yes you are right. The biggest problem is at the moment the lack of test
coverage and thereby confidence to make changes. We have some e2e tests
which you can find here [1]. These tests are, however, quite coarse grained
and are missing a lot of cases. One idea would be to add a Mesos e2e test
based on Flink's end-to-end test framework [2]. I think what needs to be
done there is to add a Mesos resource and a way to submit jobs to a Mesos
cluster to write e2e tests.

[1] https://github.com/apache/flink/tree/master/flink-jepsen
[2]
https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common

Cheers,
Till

On Tue, Oct 27, 2020 at 12:29 PM Oleksandr Nitavskyi 
wrote:

> Hello Xintong,
>
> Thanks for the insights and support.
>
> Browsing the Mesos backlog and didn't identify anything critical, which is
> left there.
>
> I see that there are were quite a lot of contributions to the Flink Mesos
> in the recent version:
> https://github.com/apache/flink/commits/master/flink-mesos.
> We plan to validate the current Flink master (or release 1.12 branch) our
> Mesos setup. In case of any issues, we will try to propose changes.
> My feeling is that our test results shouldn't affect the Flink 1.12
> release cycle. And if any potential commits will land into the 1.12.1 it
> should be totally fine.
>
> In the future, we would be glad to help you guys with any
> maintenance-related questions. One of the highest priorities around this
> component seems to be the development of the full e2e test.
>
> Kind Regards
> Oleksandr Nitavskyi
> 
> From: Xintong Song 
> Sent: Tuesday, October 27, 2020 7:14 AM
> To: dev ; user 
> Cc: Piyush Narang 
> Subject: [BULK]Re: [SURVEY] Remove Mesos support
>
> Hi Piyush,
>
> Thanks a lot for sharing the information. It would be a great relief that
> you are good with Flink on Mesos as is.
>
> As for the jira issues, I believe the most essential ones should have
> already been resolved. You may find some remaining open issues here [1],
> but not all of them are necessary if we decide to keep Flink on Mesos as is.
>
> At the moment and in the short future, I think helps are mostly needed on
> testing the upcoming release 1.12 with Mesos use cases. The community is
> currently actively preparing the new release, and hopefully we could come
> up with a release candidate early next month. It would be greatly
> appreciated if you fork as experienced Flink on Mesos users can help with
> verifying the release candidates.
>
>
> Thank you~
>
> Xintong Song
>
> [1]
> https://issues.apache.org/jira/browse/FLINK-17402?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Deployment%20%2F%20Mesos%22%20AND%20status%20%3D%20Open
> <
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-17402%3Fjql%3Dproject%2520%253D%2520FLINK%2520AND%2520component%2520%253D%2520%2522Deployment%2520%252F%2520Mesos%2522%2520AND%2520status%2520%253D%2520Open=04%7C01%7Co.nitavskyi%40criteo.com%7C3585e1f25bdf4e091af808d87a3f92db%7C2a35d8fd574d48e3927c8c398e225a01%7C1%7C0%7C637393760750820881%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=hytJFQE0MCPzMLiQTQTdbg3GVckX5M3r1NPRGrRV8j4%3D=0
> >
>
> On Tue, Oct 27, 2020 at 2:58 AM Piyush Narang  p.nar...@criteo.com>> wrote:
>
> Hi Xintong,
>
>
>
> Do you have any jiras that cover any of the items on 1 or 2? I can reach
> out to folks internally and see if I can get some folks to commit to
> helping out.
>
>
>
> To cover the other qs:
>
>   *   Yes, we’ve not got a plan at the moment to get off Mesos. We use
> Yarn for some our Flink workloads when we can. Mesos is only used when we
> need streaming capabilities in our WW dcs (as our Yarn is centralized in
> one DC)
>   *   We’re currently on Flink 1.9 (old planner). We have a plan to bump
> to 1.11 / 1.12 this quarter.
>   *   We typically upgrade once every 6 months to a year (not every
> release). We’d like to speed up the cadence but we’re not there yet.
>   *   We’d largely be good with keeping Flink on Mesos as-is and
> functional while missing out on some of the newer features. We understand
> the pain on the communities side and we can take on the work if we see some
> fancy improvement in Flink on Yarn / K8s that we want in Mesos to put in
> the request to port it over.
>
>
>
> Thanks,
>
>
>
> -- Piyush
>
>
>
>
>
> From: Xintong Song mailto:tonysong...@gmail.com>>
> Date: Sunday, October 25, 2020 at 10:57 PM
> To: dev mailto:dev@flink.apache.org>>, user <
> u...@flink.apache.org>
> Cc: Lasse Nedergaard  lassenedergaardfl...@gmail.com>>,  p.nar...@criteo.com>>
> Subject: Re: [SURVEY] Remove Mesos support
>
>
>
> Thanks for sharing the information with us, Piyush an Lasse.
>
>
>
> @Piyush
>
>
>
> Thanks for offering the help. IMO, there are currently several problems
> that make supporting Flink on Mesos 

Re: [DISCUSS] Remove flink-connector-filesystem module.

[Kostas Kloudas]

Hi all,

I am bringing the up again to see if there are any users actively
using the BucketingSink.
So far, if I am not mistaken (and really sorry if I forgot anything),
it is only a discussion between devs about the potential problems of
removing it. I totally understand Chesnay's concern about not
providing compatibility with the StreamingFileSink (SFS) and if there
are any users, then we should not remove it without trying to find a
solution for them.

But if there are no users then I would still propose to remove the
module, given that I am not aware of any efforts to provide
compatibility with the SFS any time soon.
The reasons for removing it also include the facts that we do not
actively maintain it and we do not add new features. As for potential
missing features in the SFS compared to the BucketingSink that was
mentioned before, I am not aware of any fundamental limitations and
even if there are, I would assume that the solution is not to direct
the users to a deprecated sink but rather try to increase the
functionality of the actively maintained one.

Please keep in mind that the BucketingSink is deprecated since FLINK
1.9 and there is a new File Sink that is coming as part of FLIP-143
[1].
Again, if there are any active users who cannot migrate easily, then
we cannot remove it before trying to provide a smooth migration path.

Thanks,
Kostas

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API

On Fri, Oct 16, 2020 at 4:36 PM Chesnay Schepler  wrote:
>
> @Seth: Earlier in this discussion it was said that the BucketingSink
> would not be usable in 1.12 .
>
> On 10/16/2020 4:25 PM, Seth Wiesman wrote:
> > +1 It has been deprecated for some time and the StreamingFileSink has
> > stabalized with a large number of formats and features.
> >
> > Plus, the bucketing sink only implements a small number of stable
> > interfaces[1]. I would expect users to continue to use the bucketing sink
> > from the 1.11 release with future versions for some time.
> >
> > Seth
> >
> > https://github.com/apache/flink/blob/2ff3b771cbb091e1f43686dd8e176cea6d435501/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java#L170-L172
> >
> > On Thu, Oct 15, 2020 at 2:57 PM Kostas Kloudas  wrote:
> >
> >> @Arvid Heise I also do not remember exactly what were all the
> >> problems. The fact that we added some more bulk formats to the
> >> streaming file sink definitely reduced the non-supported features. In
> >> addition, the latest discussion I found on the topic was [1] and the
> >> conclusion of that discussion seems to be to remove it.
> >>
> >> Currently, I cannot find any obvious reason why keeping the
> >> BucketingSink, apart from the fact that we do not have a migration
> >> plan unfortunately. This is why I posted this to dev@ and user@.
> >>
> >> Cheers,
> >> Kostas
> >>
> >> [1]
> >> https://lists.apache.org/thread.html/r799be74658bc7e169238cc8c1e479e961a9e85ccea19089290940ff0%40%3Cdev.flink.apache.org%3E
> >>
> >> On Wed, Oct 14, 2020 at 8:03 AM Arvid Heise  wrote:
> >>> I remember this conversation popping up a few times already and I'm in
> >>> general a big fan of removing BucketingSink.
> >>>
> >>> However, until now there were a few features lacking in StreamingFileSink
> >>> that are present in BucketingSink and that are being actively used (I
> >> can't
> >>> exactly remember them now, but I can look it up if everyone else is also
> >>> suffering from bad memory). Did we manage to add them in the meantime? If
> >>> not, then it feels rushed to remove it at this point.
> >>>
> >>> On Tue, Oct 13, 2020 at 2:33 PM Kostas Kloudas 
> >> wrote:
>  @Chesnay Schepler  Off the top of my head, I cannot find an easy way
>  to migrate from the BucketingSink to the StreamingFileSink. It may be
>  possible but it will require some effort because the logic would be
>  "read the old state, commit it, and start fresh with the
>  StreamingFileSink."
> 
>  On Tue, Oct 13, 2020 at 2:09 PM Aljoscha Krettek 
>  wrote:
> > On 13.10.20 14:01, David Anderson wrote:
> >> I thought this was waiting on FLIP-46 -- Graceful Shutdown
> >> Handling --
>  and
> >> in fact, the StreamingFileSink is mentioned in that FLIP as a
>  motivating
> >> use case.
> > Ah yes, I see FLIP-147 as a more general replacement for FLIP-46.
> >> Thanks
> > for the reminder, we should close FLIP-46 now with an explanatory
> > message to avoid confusion.
> >>>
> >>> --
> >>>
> >>> Arvid Heise | Senior Java Developer
> >>>
> >>> 
> >>>
> >>> Follow us @VervericaData
> >>>
> >>> --
> >>>
> >>> Join Flink Forward  - The Apache Flink
> >>> Conference
> >>>
> >>> Stream Processing | Event Driven | Real Time
> >>>
> >>> --
> >>>
> >>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
> >>>
> >>> --
> >>> Ververica GmbH
> >>> 

[jira] [Created] (FLINK-19849) Check NOTICE files for 1.12 release

[Robert Metzger (Jira)]

Robert Metzger created FLINK-19849:
--

 Summary: Check NOTICE files for 1.12 release
 Key: FLINK-19849
 URL: https://issues.apache.org/jira/browse/FLINK-19849
 Project: Flink
  Issue Type: Bug
  Components: Build System
Affects Versions: 1.12.0
Reporter: Robert Metzger
Assignee: Robert Metzger
 Fix For: 1.12.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19848) flink docs of Building Flink from Source bug

[jackylau (Jira)]

jackylau created FLINK-19848:


 Summary: flink docs of Building Flink from Source bug
 Key: FLINK-19848
 URL: https://issues.apache.org/jira/browse/FLINK-19848
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Affects Versions: 1.11.0
Reporter: jackylau
 Fix For: 1.12.0


To speed up the build you can skip tests, QA plugins, and JavaDocs:

 

{{mvn clean install -DskipTests -Dfast}}

 

{{mvn clean install -DskipTests -Dscala-2.12}}

{{fast and }}{{scala-2.12}}{{ is profile, not properties}}

{{}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-19847) Can we create a fast support on the Nested table join?

[xiaogang zhou (Jira)]

xiaogang zhou created FLINK-19847:
-

 Summary: Can we create a fast support on the Nested table join?
 Key: FLINK-19847
 URL: https://issues.apache.org/jira/browse/FLINK-19847
 Project: Flink
  Issue Type: Wish
  Components: API / DataStream
Affects Versions: 1.11.1
Reporter: xiaogang zhou


In CommonLookupJoin, one TODO is 

support nested lookup keys in the future,
// currently we only support top-level lookup keys

 

can we create a fast support on the Array join? thx



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] NEW FLIP-104: Add More Metrics to JobManager

[Kurt Young]

+1

Best,
Kurt


On Wed, Oct 28, 2020 at 2:44 PM Robert Metzger  wrote:

> Thank you for your UI work Yadong!
>
> +1
>
> On Tue, Oct 27, 2020 at 6:33 PM Matthias Pohl 
> wrote:
>
> > Thanks for restarting the vote, Yadong. I really like your UI proposals.
> > +1 for adding the changes of FLIP-104.
> >
> > Matthias
> >
> > On Tue, Oct 27, 2020 at 10:29 AM Xintong Song 
> > wrote:
> >
> > > Thanks for reviving this FLIP, Yandong.
> > >
> > > The proposed changes look good to me.
> > > +1 for accepting this FLIP.
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Tue, Oct 27, 2020 at 4:02 PM Yadong Xie 
> wrote:
> > >
> > > > Hi all
> > > >
> > > > I want to start a new vote for FLIP-104, which proposes to add more
> > > metrics
> > > > to the job manager in web UI.
> > > >
> > > > The new FLIP-104 was revisited and adapted following the old ML
> > > discussion
> > > > <
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-104-Add-More-Metrics-to-Jobmanager-td37901.html
> > > > >
> > > > .
> > > >
> > > > The vote will last for at least 72 hours, following the consensus
> > voting.
> > > >
> > > >
> > > > FLIP-104 wiki:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-104%3A+Add+More+Metrics+to+Jobmanager
> > > >
> > > >
> > > > Discussion thread:
> > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-104-Add-More-Metrics-to-Jobmanager-td37901.html
> > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> > > >
> > > > Thanks,
> > > >
> > > > Yadong
> > > >
> > >
> >
>

[jira] [Created] (FLINK-19846) Grammar mistakes in annotations and log

[zhouchao (Jira)]

zhouchao created FLINK-19846:


 Summary: Grammar mistakes in annotations and log
 Key: FLINK-19846
 URL: https://issues.apache.org/jira/browse/FLINK-19846
 Project: Flink
  Issue Type: Wish
Affects Versions: 1.11.2
Reporter: zhouchao
 Fix For: 1.12.0


There exit some grammar mistakes in annotations and documents. The mistakes 
include but are not limited to the following examples：
 * a entry in  WebLogAnalysis.java  [246:34] and adm-zip.js [291:33](which 
should be an entry)
 * a input in JobGraphGenerator.java [1125:69] etc(which should be an 
input)
 * a intersection 
 * an user-* in Table.java etc.  (which should be a user)

using global search in intellij idea, more mistakes could be foud like this.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Releasing Apache Flink 1.11.3

[Tzu-Li (Gordon) Tai]

Thanks for the replies so far!

Just to provide a brief update on the status of blockers for 1.11.3 so far:


*PR opened, pending reviewer*- [FLINK-19717] SourceReaderBase.pollNext may
return END_OF_INPUT if SplitReader.fetch throws (
https://github.com/apache/flink/pull/13776)

*PR opened, reviewed + close to being merged*
- [FLINK-19741] Timer service should skip restoring from raw keyed stream
if it isn't the writer (https://github.com/apache/flink/pull/13761)
- [FLINK-19748] Raw keyed stream key group iterator should be skipping
unwritten key groups (https://github.com/apache/flink/pull/13772)

*Merged*
- [FLINK-19154] Application mode deletes HA data in case of suspended
ZooKeeper connection
- [FLINK-19569] Upgrade ICU4J to 67.1+

Right now as it seems, progress is mainly blocked on a reviewer for
FLINK-19717.
Meanwhile, Xintong is keeping an eye on test instabilities [1] to see if
there are any fixes that should be applied to `release-1.11`.

This is also a reminder, that if there are other blockers that we need to
be aware of, or a need to re-establish estimated time for getting fixes in
and delay the RC for 1.11.3, please do let us know!

Cheers,
Gordon

[1]
https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20affectedVersion%20in%20(1.11.0%2C%201.11.1%2C%201.11.2)%20AND%20labels%20%3D%20test-stability

On Mon, Oct 26, 2020 at 9:43 PM Kostas Kloudas  wrote:

> +1 for releasing Flink 1.11.3 as it contains a number of important
> fixes and thanks Gordon and Xintong for volunteering.
>
> Cheers,
> Kostas
>
> On Mon, Oct 26, 2020 at 4:37 AM Yu Li  wrote:
> >
> > +1 for releasing Flink 1.11.3, and thanks Gordon and Xintong for
> > volunteering as our release managers.
> >
> > Best Regards,
> > Yu
> >
> >
> > On Mon, 26 Oct 2020 at 09:45, Xintong Song 
> wrote:
> >
> > > Thanks Gordan for starting this discussion.
> > > My pleasure to help with the release process.
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Fri, Oct 23, 2020 at 11:29 PM Till Rohrmann 
> > > wrote:
> > >
> > > > Thanks for starting this discussion Gordon. There are over 100 issues
> > > > which are fixed for 1.11.3. Hence +1 for a soonish 1.11.3 release.
> Thanks
> > > > for volunteering as our release managers Gordon and Xintong!
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Fri, Oct 23, 2020 at 5:02 PM Tzu-Li (Gordon) Tai <
> tzuli...@apache.org
> > > >
> > > > wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> Xintong and I would like to start a discussion for releasing Flink
> > > 1.11.3
> > > >> soon.
> > > >>
> > > >> It seems like we already have a few pressing issues that needs to be
> > > >> included in a new hotfix release:
> > > >>
> > > >>- Heap-based timers’ restore behaviour is causing a critical
> recovery
> > > >>issue for StateFun [1] [2] [3].
> > > >>- There are several robustness issues for the FLIP-27 new source
> API,
> > > >>such as [4]. We already have some users using the FLIP-27 API
> with
> > > >> 1.11.x,
> > > >>so it would be important to get those fixes in for 1.11.x as
> well.
> > > >>
> > > >> Apart from the issues that are already marked as blocker for 1.11.3
> in
> > > our
> > > >> JIRA [5], please let us know in this thread if there is already
> ongoing
> > > >> work for other important fixes that we should try to include.
> > > >>
> > > >> Xintong and I would like to volunteer for managing this release, and
> > > will
> > > >> try to communicate the priority of pending blockers over the next
> few
> > > >> days.
> > > >> Since the aforementioned issues are quite critical, we’d like to aim
> > > >> for a *feature
> > > >> freeze by the end of next week (Oct. 30th)* and start the release
> voting
> > > >> process the week after.
> > > >> If that is too short of a notice and you might need more time,
> please
> > > let
> > > >> us know!
> > > >>
> > > >> Cheers,
> > > >> Gordon
> > > >>
> > > >> [1] https://issues.apache.org/jira/browse/FLINK-19692
> > > >> [2] https://issues.apache.org/jira/browse/FLINK-19741
> > > >> [3] https://issues.apache.org/jira/browse/FLINK-19748
> > > >> [4] https://issues.apache.org/jira/browse/FLINK-19717
> > > >> [5]
> > > >>
> > > >>
> > >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%201.11.3
> > > >>
> > > >
> > >
>

Re: [VOTE] NEW FLIP-104: Add More Metrics to JobManager

[Robert Metzger]

Thank you for your UI work Yadong!

+1

On Tue, Oct 27, 2020 at 6:33 PM Matthias Pohl 
wrote:

> Thanks for restarting the vote, Yadong. I really like your UI proposals.
> +1 for adding the changes of FLIP-104.
>
> Matthias
>
> On Tue, Oct 27, 2020 at 10:29 AM Xintong Song 
> wrote:
>
> > Thanks for reviving this FLIP, Yandong.
> >
> > The proposed changes look good to me.
> > +1 for accepting this FLIP.
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Tue, Oct 27, 2020 at 4:02 PM Yadong Xie  wrote:
> >
> > > Hi all
> > >
> > > I want to start a new vote for FLIP-104, which proposes to add more
> > metrics
> > > to the job manager in web UI.
> > >
> > > The new FLIP-104 was revisited and adapted following the old ML
> > discussion
> > > <
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-104-Add-More-Metrics-to-Jobmanager-td37901.html
> > > >
> > > .
> > >
> > > The vote will last for at least 72 hours, following the consensus
> voting.
> > >
> > >
> > > FLIP-104 wiki:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-104%3A+Add+More+Metrics+to+Jobmanager
> > >
> > >
> > > Discussion thread:
> > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-104-Add-More-Metrics-to-Jobmanager-td37901.html
> > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> > >
> > > Thanks,
> > >
> > > Yadong
> > >
> >
>

Re: [REMINDER] Please only merge pull requests when CI is green and help keep our CI stable

[Robert Metzger]

Quick update: The issue has been resolved.

It was actually not as bad as I thought, because flinkbot was showing the
build as "FAILED", so people probably saw that something is wrong.



On Tue, Oct 27, 2020 at 3:51 PM Robert Metzger  wrote:

> Hi all,
>
> Quick update on our CI system during the "hot phase" of the release:
> 1. We are (for the first time in this release cycle) *hitting our e2e
> test processing capacity limit:* All our 20 slots are in use, 13 builds
> are in the queue. The other tests are running without delays. I will
> monitor this closely.
> Please try to use your personal Azure accounts as much as possible for
> testing, and only push to your PR once you know the change should pass.
>
> 2. I probably made a *mistake in our e2e test scripts, causing broken e2e
> tests to not fail the build*.
> I'm tracking the issue here:
> https://issues.apache.org/jira/browse/FLINK-19839 I'm trying to fix this
> in the next hours.
> You can see if the e2e test failed if there's a warning with the cache
> upload, as with this example:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=8385=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
>
> I have checked the recent master builds, and the e2e tests on master seem
> to be stable. Until I've fixed this, please take a quick look onto the CI
> overview page of your PR before merging! Thanks & sorry!
>
>
>
> On Mon, Oct 19, 2020 at 11:15 AM Till Rohrmann 
> wrote:
>
>> Thanks for the reminder Robert. I think you are completely right. Since we
>> are getting closer to the feature freeze and therefore also the testing
>> period, having green builds and only merging PRs where we are sure that CI
>> passes will decrease the likelihood of introducing new bugs and, hence, it
>> will decrease the time we need for hardening the release. Of course, this
>> should also be the modus operandi for not only shortly before the feature
>> freeze.
>>
>> Cheers,
>> Till
>>
>> On Mon, Oct 19, 2020 at 11:00 AM Robert Metzger 
>> wrote:
>>
>> > Hi folks,
>> >
>> > As we are approaching the final stages of the Flink 1.12 release cycle,
>> I
>> > would like to remind everybody to *only merge pull requests when the CI
>> > system gives green light*!
>> > I know that the CI system sometimes shows "FAILED" due to CI system
>> > instabilities, but please carefully check those cases to make sure it is
>> > really the CI systems fault, and not an unstable or failing test.
>> > It is much easier to identify and fix an unstable or failing test in the
>> > context of a pull request then after it's been merged.
>> >
>> > An essential part of keeping our CI system useful is helping to address
>> > build instabilities early and proactively.
>> > In this phase, I'll try to proactively revert commits introducing severe
>> > issues, or disable failing/unstable tests early to keep the CI system
>> > meaningful.
>> >
>> > Please post in this thread or reach out to me personally if you are
>> > uncertain about a build failure or the overall state of the CI system.
>> >
>> > Best,
>> > Robert
>> >
>>
>

[jira] [Created] (FLINK-19845) Migrate all FileSystemFormatFactory implementations

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-19845:


 Summary: Migrate all FileSystemFormatFactory implementations
 Key: FLINK-19845
 URL: https://issues.apache.org/jira/browse/FLINK-19845
 Project: Flink
  Issue Type: Sub-task
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Reporter: Jingsong Lee


We should use interfaces introduced by FLINK-19599



--
This message was sent by Atlassian Jira
(v8.3.4#803005)