Re: [VOTE] FLIP-69: Flink SQL DDL Enhancement

[Terry Wang]

Hi Rui~

Description of the database is obtained from `CatalogDatabase#getDescription()` 
method, which is implement by CatalogDatebaseImpl. Users don’t need to specify 
the description.

Best,
Terry Wang



> 2019年11月7日 15:40，Rui Li  写道：
> 
> Thanks Terry for driving this forward.
> Got one question about DESCRIBE DATABASE: the results display comment and
> description of a database. While comment can be specified when a database
> is created, I don't see how users can specify description of the database?
> 
> On Thu, Nov 7, 2019 at 4:16 AM Bowen Li  wrote:
> 
>> Thanks.
>> 
>> As Terry and I discussed offline yesterday, we added a new section to
>> explain the detailed implementation plan.
>> 
>> +1 (binding) from me.
>> 
>> Bowen
>> 
>> On Tue, Nov 5, 2019 at 6:33 PM Terry Wang  wrote:
>> 
>>> Hi Bowen:
>>> Thanks for your feedback.
>>> Your opinion convinced me and I just remove the section about catalog
>>> create statement and also remove `DBPROPERTIES` `PROPERTIES` from alter
>>> DDLs.
>>> Open to more comments or votes :) !
>>> 
>>> Best,
>>> Terry Wang
>>> 
>>> 
>>> 
 2019年11月6日 07:22，Bowen Li  写道：
 
 Hi Terry,
 
 I went over the FLIP in detail again. The FLIP mostly LGTM. A couple
>>> issues:
 
 - since we on't plan to support catalog ddl, can you remove them from
>> the
 FLIP?
 - I found there are some discrepancies in proposed database and table
>>> DDLs.
 For db ddl, the create db syntax proposes specifying k-v properties
 following "WITH". However, alter db ddl comes with a keyword
>>> "DBPROPERTIES":
 
 CREATE  DATABASE [ IF NOT EXISTS ] [ catalogName.] dataBaseName [
>> COMMENT
 database_comment ]
 [*WITH *( name=value [, name=value]*)]
 
 
 ALTER  DATABASE  [ catalogName.] dataBaseName SET *DBPROPERTIES* (
 name=value [, name=value]*)
 
 
   IIUIC, are you borrowing syntax from Hive? Note that Hive's db
>> create
 ddl comes with "DBPROPERTIES" though - "CREATE (DATABASE|SCHEMA) [IF
>> NOT
 EXISTS] database_name ...  [*WITH DBPROPERTIES* (k=v, ...)];" [1]
 
  The same applies to table ddl. The proposed alter table ddl comes
>> with
 "SET *PROPERTIES* (...)", however, Flink's existing table create ddl
>>> since
 1.9 [2] doesn't have "PROPERTIES" keyword. As opposed to Hive's syntax,
 both create and alter table ddl comes with "TBLPROPERTIES" [1].
 
  I feel it's better to be consistent among our DDLs. One option is to
 just remove the "PROPERTIES" and "DBPROPERTIES" keywords in proposed
>>> syntax.
 
 [1]
>> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL
 [2]
 
>>> 
>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sql.html#specifying-a-ddl
 
 On Tue, Nov 5, 2019 at 12:54 PM Peter Huang <
>> huangzhenqiu0...@gmail.com>
 wrote:
 
> +1 for the enhancement.
> 
> On Tue, Nov 5, 2019 at 11:04 AM Xuefu Z  wrote:
> 
>> +1 to the long missing feature in Flink SQL.
>> 
>> On Tue, Nov 5, 2019 at 6:32 AM Terry Wang 
>> wrote:
>> 
>>> Hi all,
>>> 
>>> I would like to start the vote for FLIP-69[1] which is discussed and
>>> reached consensus in the discussion thread[2].
>>> 
>>> The vote will be open for at least 72 hours. I'll try to close it by
>>> 2019-11-08 14:30 UTC, unless there is an objection or not enough
>>> votes.
>>> 
>>> [1]
>>> 
>> 
> 
>>> 
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement
>>> <
>>> 
>> 
> 
>>> 
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement
 
>>> [2]
>>> 
>> 
> 
>>> 
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html
>>> <
>>> 
>> 
> 
>>> 
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html
 
>>> Best,
>>> Terry Wang
>>> 
>>> 
>>> 
>>> 
>> 
>> --
>> Xuefu Zhang
>> 
>> "In Honey We Trust!"
>> 
> 
>>> 
>>> 
>> 
> 
> 
> -- 
> Best regards!
> Rui Li

Re: [VOTE] FLIP-69: Flink SQL DDL Enhancement

[Rui Li]

Thanks Terry for driving this forward.
Got one question about DESCRIBE DATABASE: the results display comment and
description of a database. While comment can be specified when a database
is created, I don't see how users can specify description of the database?

On Thu, Nov 7, 2019 at 4:16 AM Bowen Li  wrote:

> Thanks.
>
> As Terry and I discussed offline yesterday, we added a new section to
> explain the detailed implementation plan.
>
> +1 (binding) from me.
>
> Bowen
>
> On Tue, Nov 5, 2019 at 6:33 PM Terry Wang  wrote:
>
> > Hi Bowen:
> > Thanks for your feedback.
> > Your opinion convinced me and I just remove the section about catalog
> > create statement and also remove `DBPROPERTIES` `PROPERTIES` from alter
> > DDLs.
> > Open to more comments or votes :) !
> >
> > Best,
> > Terry Wang
> >
> >
> >
> > > 2019年11月6日 07:22，Bowen Li  写道：
> > >
> > > Hi Terry,
> > >
> > > I went over the FLIP in detail again. The FLIP mostly LGTM. A couple
> > issues:
> > >
> > > - since we on't plan to support catalog ddl, can you remove them from
> the
> > > FLIP?
> > > - I found there are some discrepancies in proposed database and table
> > DDLs.
> > >  For db ddl, the create db syntax proposes specifying k-v properties
> > > following "WITH". However, alter db ddl comes with a keyword
> > "DBPROPERTIES":
> > >
> > > CREATE  DATABASE [ IF NOT EXISTS ] [ catalogName.] dataBaseName [
> COMMENT
> > > database_comment ]
> > > [*WITH *( name=value [, name=value]*)]
> > >
> > >
> > > ALTER  DATABASE  [ catalogName.] dataBaseName SET *DBPROPERTIES* (
> > > name=value [, name=value]*)
> > >
> > >
> > >IIUIC, are you borrowing syntax from Hive? Note that Hive's db
> create
> > > ddl comes with "DBPROPERTIES" though - "CREATE (DATABASE|SCHEMA) [IF
> NOT
> > > EXISTS] database_name ...  [*WITH DBPROPERTIES* (k=v, ...)];" [1]
> > >
> > >   The same applies to table ddl. The proposed alter table ddl comes
> with
> > > "SET *PROPERTIES* (...)", however, Flink's existing table create ddl
> > since
> > > 1.9 [2] doesn't have "PROPERTIES" keyword. As opposed to Hive's syntax,
> > > both create and alter table ddl comes with "TBLPROPERTIES" [1].
> > >
> > >   I feel it's better to be consistent among our DDLs. One option is to
> > > just remove the "PROPERTIES" and "DBPROPERTIES" keywords in proposed
> > syntax.
> > >
> > > [1]
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL
> > > [2]
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sql.html#specifying-a-ddl
> > >
> > > On Tue, Nov 5, 2019 at 12:54 PM Peter Huang <
> huangzhenqiu0...@gmail.com>
> > > wrote:
> > >
> > >> +1 for the enhancement.
> > >>
> > >> On Tue, Nov 5, 2019 at 11:04 AM Xuefu Z  wrote:
> > >>
> > >>> +1 to the long missing feature in Flink SQL.
> > >>>
> > >>> On Tue, Nov 5, 2019 at 6:32 AM Terry Wang 
> wrote:
> > >>>
> >  Hi all,
> > 
> >  I would like to start the vote for FLIP-69[1] which is discussed and
> >  reached consensus in the discussion thread[2].
> > 
> >  The vote will be open for at least 72 hours. I'll try to close it by
> >  2019-11-08 14:30 UTC, unless there is an objection or not enough
> > votes.
> > 
> >  [1]
> > 
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement
> >  <
> > 
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement
> > >
> >  [2]
> > 
> > >>>
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html
> >  <
> > 
> > >>>
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html
> > >
> >  Best,
> >  Terry Wang
> > 
> > 
> > 
> > 
> > >>>
> > >>> --
> > >>> Xuefu Zhang
> > >>>
> > >>> "In Honey We Trust!"
> > >>>
> > >>
> >
> >
>


-- 
Best regards!
Rui Li

How long is the flink sql task state default ttl?

[LakeShen]

Hi community， as I know I can use idle state retention time to clear the
flink sql task state,I have a question is that how long the flink sql task
state default ttl is . Thanks

[jira] [Created] (FLINK-14650) Thread safety issue in the piece of code example of dev/stream/testing document

[Zhenghua Gao (Jira)]

Zhenghua Gao created FLINK-14650:


 Summary: Thread safety issue in the piece of code example of 
dev/stream/testing document
 Key: FLINK-14650
 URL: https://issues.apache.org/jira/browse/FLINK-14650
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Reporter: Zhenghua Gao


As mentioned by Gilles in user ML[1], the piece of code example has thread 
safety issue.

One possibility is use Collections.synchronizedList() to create a thread-safety 
list and remove the synchronized keyword.

[1] 
[http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Documentation-issue-maybe-td30929.html]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14649) Flatten all the connector properties keys to make it easy to configure in DDL

[Jark Wu (Jira)]

Jark Wu created FLINK-14649:
---

 Summary: Flatten all the connector properties keys to make it easy 
to configure in DDL
 Key: FLINK-14649
 URL: https://issues.apache.org/jira/browse/FLINK-14649
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Reporter: Jark Wu


There are some connector properties are lists. For example, Kafka connector 
specific properties have to been set in the following way:


{code}
 'connector.properties.0.key' = 'zookeeper.connect',
  'connector.properties.0.value' = 'localhost:2181',
  'connector.properties.1.key' = 'bootstrap.servers',
  'connector.properties.1.value' = 'localhost:9092',
  'connector.properties.2.key' = 'group.id',
  'connector.properties.2.value' = 'testGroup',
{code}

It woule be better to flatten the properties for user friendly, for example: 

{code}
 'connector.zookeeper.connect' = 'localhost:2181',
  'connector.bootstrap.servers' = 'localhost:9092',
  'connector.group.id' = 'testGroup',
{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14648) When using udaf, the startup task has a “Cannot determine simple type name 'com' ” exception

[Ruiliang Li (Jira)]

Ruiliang Li created FLINK-14648:
---

 Summary: When using udaf, the startup task has a “Cannot determine 
simple type name 'com' ” exception
 Key: FLINK-14648
 URL: https://issues.apache.org/jira/browse/FLINK-14648
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API
Affects Versions: 1.7.2
Reporter: Ruiliang Li
 Attachments: image-2019-11-07-14-58-17-172.png

My flink streaming job use a udaf, set 60 parallelisms，submit job in yarn 
cluster mode，and then happens every time I start.

!image-2019-11-07-14-58-17-172.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14647) Improve the exception message when required property is not matched

[Jark Wu (Jira)]

Jark Wu created FLINK-14647:
---

 Summary: Improve the exception message when required property is 
not matched
 Key: FLINK-14647
 URL: https://issues.apache.org/jira/browse/FLINK-14647
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Reporter: Jark Wu
 Fix For: 1.10.0


Currently, all the required properties should exist and match, otherwise, 
{{NoMatchingTableFactoryException}} will be thrown.

For example, if we have {{connector.type=hbase,  connector.versions=1.1.1}}, 
the following exception will be thrown.


{code}
org.apache.flink.table.api.NoMatchingTableFactoryException: Could not find a 
suitable table factory for 'org.apache.flink.addons.hbase.HBaseTableFactory' in
the classpath.

Reason: No context matches.

The following properties are requested:
connector.type=hbase
connector.version=1.1.1
{code}

It's hard to know the problem is the version is wrong. A quick fixing is move 
version out of {{requiredContext()}} if we only support one version and throw a 
readable exception in {{ConnectorDescriptorValidator#validate}}. 

However, for the multiple-version connectors, e.g. Kafka, maybe we should 
improve the design of {{TableFactory}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14646) Check null for key in KeyGroupStreamPartitioner

[Jiayi Liao (Jira)]

Jiayi Liao created FLINK-14646:
--

 Summary: Check null for key in KeyGroupStreamPartitioner
 Key: FLINK-14646
 URL: https://issues.apache.org/jira/browse/FLINK-14646
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / State Backends
Affects Versions: 1.9.0
Reporter: Jiayi Liao


We should check null for {{key}} in {{selectChannel}} function in 
{{KeyGroupStreamPartitioner}} because {{KeyGroupRangeAssignment}} does not 
allow null key for assignment. And if we produce a null key in 
{{KeyGroupStreamPartitioner}}, the exception is not clear enough.


{code:java}
Caused by: java.lang.NullPointerException
at 
org.apache.flink.runtime.state.KeyGroupRangeAssignment.assignToKeyGroup(KeyGroupRangeAssignment.java:60)
at 
org.apache.flink.runtime.state.KeyGroupRangeAssignment.assignKeyToParallelOperator(KeyGroupRangeAssignment.java:49)
at 
org.apache.flink.streaming.runtime.partitioner.KeyGroupStreamPartitioner.selectChannel(KeyGroupStreamPartitioner.java:58)
at 
org.apache.flink.streaming.runtime.partitioner.KeyGroupStreamPartitioner.selectChannel(KeyGroupStreamPartitioner.java:32)
at 
org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:120)
at 
org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:110)
... 9 more
{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14645) Data types defined in DDL will lose precision and nullability when converting to properties

[Jark Wu (Jira)]

Jark Wu created FLINK-14645:
---

 Summary: Data types defined in DDL will lose precision and 
nullability when converting to properties
 Key: FLINK-14645
 URL: https://issues.apache.org/jira/browse/FLINK-14645
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Reporter: Jark Wu
 Fix For: 1.10.0


Currently, data types defined in DDL will be converted to {{TypeInformation}} 
and use {{TypeStringUtils}} to serialize/deserialize which will lose the 
precision and nullablitiy information. 

We can use {{LogicalType#asSerializableString}} and {{LogicalTypeParser}} to 
serialize/deserialize data types which keeps all the information. But we need 
to figure out how to keep compability with previous versions.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14644) Improve CREATE TABLE DDL WITH properties

[Jark Wu (Jira)]

Jark Wu created FLINK-14644:
---

 Summary: Improve CREATE TABLE DDL WITH properties
 Key: FLINK-14644
 URL: https://issues.apache.org/jira/browse/FLINK-14644
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / API
Reporter: Jark Wu
 Fix For: 1.10.0


This is an umbrella issue to collect problems of current WITH properties. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14643) Deprecate metric `fullRestarts`

[Zhu Zhu (Jira)]

Zhu Zhu created FLINK-14643:
---

 Summary: Deprecate metric `fullRestarts`
 Key: FLINK-14643
 URL: https://issues.apache.org/jira/browse/FLINK-14643
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.10.0
Reporter: Zhu Zhu
 Fix For: 1.10.0


FLINK-14164 introduces a metric 'numberOfRestarts' that counts all kinds of 
restarts. 
The metric 'fullRestarts' is superseded and we should deprecate it for future 
removal.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14642) Flink TupleSerializer and CaseClassSerializer shoud support copy NULL values

[Victor Wong (Jira)]

Victor Wong created FLINK-14642:
---

 Summary: Flink TupleSerializer and CaseClassSerializer shoud 
support copy NULL values
 Key: FLINK-14642
 URL: https://issues.apache.org/jira/browse/FLINK-14642
 Project: Flink
  Issue Type: Bug
  Components: API / Type Serialization System
Affects Versions: 1.9.1
Reporter: Victor Wong


Currently, TupleSerializer and CaseCassSerializer do not support serialize NULL 
values, which I think is acceptable. But not supporting to copy NULL values 
will cause the following codes to throw an exception, which I think is not 
matched with users' expectations.

*codes:*

 
{code:java}
stream.map(xxx).filter(_ != null).xxx //the return type of the map function is 
Tuple and it may return null{code}
 

*exception info:*

 
{code:java}
Caused by: java.lang.NullPointerExceptionCaused by: 
java.lang.NullPointerException at 
org.apache.flink.api.scala.typeutils.CaseClassSerializer.copy(CaseClassSerializer.scala:92)
 at 
org.apache.flink.api.scala.typeutils.CaseClassSerializer.copy(CaseClassSerializer.scala:32)
 at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:635)
{code}
 

*suggestion:*

Can we make the `copy` method of TupleSerializer/CaseClassSerializer to handle 
NULL values? e.g.
{code:java}
// org.apache.flink.api.scala.typeutils.CaseClassSerializer#copy
def copy(from: T): T = {
  // handle NULL values.
  if(from == null) {
return from
  }
  initArray()
  var i = 0
  while (i < arity) {
fields(i) = 
fieldSerializers(i).copy(from.productElement(i).asInstanceOf[AnyRef])
i += 1
  }
  createInstance(fields)
}
{code}
 

 

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14641) Fix description of metric `fullRestarts`

[Zhu Zhu (Jira)]

Zhu Zhu created FLINK-14641:
---

 Summary: Fix description of metric `fullRestarts`
 Key: FLINK-14641
 URL: https://issues.apache.org/jira/browse/FLINK-14641
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.9.2
Reporter: Zhu Zhu
 Fix For: 1.10.0, 1.9.2


The metric `fullRestarts` counts both full restarts and fine grained restarts 
since 1.9.2.
We should update the metric description doc accordingly.

We need to pointing out the the metric counts full restarts in 1.9.1 or earlier 
versions, and turned to count all kinds of restarts since 1.9.2.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Flunk savepoin(checkpoint) load api or debug

[Congxian Qiu]

Hi,
If you just want to debug, maybe you can do this in UT class in module
flink-runtime :) so that you do not need to handle the dependency problem,
and access problem.

Best,
Congxian


Jark Wu  于2019年11月6日周三 下午3:39写道：

> Btw, user questions should be asked in user@f.a.o or user-zh@f.a.o. The
> dev
> ML is mainly used to discuss development.
>
> Best,
> Jark
>
> On Wed, 6 Nov 2019 at 15:36, Jark Wu  wrote:
>
> > Hi,
> >
> > Savepoint.load(env, path) is in state processor API library, you should
> > add the following dependency in your project.
> >
> > 
> >   org.apache.flink
> >   flink-state-processor-api_2.11
> >   1.9.1
> > 
> >
> >
> > You can see the docuementation for more detailed instructions [1].
> >
> > Best,
> > Jark
> >
> > [1]:
> >
> https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/libs/state_processor_api.html
> >
> > On Wed, 6 Nov 2019 at 09:21, qq <471237...@qq.com> wrote:
> >
> >> Hi all,
> >>I want to load checkpoint or savepoint metadata on dev . in this case
> >> , I want to debug saved checkpoint metadata. And I knew flink provided a
> >> api which is Savepoint.load(env, path), but I can’t find it and can’t
> use
> >> it. Anyone who know about this ? Could you help me ? Thanks very much;
> >>
> >>
>

Re: RocksDB state on HDFS seems not being cleanned up

[shuwen zhou]

Hi Yun,
Thank you for your detailed explanation,It brings me a lot to research. I
think
1. I should try reduce number of "state.checkpoints.num-retained", maybe to
3, which could decrease amount of shared folder.
2. Does Flink 1.9.0 has the possibility of orphan files? Seems the answer
is yes, maybe. I could have use the state process API you mentioned to
figure it out and get back to you.
3. I have a look in file
/flink/c344b61c456af743e4568a70b626837b/chk-172/_metadata,
there are a lot file names
like 
hdfs://hadoop/flink/c344b61c456af743e4568a70b626837b/shared/e9e10c8a-6d73-48e4-9e17-45838d276b03,
sum those file's size up is the total size of each chekpoint, am I correct?
4. My checkpoint interval is 16 minutes.





On Wed, 6 Nov 2019 at 15:57, Yun Tang  wrote:

> Hi Shuwen
>
>
>
> Since you just have 10 “chk-“ folders as expected and when subsuming
> checkpoints, the “chk-” folder would be removed after we successfully
> removed shared state [1]. That is to say, I think you might not have too
> many orphan states files left. To ensure this, you could use state process
> API [2] to load your checkpoints and compare all the files under “shared”
> folder to see whether there existed too many orphan files. If this is true,
> we might think of the custom compaction filter future of FRocksDB.
>
>
>
> Secondly, your judgment of “20GB each checkpoint” might not be accurate
> when RocksDB incremental checkpoint is enabled, the UI showed is only the
> incremental size [3], I suggest you to count your files’s size within your
> checkpoint meta to know the accurate checkpoint size for each checkpoint.
>
>
>
> Last but not least, RocksDB’s future of compaction filter to delete
> expired data only happened during compaction [4], I’m afraid you might need
> to look up your rocksDB’s LOG file to see the frequency of compaction on
> task managers. And I think the increasing size might be related with the
> interval of your checkpoints, what the interval when you executing
> checkpoints?
>
>
>
>
>
> [1]
> https://github.com/apache/flink/blob/2ea14169a1997434d45d6f1da6dfe9acd6bd8da3/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CompletedCheckpoint.java#L264
>
> [2]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html
>
> [3] https://issues.apache.org/jira/browse/FLINK-13390
>
> [4]
> https://github.com/facebook/rocksdb/blob/834feaff05a4bf7ae49c736305d5eb180aed4011/include/rocksdb/compaction_filter.h#L61
>
>
>
> Best
>
> Yun Tang
>
>
>
> *From: *shuwen zhou 
> *Date: *Wednesday, November 6, 2019 at 12:02 PM
> *To: *dev , Yun Tang , Till
> Rohrmann 
> *Subject: *Re: RocksDB state on HDFS seems not being cleanned up
>
>
>
> Hi Yun and Till,
>
> Thank you for your response.
>
> For @Yun
> 1. No, I just renamed the checkpoint directory name since the directory
> name contains company data. Sorry for the confusion.
>
> 2. Yes, I set
>
> *state.checkpoints.num-retained: *10
> *state.backend.rocksdb.predefined-options: *FLASH_SSD_OPTIMIZED
>
> In flink.conf
>
> I was expecting, shared folder will no longer contains outdated state, since 
> my TTL is set to 30 mins, I shouldn't have seen date older than 1 day. 
> However I could still see those outdated data in shared folder
>
> For example, current time is 2019-11-06 03:58:00 UTC, I could see following 
> file on HDFS
>
> 65.1 M 2019-11-04 17:58
> /flink/checkpoint/c344b61c456af743e4568a70b626837b/shared/03dea380-758b-4d52-b335-5e6318ba6c40
> 2.1 K 2019-11-04 17:28
> /flink/checkpoint/c344b61c456af743e4568a70b626837b/shared/1205f112-f5ba-4516-ae32-1424afda08ac
> 65.1 M 2019-11-04 17:58
> /flink/checkpoint/c344b61c456af743e4568a70b626837b/shared/2298e34d-8cdc-4f8a-aac0-76cf4b9ac0f5
> 65.1 M 2019-11-04 17:58
> /flink/checkpoint/c344b61c456af743e4568a70b626837b/shared/25e58576-f86f-4ac9-83b8-08ce0be036c4
>
> 65.1 M 2019-11-05 17:42
> /flink/checkpoint/c344b61c456af743e4568a70b626837b/shared/27031a93-3ae5-4247-a751-62552c29f325
>
>
> 3.I actually mean that, only latest 10 checkpoint containing full state will 
> be retained on HDFS. In my case, around 20G for each checkpoint. In such way 
> I could have control on how much data was stored on HDFS, Rather than having 
> a increasing shared folder.
>
> But it takes a lot of time to store full state on HDFS. Thus I would still 
> like to use incremental.
>
>
>
>
>
> For @Till
>
> I would have a try on cleanupInRocksdbCompactFilter to see if it works. Thank 
> you.
>
>
>
> On Wed, 6 Nov 2019 at 01:50, Yun Tang  wrote:
>
> @Till Rohrmann , I think just set `cleanupInBackground()` should be enough
> for RocksDB to clean up in compaction filter after Flink-1.9.0 [1]
>
> @Shuwen , I have several questions for your behavior:
> 1. Is the ` flink-chk743e4568a70b626837b` real folder for checkpoints? I
> don't think a job-id would act like this.
> 2. why you have 10 checkpoints left under checkpoint folder, did you
> configure the retained checkpoints as 10?
> 3. what do you mean "while I 

Re: [DISCUSS] Stateful Functions - in which form to contribute? (same or different repository)

[Bowen Li]

+1 for separate repo right now for all the good discussed

On Wed, Nov 6, 2019 at 3:35 PM Becket Qin  wrote:

> +1 on having a separate repository.
>
> I am always an advocate of separate repositories. All the substantial
> benefits of doing that are quite convincing. The only reason we might want
> to make Stateful Function in main repo is probably because it looks just
> like CEP, Gelly and other libraries that are for specific use cases. It is
> kind of philosophical. But given Stateful Function seems no longer a "data
> processing" use case, it looks also reasonable to treat it differently. And
> as others mentioned, we can always put it into the main repo later if we
> want to.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Wed, Nov 6, 2019 at 6:25 PM Stephan Ewen  wrote:
>
> > Are still open questions here?
> >
> > Or can I treat this discussion as converged in the sense of concluding
> > that:
> >   - we start initially with a separate repository to allow for individual
> > releases in the early stages
> >   - we later revisit this discussion once the project is a bit further
> > along and more converged
> >
> > Best,
> > Stephan
> >
> >
> > On Wed, Oct 16, 2019 at 3:03 PM Stephan Ewen  wrote:
> >
> > > Whether the side project will be overlooked of not will depends a lot
> on
> > > how we integrate it with the current Flink website and documentation.
> > >
> > > I would think that a separate repository is not necessarily a big
> problem
> > > there.
> > > It might also help, because a link to that repo shows prominently that
> > > particular angle of the project (application development), rather than
> it
> > > being an API hidden between 100 modules.
> > >
> > > On Wed, Oct 16, 2019 at 10:02 AM Timo Walther 
> > wrote:
> > >
> > >> Hi Stephan,
> > >>
> > >> +1 for keeping it in a separate repository for fast release cycles and
> > >> stability until it is mature enough. But we should definitely merge it
> > >> back to the core repo also for marketing reasons.
> > >>
> > >> IMHO side projects tend to be overlooked by the outside world even
> > >> though they are great technology.
> > >>
> > >> Would we still document the code in our main documentation or on a
> > >> separate website?
> > >>
> > >> Thanks,
> > >> Timo
> > >>
> > >>
> > >> On 16.10.19 09:15, Aljoscha Krettek wrote:
> > >> > I would keep statefun in a separate repo in the beginning, for the
> > >> reasons you mentioned.
> > >> >
> > >> > Best,
> > >> > Aljoscha
> > >> >
> > >> >> On 15. Oct 2019, at 23:40, Flavio Pompermaier <
> pomperma...@okkam.it>
> > >> wrote:
> > >> >>
> > >> >> Definitely on the same page..+1 to keep it in a separate repo (at
> > least
> > >> >> until the cose becomes "stable" and widely adopted from the
> > community)
> > >> >>
> > >> >> Il Mar 15 Ott 2019, 23:17 Stephan Ewen  ha
> > scritto:
> > >> >>
> > >> >>> Hi Flink folks!
> > >> >>>
> > >> >>> After the positive reaction to the contribution proposal for
> > Stateful
> > >> >>> Functions, I would like to kick off the discussion for the big
> > >> question: In
> > >> >>> which form should it go into Flink?
> > >> >>>
> > >> >>> Before jumping into the "repository" question directly, let's get
> > some
> > >> >>> clarity on what would be our high-level goal with this project and
> > the
> > >> >>> contribution.
> > >> >>> My thinking so far was:
> > >> >>>
> > >> >>>   - Stateful Functions is a way for Flink and stream processing to
> > >> become
> > >> >>> applicable for more general application development. That is a
> > chance
> > >> to
> > >> >>> grow our community to a new crowd of developers.
> > >> >>>
> > >> >>>   - While adding this to Flink gives synergies with the runtime it
> > >> build on
> > >> >>> top of, it makes sense to offer the new developers a lightweight
> way
> > >> to get
> > >> >>> involved. Simple setup, easy contributions.
> > >> >>>
> > >> >>>   - This is a new project, the API and many designs are not frozen
> > at
> > >> this
> > >> >>> point and may still change heavily.
> > >> >>> To become really good, the project needs to still make a bunch
> > of
> > >> >>> iterations (no pun intended) and change many things quickly.
> > >> >>>
> > >> >>>   - The Stateful Functions project will likely try to release very
> > >> >>> frequently in its early days, to improve quickly and gather
> feedback
> > >> fast.
> > >> >>> Being bound to Flink core release cycle would hurt here.
> > >> >>>
> > >> >>>
> > >> >>> I believe that with all those goals, adding Stateful Functions to
> > the
> > >> Flink
> > >> >>> core repository would not make sense. Flink core has processes
> that
> > >> make
> > >> >>> sense for an established project that needs to guarantee
> stability.
> > >> These
> > >> >>> processes are simply prohibitive for new projects to develop.
> > >> >>> In addition, the Flink main repository is gigantic, has a build
> > >> system and
> > >> >>> CI system that cannot handle the size of the project any more. Not
> > >> the best

[jira] [Created] (FLINK-14640) Change Type of Field currentExecutions from ConcurrentHashMap to HashMap

[vinoyang (Jira)]

vinoyang created FLINK-14640:


 Summary: Change Type of Field currentExecutions from 
ConcurrentHashMap to HashMap
 Key: FLINK-14640
 URL: https://issues.apache.org/jira/browse/FLINK-14640
 Project: Flink
  Issue Type: Sub-task
Reporter: vinoyang


After FLINK-11417, we made ExecutionGraph be a single-thread mode. It will no 
longer be plagued by concurrency issues. So, we can degenerate the current 
ConcurrentHashMap type of currentExecutions to a normal HashMap type.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14639) Fix the document of Metircs that has an error for `User Scope`

[hehuiyuan (Jira)]

hehuiyuan created FLINK-14639:
-

 Summary: Fix the document of Metircs  that has an error for `User 
Scope` 
 Key: FLINK-14639
 URL: https://issues.apache.org/jira/browse/FLINK-14639
 Project: Flink
  Issue Type: Wish
  Components: Documentation
 Environment: Hi , i think it should be `MetricGroup{{#addGroup(String 
key, String value)}}`
Reporter: hehuiyuan
 Attachments: image-2019-11-07-10-42-37-862.png

!image-2019-11-07-10-42-37-862.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Stateful Functions - in which form to contribute? (same or different repository)

[Becket Qin]

+1 on having a separate repository.

I am always an advocate of separate repositories. All the substantial
benefits of doing that are quite convincing. The only reason we might want
to make Stateful Function in main repo is probably because it looks just
like CEP, Gelly and other libraries that are for specific use cases. It is
kind of philosophical. But given Stateful Function seems no longer a "data
processing" use case, it looks also reasonable to treat it differently. And
as others mentioned, we can always put it into the main repo later if we
want to.

Thanks,

Jiangjie (Becket) Qin

On Wed, Nov 6, 2019 at 6:25 PM Stephan Ewen  wrote:

> Are still open questions here?
>
> Or can I treat this discussion as converged in the sense of concluding
> that:
>   - we start initially with a separate repository to allow for individual
> releases in the early stages
>   - we later revisit this discussion once the project is a bit further
> along and more converged
>
> Best,
> Stephan
>
>
> On Wed, Oct 16, 2019 at 3:03 PM Stephan Ewen  wrote:
>
> > Whether the side project will be overlooked of not will depends a lot on
> > how we integrate it with the current Flink website and documentation.
> >
> > I would think that a separate repository is not necessarily a big problem
> > there.
> > It might also help, because a link to that repo shows prominently that
> > particular angle of the project (application development), rather than it
> > being an API hidden between 100 modules.
> >
> > On Wed, Oct 16, 2019 at 10:02 AM Timo Walther 
> wrote:
> >
> >> Hi Stephan,
> >>
> >> +1 for keeping it in a separate repository for fast release cycles and
> >> stability until it is mature enough. But we should definitely merge it
> >> back to the core repo also for marketing reasons.
> >>
> >> IMHO side projects tend to be overlooked by the outside world even
> >> though they are great technology.
> >>
> >> Would we still document the code in our main documentation or on a
> >> separate website?
> >>
> >> Thanks,
> >> Timo
> >>
> >>
> >> On 16.10.19 09:15, Aljoscha Krettek wrote:
> >> > I would keep statefun in a separate repo in the beginning, for the
> >> reasons you mentioned.
> >> >
> >> > Best,
> >> > Aljoscha
> >> >
> >> >> On 15. Oct 2019, at 23:40, Flavio Pompermaier 
> >> wrote:
> >> >>
> >> >> Definitely on the same page..+1 to keep it in a separate repo (at
> least
> >> >> until the cose becomes "stable" and widely adopted from the
> community)
> >> >>
> >> >> Il Mar 15 Ott 2019, 23:17 Stephan Ewen  ha
> scritto:
> >> >>
> >> >>> Hi Flink folks!
> >> >>>
> >> >>> After the positive reaction to the contribution proposal for
> Stateful
> >> >>> Functions, I would like to kick off the discussion for the big
> >> question: In
> >> >>> which form should it go into Flink?
> >> >>>
> >> >>> Before jumping into the "repository" question directly, let's get
> some
> >> >>> clarity on what would be our high-level goal with this project and
> the
> >> >>> contribution.
> >> >>> My thinking so far was:
> >> >>>
> >> >>>   - Stateful Functions is a way for Flink and stream processing to
> >> become
> >> >>> applicable for more general application development. That is a
> chance
> >> to
> >> >>> grow our community to a new crowd of developers.
> >> >>>
> >> >>>   - While adding this to Flink gives synergies with the runtime it
> >> build on
> >> >>> top of, it makes sense to offer the new developers a lightweight way
> >> to get
> >> >>> involved. Simple setup, easy contributions.
> >> >>>
> >> >>>   - This is a new project, the API and many designs are not frozen
> at
> >> this
> >> >>> point and may still change heavily.
> >> >>> To become really good, the project needs to still make a bunch
> of
> >> >>> iterations (no pun intended) and change many things quickly.
> >> >>>
> >> >>>   - The Stateful Functions project will likely try to release very
> >> >>> frequently in its early days, to improve quickly and gather feedback
> >> fast.
> >> >>> Being bound to Flink core release cycle would hurt here.
> >> >>>
> >> >>>
> >> >>> I believe that with all those goals, adding Stateful Functions to
> the
> >> Flink
> >> >>> core repository would not make sense. Flink core has processes that
> >> make
> >> >>> sense for an established project that needs to guarantee stability.
> >> These
> >> >>> processes are simply prohibitive for new projects to develop.
> >> >>> In addition, the Flink main repository is gigantic, has a build
> >> system and
> >> >>> CI system that cannot handle the size of the project any more. Not
> >> the best
> >> >>> way to start expanding into a new community.
> >> >>>
> >> >>> In some sense, Stateful Functions could make sense as an independent
> >> >>> project, but it is so tightly coupled to Flink right now that I
> think
> >> an
> >> >>> even better fit is a separate repository in Flink.
> >> >>> Think Hive and Hadoop in the early days. That way, we get the
> synergy
> >> >>> between the two (the same 

[jira] [Created] (FLINK-14638) move functions related docs to a new single, unified dir /dev/table/functions

[Bowen Li (Jira)]

Bowen Li created FLINK-14638:


 Summary: move functions related docs to a new single, unified dir 
/dev/table/functions
 Key: FLINK-14638
 URL: https://issues.apache.org/jira/browse/FLINK-14638
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Reporter: Bowen Li
Assignee: Bowen Li
 Fix For: 1.10.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-69: Flink SQL DDL Enhancement

[Bowen Li]

Thanks.

As Terry and I discussed offline yesterday, we added a new section to
explain the detailed implementation plan.

+1 (binding) from me.

Bowen

On Tue, Nov 5, 2019 at 6:33 PM Terry Wang  wrote:

> Hi Bowen:
> Thanks for your feedback.
> Your opinion convinced me and I just remove the section about catalog
> create statement and also remove `DBPROPERTIES` `PROPERTIES` from alter
> DDLs.
> Open to more comments or votes :) !
>
> Best,
> Terry Wang
>
>
>
> > 2019年11月6日 07:22，Bowen Li  写道：
> >
> > Hi Terry,
> >
> > I went over the FLIP in detail again. The FLIP mostly LGTM. A couple
> issues:
> >
> > - since we on't plan to support catalog ddl, can you remove them from the
> > FLIP?
> > - I found there are some discrepancies in proposed database and table
> DDLs.
> >  For db ddl, the create db syntax proposes specifying k-v properties
> > following "WITH". However, alter db ddl comes with a keyword
> "DBPROPERTIES":
> >
> > CREATE  DATABASE [ IF NOT EXISTS ] [ catalogName.] dataBaseName [ COMMENT
> > database_comment ]
> > [*WITH *( name=value [, name=value]*)]
> >
> >
> > ALTER  DATABASE  [ catalogName.] dataBaseName SET *DBPROPERTIES* (
> > name=value [, name=value]*)
> >
> >
> >IIUIC, are you borrowing syntax from Hive? Note that Hive's db create
> > ddl comes with "DBPROPERTIES" though - "CREATE (DATABASE|SCHEMA) [IF NOT
> > EXISTS] database_name ...  [*WITH DBPROPERTIES* (k=v, ...)];" [1]
> >
> >   The same applies to table ddl. The proposed alter table ddl comes with
> > "SET *PROPERTIES* (...)", however, Flink's existing table create ddl
> since
> > 1.9 [2] doesn't have "PROPERTIES" keyword. As opposed to Hive's syntax,
> > both create and alter table ddl comes with "TBLPROPERTIES" [1].
> >
> >   I feel it's better to be consistent among our DDLs. One option is to
> > just remove the "PROPERTIES" and "DBPROPERTIES" keywords in proposed
> syntax.
> >
> > [1] https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL
> > [2]
> >
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sql.html#specifying-a-ddl
> >
> > On Tue, Nov 5, 2019 at 12:54 PM Peter Huang 
> > wrote:
> >
> >> +1 for the enhancement.
> >>
> >> On Tue, Nov 5, 2019 at 11:04 AM Xuefu Z  wrote:
> >>
> >>> +1 to the long missing feature in Flink SQL.
> >>>
> >>> On Tue, Nov 5, 2019 at 6:32 AM Terry Wang  wrote:
> >>>
>  Hi all,
> 
>  I would like to start the vote for FLIP-69[1] which is discussed and
>  reached consensus in the discussion thread[2].
> 
>  The vote will be open for at least 72 hours. I'll try to close it by
>  2019-11-08 14:30 UTC, unless there is an objection or not enough
> votes.
> 
>  [1]
> 
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement
>  <
> 
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement
> >
>  [2]
> 
> >>>
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html
>  <
> 
> >>>
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-69-Flink-SQL-DDL-Enhancement-td33090.html
> >
>  Best,
>  Terry Wang
> 
> 
> 
> 
> >>>
> >>> --
> >>> Xuefu Zhang
> >>>
> >>> "In Honey We Trust!"
> >>>
> >>
>
>

[jira] [Created] (FLINK-14637) Introduce framework off heap memory config

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-14637:
---

 Summary: Introduce framework off heap memory config
 Key: FLINK-14637
 URL: https://issues.apache.org/jira/browse/FLINK-14637
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Task
Reporter: Andrey Zagrebin
 Fix For: 1.10.0


At the moment after 
[-FLINK-13982-|https://issues.apache.org/jira/browse/FLINK-13982], when we do 
not account for adhoc direct memory allocations for Flink framework (except 
network buffers) or done by some used libraries used in Flink. In general, we 
expect this allocations to stay under a certain reasonably low limit but we 
have to have some margin for them so that JVM direct memory limit is not 
exactly equal to network buffers and does not fail. We can address it by 
introducing framework off heap memory config option.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14636) Handle schedule mode LAZY_FROM_SOURCES_WITH_BATCH_SLOT_REQUEST correctly in DefaultScheduler

[Gary Yao (Jira)]

Gary Yao created FLINK-14636:


 Summary: Handle schedule mode 
LAZY_FROM_SOURCES_WITH_BATCH_SLOT_REQUEST correctly in DefaultScheduler
 Key: FLINK-14636
 URL: https://issues.apache.org/jira/browse/FLINK-14636
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.10.0
Reporter: Gary Yao
Assignee: Gary Yao
 Fix For: 1.10.0


It should be possible to schedule a job with 
{{ScheduleMode.LAZY_FROM_SOURCES_WITH_BATCH_SLOT_REQUEST}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14635) Don't use relocated imports in Kinesis End-2-End Tests

[Stephan Ewen (Jira)]

Stephan Ewen created FLINK-14635:


 Summary: Don't use relocated imports in Kinesis End-2-End Tests
 Key: FLINK-14635
 URL: https://issues.apache.org/jira/browse/FLINK-14635
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kinesis
Affects Versions: 1.9.1
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 1.10.0


Using relocated imports in {{KinesisPubsubClient}} makes it not possible to 
build the code in the IDE any more.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14634) Metric Option of 'metrics.scope.delimiter' won't work probably.

[Eric Lee (Jira)]

Eric Lee created FLINK-14634:


 Summary: Metric Option of 'metrics.scope.delimiter' won't work 
probably.
 Key: FLINK-14634
 URL: https://issues.apache.org/jira/browse/FLINK-14634
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Metrics
 Environment: Flink 1.9.0

JDK 1.8
Reporter: Eric Lee
 Attachments: 5506F8C7-ACCA-44E8-9BA9-D6023E02B633.png, 
C99D8A02-9C7A-4809-B151-93CE166E92DC.png

In the `master` branch, when initializing the metricRegistryConfiguration, the 
configuration of `metrics.scope.delimiter` doesn't work probably because the 
user-defined delimiter is not used in the process of constructing `ScopeFormat` 
which uses the fixed character - `.` as the delimiter.

1. Attachment 1 shows that in the process of constructing `ScopeFormat`, `.` is 
used as the default delimiter rather than user-defined delimiter.

2. Attachment 2 illustrates that the wrong order of using user-defined 
delimiter.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14633) Account for netty direct allocations in direct memory limit (Queryable state)

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-14633:
---

 Summary: Account for netty direct allocations in direct memory 
limit (Queryable state)
 Key: FLINK-14633
 URL: https://issues.apache.org/jira/browse/FLINK-14633
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Queryable State, Runtime / Task
Reporter: Andrey Zagrebin
 Fix For: 1.10.0


At the moment after 
[-FLINK-13982-|https://issues.apache.org/jira/browse/FLINK-13982], when we 
calculate JVM direct memory limit, we do not account for direct allocations 
from netty arenas in org.apache.flink.queryablestate.network.NettyBufferPool.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14632) ExecutionGraphSchedulingTest.testSlotReleasingFailsSchedulingOperation deadlocks

[Gary Yao (Jira)]

Gary Yao created FLINK-14632:


 Summary: 
ExecutionGraphSchedulingTest.testSlotReleasingFailsSchedulingOperation deadlocks
 Key: FLINK-14632
 URL: https://issues.apache.org/jira/browse/FLINK-14632
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination, Tests
Affects Versions: 1.10.0
Reporter: Gary Yao


https://api.travis-ci.com/v3/job/253364947/log.txt

{noformat}
"main" #1 prio=5 os_prio=0 tid=0x7fac3c00b800 nid=0x956 waiting on 
condition [0x7fac449f1000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x8eb74600> (a 
java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at 
java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693)
at 
java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at 
java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729)
at 
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at 
org.apache.flink.runtime.executiongraph.ExecutionGraphSchedulingTest.testSlotReleasingFailsSchedulingOperation(ExecutionGraphSchedulingTest.java:501)
{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-14631) Account for netty direct allocations in direct memory limit

[Andrey Zagrebin (Jira)]

Andrey Zagrebin created FLINK-14631:
---

 Summary: Account for netty direct allocations in direct memory 
limit
 Key: FLINK-14631
 URL: https://issues.apache.org/jira/browse/FLINK-14631
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Task
Reporter: Andrey Zagrebin
 Fix For: 1.10.0


At the moment after FLINK-13982, when we calculate JVM direct memory limit, we 
only account for memory segment network buffers but not for direct allocations 
from netty arenas in org.apache.flink.runtime.io.network.netty.NettyBufferPool. 
We should include netty arenas into shuffle memory calculations.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Flink Avro Cloudera Registry (FLINK-14577)

[Gyula Fóra]

Hi Dawid,

In general I agree if we can provide a completely unified way of handling
this registries that would be great but I wonder if that makes sense in the
long term. While the cloudera schema registry only supports Avro at the
moment, it aims to support other formats in the future, and accessing this
functionality will probably rely on using those specific
serializer/deserializer implementations. This might not be a valid concern
at this point though :)

The reason why we went with wrapping the KafkaAvroDeserializer/Serializer
directly now, is that it was super simple to do and the current SchemaCoder
approach lacks a lot of flexibility/functionality.

The schema itself doesn't always come from the serialized data (I believe
in this case it is either stored in the serialized data or the kafka record
metadata) and also we want to be able to handle kafka message keys. I guess
these could be solved by making the deserialization logic Kafka specific
and exposing the ConsumerRecord but that would completely change the
current schemacoder related interfaces.

Cheers,
Gyula

On Wed, Nov 6, 2019 at 10:17 AM Dawid Wysakowicz 
wrote:

> Hi Gyula,
>
> I did not want to discourage this contribution. I do agree we should
> treat this connector equally to the confluent's schema registry. I just
> wanted to express my uncertainty about general approach to new
> connectors contributions. By no means I wanted to discourage this
> contribution.
>
> As for the second point. Do you mean that you are wrapping the
> KafkaAvroDeserializer/Serializer provided by cloudera/hortonworks schema
> registry?
>
> Personally I would very much prefer using the SchemaCoder approach. All
> schemas boil down to two steps. (De)Serializing the schema with registry
> specific protocol + (de)serializing the record itself. I think the
> approach with SchemaCoder has the benefit that we can optimize
> instantiation of Avro's readers and writers in a unified way. It's also
> easier to maintain as we have just a single point where the actual
> record (de)serialization happens. It also provides a unified way of
> instantiating the TypeInformation. Could you give some explanation why
> would you prefer not to use this approach?
>
> Best,
>
> Dawid
>
> On 05/11/2019 14:48, Gyula Fóra wrote:
> > Thanks Matyas for starting the discussion!
> > I think this would be a very valuable addition to Flink as many companies
> > are already using the Hortonworks/Cloudera registry and it would enable
> > them to connect to Flink easily.
> >
> > @Dawid:
> > Regarding the implementation this a much more lightweight connector than
> > what we have now for the Confluent registry and the PR you linked. This
> > wraps the cloudera registry directly, providing a very thin wrapper +
> some
> > enhanced functionality regarding handling of Kafka messages keys.
> >
> > As for the question of main repo outside, I would prefer this to be
> > included in the main repo, similar to the Confluent registry connector.
> > Unless we decide to move all of these connectors out I would like to
> take a
> > consistent approach.
> >
> > Cheers,
> > Gyula
> >
> >
> > On Tue, Nov 5, 2019 at 1:44 PM Dawid Wysakowicz 
> > wrote:
> >
> >> Hi Matyas,
> >>
> >> I think this would be a valuable addition. You may reuse some of the
> >> already available abstractions for writing avro deserialization schema
> >> based on a schema registry (have a look at RegistryDeserializationSchema
> >> and SchemaCoderProvider). There is also an opened PR for adding a
> >> similar serialization schema[1].
> >>
> >> The only concern is that I am not 100% sure what is the consensus on
> >> which connectors do we want to adapt into the main repository and which
> >> would we prefer to be hosted separately and included in the ecosystem
> >> webpage[2] (that I hope will be published soon).
> >>
> >> Whatever option will be preferred I could help review the code.
> >>
> >> Best,
> >>
> >> Dawid
> >>
> >> [1] https://github.com/apache/flink/pull/8371
> >>
> >> [2]
> >>
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html
> >>
> >> On 05/11/2019 12:40, Őrhidi Mátyás wrote:
> >>> Dear Flink Community!
> >>>
> >>> We have noticed a recent request for Hortonworks schema registry
> support
> >> (
> >>> FLINK-14577 ). We
> >> have
> >>> an implementation for it already, and we would be happy to contribute
> it
> >> to
> >>> Apache Flink.
> >>>
> >>> You can find the documentation below[1]. Let us know your thoughts!
> >>>
> >>> Best Regards,
> >>> Matyas
> >>>
> >>> [1] Flink Avro Cloudera Registry User Guide
> >>> ---
> >>>
> >>> Add the following dependency to use the schema registry integration:
> >>> 
> >>> org.apache.flink
> >>> flink-avro-cloudera-registry
> >>> ${flink.version}
> >>> 
> >>>
> >>>
> >>> The schema registry can be 

Re: [DISCUSS] Stateful Functions - in which form to contribute? (same or different repository)

[Stephan Ewen]

Are still open questions here?

Or can I treat this discussion as converged in the sense of concluding that:
  - we start initially with a separate repository to allow for individual
releases in the early stages
  - we later revisit this discussion once the project is a bit further
along and more converged

Best,
Stephan


On Wed, Oct 16, 2019 at 3:03 PM Stephan Ewen  wrote:

> Whether the side project will be overlooked of not will depends a lot on
> how we integrate it with the current Flink website and documentation.
>
> I would think that a separate repository is not necessarily a big problem
> there.
> It might also help, because a link to that repo shows prominently that
> particular angle of the project (application development), rather than it
> being an API hidden between 100 modules.
>
> On Wed, Oct 16, 2019 at 10:02 AM Timo Walther  wrote:
>
>> Hi Stephan,
>>
>> +1 for keeping it in a separate repository for fast release cycles and
>> stability until it is mature enough. But we should definitely merge it
>> back to the core repo also for marketing reasons.
>>
>> IMHO side projects tend to be overlooked by the outside world even
>> though they are great technology.
>>
>> Would we still document the code in our main documentation or on a
>> separate website?
>>
>> Thanks,
>> Timo
>>
>>
>> On 16.10.19 09:15, Aljoscha Krettek wrote:
>> > I would keep statefun in a separate repo in the beginning, for the
>> reasons you mentioned.
>> >
>> > Best,
>> > Aljoscha
>> >
>> >> On 15. Oct 2019, at 23:40, Flavio Pompermaier 
>> wrote:
>> >>
>> >> Definitely on the same page..+1 to keep it in a separate repo (at least
>> >> until the cose becomes "stable" and widely adopted from the community)
>> >>
>> >> Il Mar 15 Ott 2019, 23:17 Stephan Ewen  ha scritto:
>> >>
>> >>> Hi Flink folks!
>> >>>
>> >>> After the positive reaction to the contribution proposal for Stateful
>> >>> Functions, I would like to kick off the discussion for the big
>> question: In
>> >>> which form should it go into Flink?
>> >>>
>> >>> Before jumping into the "repository" question directly, let's get some
>> >>> clarity on what would be our high-level goal with this project and the
>> >>> contribution.
>> >>> My thinking so far was:
>> >>>
>> >>>   - Stateful Functions is a way for Flink and stream processing to
>> become
>> >>> applicable for more general application development. That is a chance
>> to
>> >>> grow our community to a new crowd of developers.
>> >>>
>> >>>   - While adding this to Flink gives synergies with the runtime it
>> build on
>> >>> top of, it makes sense to offer the new developers a lightweight way
>> to get
>> >>> involved. Simple setup, easy contributions.
>> >>>
>> >>>   - This is a new project, the API and many designs are not frozen at
>> this
>> >>> point and may still change heavily.
>> >>> To become really good, the project needs to still make a bunch of
>> >>> iterations (no pun intended) and change many things quickly.
>> >>>
>> >>>   - The Stateful Functions project will likely try to release very
>> >>> frequently in its early days, to improve quickly and gather feedback
>> fast.
>> >>> Being bound to Flink core release cycle would hurt here.
>> >>>
>> >>>
>> >>> I believe that with all those goals, adding Stateful Functions to the
>> Flink
>> >>> core repository would not make sense. Flink core has processes that
>> make
>> >>> sense for an established project that needs to guarantee stability.
>> These
>> >>> processes are simply prohibitive for new projects to develop.
>> >>> In addition, the Flink main repository is gigantic, has a build
>> system and
>> >>> CI system that cannot handle the size of the project any more. Not
>> the best
>> >>> way to start expanding into a new community.
>> >>>
>> >>> In some sense, Stateful Functions could make sense as an independent
>> >>> project, but it is so tightly coupled to Flink right now that I think
>> an
>> >>> even better fit is a separate repository in Flink.
>> >>> Think Hive and Hadoop in the early days. That way, we get the synergy
>> >>> between the two (the same community drives them) while letting both
>> move at
>> >>> their own speed.
>> >>> It would somehow mean two closely related projects shepherded by the
>> same
>> >>> community.
>> >>>
>> >>> It might be possible at a later stage to either merge this into Flink
>> core
>> >>> (once Stateful Functions is more settled) or even spin this out as a
>> >>> standalone Apache project, if that is how the community develops.
>> >>>
>> >>> That is my main motivation. It is not driven primarily by
>> technicalities
>> >>> like code versioning and dependencies, but much rather by what is the
>> best
>> >>> setup to develop this as Flink's way to expand its community towards
>> new
>> >>> users from a different background.
>> >>>
>> >>> Curious to hear if that makes sense to you.
>> >>>
>> >>> Best,
>> >>> Stephan
>> >>>
>>
>>

[jira] [Created] (FLINK-14630) Yarn tests no longer create jobmanager log files

[Till Rohrmann (Jira)]

Till Rohrmann created FLINK-14630:
-

 Summary: Yarn tests no longer create jobmanager log files
 Key: FLINK-14630
 URL: https://issues.apache.org/jira/browse/FLINK-14630
 Project: Flink
  Issue Type: Bug
  Components: Deployment / YARN, Tests
Affects Versions: 1.10.0
Reporter: Till Rohrmann
Assignee: Kostas Kloudas
 Fix For: 1.10.0


With FLINK-14502 we changed how log property files are being discovered by the 
{{YarnClusterDescriptor}}. In fact we moved the responsibility to the 
{{FlinkYarnSessionCli}}. This is problematic because the {{YarnTestBase}} 
creates for Yarn tests a configuration directory which contains the Flink 
configuration and the log property files. These files are no longer respected 
because the respective code path has been removed.

I think it is important to fix this problem so that our Yarn tests start to log 
again. Otherwise it becomes harder to debug Yarn problems and the tests can no 
longer check for the absence of exceptions in the logs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Flink Avro Cloudera Registry (FLINK-14577)

[Dawid Wysakowicz]

Hi Gyula,

I did not want to discourage this contribution. I do agree we should
treat this connector equally to the confluent's schema registry. I just
wanted to express my uncertainty about general approach to new
connectors contributions. By no means I wanted to discourage this
contribution.

As for the second point. Do you mean that you are wrapping the
KafkaAvroDeserializer/Serializer provided by cloudera/hortonworks schema
registry?

Personally I would very much prefer using the SchemaCoder approach. All
schemas boil down to two steps. (De)Serializing the schema with registry
specific protocol + (de)serializing the record itself. I think the
approach with SchemaCoder has the benefit that we can optimize
instantiation of Avro's readers and writers in a unified way. It's also
easier to maintain as we have just a single point where the actual
record (de)serialization happens. It also provides a unified way of
instantiating the TypeInformation. Could you give some explanation why
would you prefer not to use this approach?

Best,

Dawid

On 05/11/2019 14:48, Gyula Fóra wrote:
> Thanks Matyas for starting the discussion!
> I think this would be a very valuable addition to Flink as many companies
> are already using the Hortonworks/Cloudera registry and it would enable
> them to connect to Flink easily.
>
> @Dawid:
> Regarding the implementation this a much more lightweight connector than
> what we have now for the Confluent registry and the PR you linked. This
> wraps the cloudera registry directly, providing a very thin wrapper + some
> enhanced functionality regarding handling of Kafka messages keys.
>
> As for the question of main repo outside, I would prefer this to be
> included in the main repo, similar to the Confluent registry connector.
> Unless we decide to move all of these connectors out I would like to take a
> consistent approach.
>
> Cheers,
> Gyula
>
>
> On Tue, Nov 5, 2019 at 1:44 PM Dawid Wysakowicz 
> wrote:
>
>> Hi Matyas,
>>
>> I think this would be a valuable addition. You may reuse some of the
>> already available abstractions for writing avro deserialization schema
>> based on a schema registry (have a look at RegistryDeserializationSchema
>> and SchemaCoderProvider). There is also an opened PR for adding a
>> similar serialization schema[1].
>>
>> The only concern is that I am not 100% sure what is the consensus on
>> which connectors do we want to adapt into the main repository and which
>> would we prefer to be hosted separately and included in the ecosystem
>> webpage[2] (that I hope will be published soon).
>>
>> Whatever option will be preferred I could help review the code.
>>
>> Best,
>>
>> Dawid
>>
>> [1] https://github.com/apache/flink/pull/8371
>>
>> [2]
>>
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html
>>
>> On 05/11/2019 12:40, Őrhidi Mátyás wrote:
>>> Dear Flink Community!
>>>
>>> We have noticed a recent request for Hortonworks schema registry support
>> (
>>> FLINK-14577 ). We
>> have
>>> an implementation for it already, and we would be happy to contribute it
>> to
>>> Apache Flink.
>>>
>>> You can find the documentation below[1]. Let us know your thoughts!
>>>
>>> Best Regards,
>>> Matyas
>>>
>>> [1] Flink Avro Cloudera Registry User Guide
>>> ---
>>>
>>> Add the following dependency to use the schema registry integration:
>>> 
>>> org.apache.flink
>>> flink-avro-cloudera-registry
>>> ${flink.version}
>>> 
>>>
>>>
>>> The schema registry can be plugged directly into the FlinkKafkaConsumer
>> and
>>> FlinkKafkaProducer using the appropriate schema:
>>> -
>>>
>> org.apache.flink.formats.avro.registry.cloudera.SchemaRegistryDeserializationSchema
>>> -
>>>
>> org.apache.flink.formats.avro.registry.cloudera.SchemaRegistrySerializationSchema
>>>
>>> Supported types
>>> --
>>> - Avro Specific Record types
>>> - Avro Generic Records
>>> - Basic Java Data types: byte[], Byte, Integer, Short, Double, Float,
>> Long,
>>> String, Boolean
>>>
>>> SchemaRegistrySerializationSchema
>>> --
>>> The serialization schema can be constructed using the included builder
>>> object SchemaRegistrySerializationSchema.builder(..).
>>>
>>> Required settings:
>>> - Topic configuration when creating the builder. Can be static or dynamic
>>> (extracted from the data)
>>> - RegistryAddress parameter on the builder to establish the connection
>>>
>>> Optional settings:
>>> - Arbitrary SchemaRegistry client configuration using the setConfig
>> method
>>> - Key configuration for the produced Kafka messages
>>>  - By specifying a KeySelector function that extracts the key from each
>>> record
>>>  - Using a Tuple2 stream for (key, value) pairs directly
>>> - Security configuration
>>>
>>> Example:
>>> KafkaSerializationSchema schema =
>>> 

[jira] [Created] (FLINK-14628) Wordcount on Docker test (custom fs plugin) fails on Travis

[Gary Yao (Jira)]

Gary Yao created FLINK-14628:


 Summary: Wordcount on Docker test (custom fs plugin) fails on 
Travis
 Key: FLINK-14628
 URL: https://issues.apache.org/jira/browse/FLINK-14628
 Project: Flink
  Issue Type: Bug
  Components: FileSystems, Tests
Affects Versions: 1.10.0
Reporter: Gary Yao


https://api.travis-ci.org/v3/job/607616429/log.txt

{noformat}
Successfully tagged test_docker_embedded_job:latest
~/build/apache/flink
sort: cannot read: 
'/home/travis/build/apache/flink/flink-end-to-end-tests/test-scripts/temp-test-directory-53405131685/out/docker_wc_out*':
 No such file or directory
FAIL WordCount: Output hash mismatch.  Got d41d8cd98f00b204e9800998ecf8427e, 
expected 72a690412be8928ba239c2da967328a5.
head hexdump of actual:
head: cannot open 
'/home/travis/build/apache/flink/flink-end-to-end-tests/test-scripts/temp-test-directory-53405131685/out/docker_wc_out*'
 for reading: No such file or directory
[FAIL] Test script contains errors.
Checking for errors...
No errors in log files.
Checking for exceptions...
No exceptions in log files.
Checking for non-empty .out files...
grep: 
/home/travis/build/apache/flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/log/*.out:
 No such file or directory
No non-empty .out files.

{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] FLIP-84: Improve & Refactor execute/sqlQuery/sqlUpdate APIS of TableEnvironment

[Terry Wang]

Hi Jark,

Thanks for your suggestion!
Change the title and wait for more comments.

Best,
Terry Wang



> 2019年11月6日 15:52，Jark Wu  写道：
> 
> Hi Terry,
> 
> I would suggest to change the title a bit.
> For example, "Improve & Refactor TableEnvironment APIs".
> Or more specifically, "Improve & Refactor TableEnvironment
> execute/sqlQuery/sqlUpdate.. APIs"
> 
> Currently, the title is a little wide (there are so many APIs in table
> module) .
> Make the title more specifically can attract more people who care about it.
> 
> Best,
> Jark
> 
> 
> 
> On Tue, 5 Nov 2019 at 14:51, Kurt Young  wrote:
> 
>> cc @Fabian here, thought you might be interesting to review this.
>> 
>> Best,
>> Kurt
>> 
>> 
>> On Thu, Oct 31, 2019 at 1:39 PM Kurt Young  wrote:
>> 
>>> Thanks Terry for bringing this up. TableEnv's interface is really
>> critical
>>> not only
>>> to users, but also for components built upon it like SQL CLI. Your
>>> proposal
>>> solved some pain points we currently have, so +1 to the proposal.
>>> 
>>> I left some comments in the document.
>>> 
>>> Best,
>>> Kurt
>>> 
>>> 
>>> On Thu, Oct 31, 2019 at 10:38 AM Terry Wang  wrote:
>>> 
 Hi everyone,
 
 TableEnvironment has provided two `Table sqlQuery(String sql)` and `void
 sqlUpdate(String sql)` interfaces to create a table(actually a view
>> here)
 or describe an update action from one sql string.
 But with more use cases come up, there are some fatal shortcomings in
 current API design. Such as  `sqlUpdate()` don’t support get a return
>> value
 and buggy support for buffer sql exception and so on.
 
 So I’d like to kick off a discussion on improvement and refactor the api
 of table module:
 
 google doc:
 
>> https://docs.google.com/document/d/19-mdYJjKirh5aXCwq1fDajSaI09BJMMT95wy_YhtuZk/edit?usp=sharing
 <
 
>> https://docs.google.com/document/d/19-mdYJjKirh5aXCwq1fDajSaI09BJMMT95wy_YhtuZk/edit?usp=sharing
> 
 Flip link:
 
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=134745878
 <
 
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=134745878
> 
 
 In short, it:
- Discuss buffering sql execute problem
- Discuss current `sqlQuery/sqlUpdate` and propose two new api
- Introduce one new `executeBatch` method to support batch sql
 execute
- Discuss how SQL CLI should deal with multiple statements
 
 Looking forward to all your guys comments.
 
 Best,
 Terry Wang
 
 
 
 
>> 

[jira] [Created] (FLINK-14627) Refactor ExecutionGraph creation in tests as TestingExecutionGraphBuilder

[Zili Chen (Jira)]

Zili Chen created FLINK-14627:
-

 Summary: Refactor ExecutionGraph creation in tests as 
TestingExecutionGraphBuilder
 Key: FLINK-14627
 URL: https://issues.apache.org/jira/browse/FLINK-14627
 Project: Flink
  Issue Type: Improvement
  Components: Tests
Reporter: Zili Chen
Assignee: Zili Chen
 Fix For: 1.10.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)