[DISCUSS] Introduce SupportsParallelismReport and SupportsStatisticsReport for Hive and Filesystem

2020-07-29 Thread Jingsong Li
Hi all, ## SupportsParallelismReport Now that FLIP-95 [1] is ready, only Hive and Filesystem are still using the old interfaces. We are considering migrating to the new interface. However, one problem is that in the old interface implementation, connectors infer parallelism by itself instead of

Re: [DISCUSS] Align the semantic of returning null from InputFormat#nextRecord

2020-07-29 Thread Jingsong Li
Hi Benchao, My understanding is that #1 treats null as the end of input. This means we should try our best to avoid returning null before the end of input in the implementation of `InputFormat`. Even if we have to return null, we have to return it at the end of input. I think this is the most safe

Re: [VOTE] FLIP-129: Refactor Descriptor API to register connector in Table API

2020-07-28 Thread Jingsong Li
+1 Thanks Jark for driving this. On Mon, Jul 27, 2020 at 4:14 PM Leonard Xu wrote: > Thanks Jark > > +1(non-binding) > > Best > Leonard > > 在 2020年7月27日,15:32,jincheng sun 写道: > > > > +1(binding) > > > > Best, > > Jincheng > > > > > > Jark Wu 于2020年7月27日周一 上午10:01写道: > > > >> +1 (binding) > >

Re: [DISCUSS] FLIP-129: Refactor Descriptor API to register connector in Table API

2020-07-24 Thread Jingsong Li
gt; `TableSourceTable#tableIdentifier` which is used to calculate the digest. > > This requires that the registered table must have an unique identifier. > The > > old `TableSourceQueryOperation` will also generate the identifier > according > > to the hashcode of the TableSourc

Re: [ANNOUNCE] Apache Flink 1.11.1 released

2020-07-22 Thread Jingsong Li
Thanks for being the release manager for the 1.11.1 release, Dian. Best, Jingsong On Thu, Jul 23, 2020 at 10:12 AM Zhijiang wrote: > Thanks for being the release manager and the efficient work, Dian! > > Best, > Zhijiang > > -- > F

Re: [DISCUSS] FLIP-129: Refactor Descriptor API to register connector in Table API

2020-07-16 Thread Jingsong Li
Thanks for the discussion. Descriptor lacks the watermark and the computed column is too long. 1) +1 for just `column(...)` 2) +1 for being consistent with Table API, the Java Table API should be Expression DSL. We don't need pure string support, users should just use DDL instead. I think this i

Re: [DISCUSS] Releasing Flink 1.11.1 soon?

2020-07-09 Thread Jingsong Li
FLINK-18461 is really a blocker for the CDC feature. So +1 for releasing Flink 1.11.1 soon. Best, Jingsong On Thu, Jul 9, 2020 at 5:34 PM jincheng sun wrote: > Thanks for bring up this discussion Jark. > +1, looking forward the first bugfix version of Flink 1.11. > > Best, > Jincheng > > Dian

Re: [ANNOUNCE] Apache Flink 1.11.0 released

2020-07-07 Thread Jingsong Li
Congratulations! Thanks Zhijiang and Piotr as release managers, and thanks everyone. Best, Jingsong On Wed, Jul 8, 2020 at 10:51 AM chaojianok wrote: > Congratulations! > > Very happy to make some contributions to Flink! > > > > > > At 2020-07-07 22:06:05, "Zhijiang" wrote: > > The Apache Fli

Re: [ANNOUNCE] New PMC member: Piotr Nowojski

2020-07-06 Thread Jingsong Li
Congratulations Piotr! Well deserved! Best, Jingsong On Tue, Jul 7, 2020 at 9:37 AM Xintong Song wrote: > Congratulations Piotr~! > > Thank you~ > > Xintong Song > > > > On Tue, Jul 7, 2020 at 9:35 AM Zhu Zhu wrote: > > > Congratulations! > > > > > > Leonard Xu 于2020年7月7日周二 上午9:16写道: > > > >

Re: [VOTE] Release 1.11.0, release candidate #4

2020-07-05 Thread Jingsong Li
+1 (non-binding) - verified signature and checksum - build from source - checked webui and log sanity - played with filesystem and new connectors - played with Hive connector Best, Jingsonga On Mon, Jul 6, 2020 at 9:50 AM Xintong Song wrote: > +1 (non-binding) > > - verified signature and chec

Re: [DISCUSS] Sql-client lack support for new features

2020-07-05 Thread Jingsong Li
, 18 Jun 2020 at 15:40, Rui Li wrote: > > > > > Thanks Jingsong for bringing up the discussion. Perhaps we can do both > #1 > > > and #2? I like the idea that SQL Client should just forward SQL > > statements > > > to TableEnvironment. IIUC, with Tabl

Re: [DISCUSS] [FLINK-16824] Creating Temporal Table Function via DDL

2020-06-27 Thread Jingsong Li
Thanks for your discussion. Looks like the problem is supporting the versioned temporal table for the changelog source. I want to share more of my thoughts: When I think about changelog sources, I treat it as a view like: "CREATE VIEW changelog_table AS SELECT ... FROM origin_table GROUP BY ..."

[DISCUSS] Sql-client lack support for new features

2020-06-18 Thread Jingsong Li
Hi all, I'd like to start a discussion for the new features lacking support of Sql-client. I've seen the new DDL syntax that SQL client lacks support for many times. For every new DDL syntax, we need to add support in sql-client. Add a corresponding SqlCommand in sql-client, otherwise this DDL is

Re: [ANNOUNCE] Apache Flink 1.11.0, release candidate #2

2020-06-17 Thread Jingsong Li
Hi Piotr and Zhijiang, I merged three commits in 1 hour ago: one blocker: 0dafcf8792220dbe5b77544261f726a566b054f5 two critical issues: 1830c1c47b8a985ec328a7332e92d21433c0a4df 80fa0f5c5b8600f4b386487f267bde80b882bd07 I have synced with Zhijiang, they were merged after RC2 branch cutting. So RC2 d

Re: [ANNOUNCE] Yu Li is now part of the Flink PMC

2020-06-16 Thread Jingsong Li
Congratulations Yu, well deserved! Best, Jingsong On Wed, Jun 17, 2020 at 1:42 PM Yuan Mei wrote: > Congrats, Yu! > > GXGX & well deserved!! > > Best Regards, > > Yuan > > On Wed, Jun 17, 2020 at 9:15 AM jincheng sun > wrote: > >> Hi all, >> >> On behalf of the Flink PMC, I'm happy to announce

Re: [DISCUSS] Update our EditorConfig file

2020-06-10 Thread Jingsong Li
+1 looks more friendly to Flink newbies. Best, Jingsong Lee On Wed, Jun 10, 2020 at 8:38 PM Aljoscha Krettek wrote: > Hi, > > is anyone actually using our .editorconfig file? IntelliJ has a plugin > for this that is actually quite powerful. > > I managed to write a .editorconfig file that I qui

Re: [ANNOUNCE] New Flink Committer: Benchao Li

2020-06-08 Thread Jingsong Li
Congratulations, Benchao. Well deserved! Best, Jingsong Lee On Tue, Jun 9, 2020 at 2:13 PM Forward Xu wrote: > Congratulations, Benchao > > Best, > Forward > > Jark Wu 于2020年6月9日周二 下午2:10写道: > > > Hi everyone, > > > > On behalf of the PMC, I'm very happy to announce Benchao Li as a new > Apach

Re: [DISCUSS] Releasing "fat" and "slim" Flink distributions

2020-06-07 Thread Jingsong Li
年6月5日周五 下午2:31写道: > > > > > >> +1, at least, we should keep an out of the box SQL-CLI, it’s very poor > > >> experience to add such required format jars for SQL users. > > >> > > >> Best, > > >> Danny Chan > > >> 在 20

Re: [ANNOUNCE] New Apache Flink Committer - Xintong Song

2020-06-04 Thread Jingsong Li
Congratulations Xintong, well deserved! Best, Jingsong Lee On Fri, Jun 5, 2020 at 2:00 PM Yang Wang wrote: > Congratulations, Xintong. > Well deserved. > > Best, > Yang > > Weihua Hu 于2020年6月5日周五 下午1:57写道: > > > Congratulations! > > > > Best > > Weihua Hu > > > > > 2020年6月5日 13:50,Xingbo Huang

Re: [DISCUSS] Releasing "fat" and "slim" Flink distributions

2020-06-04 Thread Jingsong Li
Jingsong Li wrote: > Thanks for your discussion. > > Sorry to start discussing another thing: > > The biggest problem I see is the variety of problems caused by users' lack > of format dependency. > As Aljoscha said, these three formats are very small and no third party

Re: [DISCUSS] Semantics of our JIRA fields

2020-05-25 Thread Jingsong Li
+1 for the proposal. It makes me clearer. Best, Jingsong Lee On Mon, May 25, 2020 at 2:51 PM Zhijiang wrote: > Thanks for launching this discussion and giving so detailed infos, > Robert! +1 on my side for the proposal. > > For "Affects Version", I previously thought it was only for the alread

Re: [DISCUSS] FLIP-84 Feedback Summary

2020-05-17 Thread Jingsong Li
+1 Best, Jingsong Lee On Sun, May 17, 2020 at 3:42 PM Kurt Young wrote: > +1 from my side. > > Best, > Kurt > > > On Sun, May 17, 2020 at 12:41 PM godfrey he wrote: > > > hi everyone, > > > > I would like to bring up another topic about the return value of > > TableResult#collect() method. > >

Re: [VOTE] Guarantee that @PublicEvolving classes are API and binary compatible across bug fix releases (x.y.u and x.y.v)

2020-05-15 Thread Jingsong Li
+1 (non-binding) Best, Jingsong Lee On Fri, May 15, 2020 at 9:26 PM Thomas Weise wrote: > +1 > > > On Fri, May 15, 2020 at 6:15 AM Till Rohrmann > wrote: > > > Dear community, > > > > with reference to the dev ML thread about guaranteeing API and binary > > compatibility for @PublicEvolving cl

Re: [DISCUSS] Exact feature freeze date

2020-05-15 Thread Jingsong Li
+1 for Monday morning. Best, Jingsong Lee On Fri, May 15, 2020 at 8:45 PM Till Rohrmann wrote: > +1 from my side extend the feature freeze until Monday morning. > > Cheers, > Till > > On Fri, May 15, 2020 at 2:04 PM Robert Metzger > wrote: > > > I'm okay, but I would suggest to agree on a time

Re: [DISCUSS] Stability guarantees for @PublicEvolving classes

2020-05-14 Thread Jingsong Li
Thanks Till for starting this discussion. +1 for enabling the japicmp-maven-plugin for @PublicEvolving for bug fix releases. Bug fix should just be user imperceptible bug fix. Should not affect API and binary compatibility. And even PublicEvolving api change for "y" release, we should expose it i

Re: Reading parquet from S3

2020-05-13 Thread Jingsong Li
Hi Kirill, For DataSet API, yes, we have "ParquetRowInputFormat", "ParquetPojoInputFormat" and "ParquetMapInputFormat" [1]. For file system parquet files in table: - Before 1.11, you can register a ParquetTableSource to batch table environment with legacy planner. - In 1.11, you can create file s

Re: [DISCUSS] Releasing "fat" and "slim" Flink distributions

2020-05-12 Thread Jingsong Li
t; > >>>>>> 2. I think there is also the problem that the matrix of possible > >>>>>> combinations that can be useful is already big. Do we want to have a > >>>>>> distribution for: > >>>>>> > >>>>>&g

Re: UpsertStreamTableSink vs OverwritableTableSink

2020-05-06 Thread Jingsong Li
Hi, Insert overwrite comes from Batch SQL in Hive. It means overwriting whole table/partition instead of overwriting per key. So if "insert overwrite kudu_table", should delete whole table in kudu first, and then insert new data to the table in kudu. The same semantics should be used in streaming

Re: [DISCUSS] Hierarchies in ConfigOption

2020-05-06 Thread Jingsong Li
Hi, +1 to: format = parquet parquet.compression = ... parquet.block.size = ... parquet.page.size = ... For the formats like parquet and orc, Not just Flink itself, but this way also let Flink keys compatible with the property keys of Hadoop / Hive / Spark. And like Jark said, this way works for

Re: [DISCUSS]Refactor flink-jdbc connector structure

2020-05-05 Thread Jingsong Li
Thanks Leonard for starting this discussion. +1 from my side. > JDBCOutputFormat Already deprecated, so let it in old package. > JDBCInputFormat Not only input format, but also ParameterValuesProvider, looks good to me for keeping them in old package. You can wait until the next time you refacto

Re: [DISCUSS] Introduce TableFactory for StatefulSequenceSource

2020-05-05 Thread Jingsong Li
a generating table source to > an > > > ecosystem project promoted on flink-packages.org. I think this has a > lot > > > of > > > potential (e.g. in combination with Java Faker [1]), but it would > > probably > > > be better served in a small separately maintained re

Re: [DISCUSS] Hierarchies in ConfigOption

2020-04-29 Thread Jingsong Li
n' value.format.option: option1: '...' option2: '...' Best, Jingsong Lee On Thu, Apr 30, 2020 at 10:16 AM Jingsong Li wrote: > Thanks Timo for staring the discussion. > > I am +1 for "format: 'json'". > Take a look to Dawid's y

Re: [DISCUSS] Hierarchies in ConfigOption

2020-04-29 Thread Jingsong Li
Thanks Timo for staring the discussion. I am +1 for "format: 'json'". Take a look to Dawid's yaml case: connector: 'filesystem' path: '...' format: 'json' format: option1: '...' option2: '...' option3: '...' Is this work? According to my understanding, 'format' key is the attribute o

Re: [ANNOUNCE] Apache Flink 1.9.3 released

2020-04-26 Thread Jingsong Li
Thanks Dian for managing this release! Best, Jingsong Lee On Sun, Apr 26, 2020 at 7:17 PM Jark Wu wrote: > Thanks Dian for being the release manager and thanks all who make this > possible. > > Best, > Jark > > On Sun, 26 Apr 2020 at 18:06, Leonard Xu wrote: > > > Thanks Dian for the release a

Re: [DISCUSS] Exact feature freeze date

2020-04-23 Thread Jingsong Li
+1 Best, Jingsong Lee On Fri, Apr 24, 2020 at 2:27 AM Zhu Zhu wrote: > +1 for extending the code freeze date. > FLIP-119 could benefit from it. > May 15th sounds reasonable. > > Thanks, > Zhu Zhu > > Jark Wu 于2020年4月24日周五 上午12:01写道: > > > +1 > > > > Thanks, > > Jark > > > > On Thu, 23 Apr 2020

Re: Reading a single input file in parallel?

2020-04-23 Thread Jingsong Li
Hi Niels, Thanks for start this discussion. Share some thought about your questions/proposals. > Judge whether splittable for each individual file Looks good to me. > BZIP2 support splittable Looks good to me. > the Flink implementation is controlled by the number of splits Can you check again?

Re: [ANNOUNCE] New Apache Flink PMC Member - Hequn Chen

2020-04-19 Thread Jingsong Li
Congratulations Hequn! Best, Jingsong Lee On Mon, Apr 20, 2020 at 9:52 AM jincheng sun wrote: > Congratulations and welcome on board Hequn! > > Best, > Jincheng > > > > Zhijiang 于2020年4月19日周日 下午10:47写道: > > > Congratulations, Hequn! > > > > Best, > > Zhijiang > > > > > > --

Re: [DISCUSS] Generating java code?

2020-04-15 Thread Jingsong Li
Hi Till, +1 to define an interface and load it at runtime if we can do. No disrupting the workflows of devs and throw an exception with good description look good to me. This also force us to do a good dependent class abstract. Best, Jingsong Lee On Wed, Apr 15, 2020 at 10:31 PM Till Rohrmann w

Re: [PROPOSAL] Contribute training materials to Apache Flink

2020-04-15 Thread Jingsong Li
+1. It's very useful for Flink newcomers. Best, Jingsong Lee On Wed, Apr 15, 2020 at 10:23 PM Yun Tang wrote: > +1 for this idea. > > I think there would existed many details to discuss once community ready > to host the materials: > >1. How to judge whether a lab exercise should be added?

Re: [DISCUSS] Releasing "fat" and "slim" Flink distributions

2020-04-15 Thread Jingsong Li
> > > connectors and formats > > > into our "lib" directory, like kafka, csv, json metioned above, and > still > > > leave some other connectors out of it. > > > If this is the case, then why not we just provide this distribution to > > > user

Re: [DISCUSS] Releasing "fat" and "slim" Flink distributions

2020-04-15 Thread Jingsong Li
Big +1. I like "fat" and "slim". For csv and json, like Jark said, they are quite small and don't have other dependencies. They are important to kafka connector, and important to upcoming file system connector too. So can we move them to both "fat" and "slim"? They're so important, and they're so

Re: flink-orc or flink-orc-nohive

2020-04-14 Thread Jingsong Li
slightly newer version - maybe 1.5.x or even 1.6.0? > > - > Sivaprasanna > > On Tue, Apr 14, 2020 at 1:42 PM Jingsong Li > wrote: > > > Hi, > > > > Maybe you should use flink-orc. And use orc-core instead of orc-core with > > nohive classifier. We can

Re: [VOTE] FLIP-123: DDL and DML compatibility for Hive connector

2020-04-14 Thread Jingsong Li
+1 Best, Jingsong Lee On Tue, Apr 14, 2020 at 9:29 PM Kurt Young wrote: > +1 > > Best, > Kurt > > > On Mon, Apr 13, 2020 at 9:26 PM Rui Li wrote: > > > Hi all, > > > > I'd like to start the vote for FLIP-123[1], which has been discussed in > > thread[2]. > > > > The vote will be open for 72h,

Re: flink-orc or flink-orc-nohive

2020-04-14 Thread Jingsong Li
Hi, Maybe you should use flink-orc. And use orc-core instead of orc-core with nohive classifier. We can provide nohive version in the future. Because orc and hive are so close, orc still relies on some classes of hive currently. Apache orc with nohive classifier is for create a variant of core an

Re: [VOTE] FLIP-105: Support to Interpret and Emit Changelog in Flink SQL

2020-04-13 Thread Jingsong Li
+1 Best, Jingsong Lee On Tue, Apr 14, 2020 at 1:46 PM Jark Wu wrote: > +1 (binding) > > Best, > Jark > > On Sun, 12 Apr 2020 at 09:24, Benchao Li wrote: > > > +1 (non-binding) > > > > Jark Wu 于2020年4月11日周六 上午11:31写道: > > > > > Hi all, > > > > > > I would like to start the vote for FLIP-105 [1

Re: [DISCUSS] FLIP-123: DDL and DML compatibility for Hive connector

2020-04-13 Thread Jingsong Li
ate parser only requires minimum change in the planner, which I > think is acceptable compared to the benefits it brings us. > > On Thu, Apr 9, 2020 at 10:42 PM Jingsong Li > wrote: > > > Thanks Rui for diving. > > > > +1 for this proposal. > > > > Th

Re: [VOTE] FLIP-71: E2E View support in Flink SQL

2020-04-09 Thread Jingsong Li
+1 Hope we can finish it in 1.11, we have been looking forward to it for a long time. Best, Jingsong Lee On Thu, Apr 9, 2020 at 9:23 PM Timo Walther wrote: > +1 (binding) > > Thanks for your efforts. > > Regards, > Timo > > > On 09.04.20 14:46, Zhenghua Gao wrote: > > Hi all, > > > > I'd like

Re: [DISCUSS] FLIP-123: DDL and DML compatibility for Hive connector

2020-04-09 Thread Jingsong Li
Thanks Rui for diving. +1 for this proposal. There are still lots of people who love Hive SQL. And I have seen some people support HQL on presto. Presto, as a famous computing engine, also supports ANSI SQL as we do. This is quite different from HQL. Do you think we must need import `FlinkHiveSq

Re: [ANNOUNCE] New Flink committer: Seth Wiesman

2020-04-06 Thread Jingsong Li
Congratulations Seth! Best, Jingsong Lee On Tue, Apr 7, 2020 at 2:46 PM Dawid Wysakowicz wrote: > Congratulations Seth. Happy to have you in the community! > > Best, > > Dawid > > On 07/04/2020 08:43, Dian Fu wrote: > > Congratulations! > > > >> 在 2020年4月7日,下午2:35,Konstantin Knauf 写道: > >> > >

Re: [VOTE] FLIP-122: New Connector Property Keys for New Factory

2020-04-02 Thread Jingsong Li
+1 Best, Jingsong Lee On Fri, Apr 3, 2020 at 9:50 AM Benchao Li wrote: > +1 (non-binding) > > Dawid Wysakowicz 于2020年4月3日周五 上午12:33写道: > > > +1 > > > > Best, > > > > Dawid > > > > On 02/04/2020 18:28, Timo Walther wrote: > > > +1 > > > > > > Thanks, > > > Timo > > > > > > On 02.04.20 17:22, Ja

Re: [DISCUSS]FLIP-113: Support SQL and planner hints

2020-04-01 Thread Jingsong Li
Hi Dawid, > When a factory is instantiated it has access to the CatalogTable, therefore it has access to all the original properties. In turn it knows the original format and can call FormatFactory#supportedHintOptions(). Factory can only get CatalogTable when creating source or sink, right? IIUC

Re: [ANNOUNCE] New Committers and PMC member

2020-04-01 Thread Jingsong Li
Congratulations! Konstantin, Dawid, Zhijiang. Well deserved. Best, Jingsong Lee On Wed, Apr 1, 2020 at 4:52 PM Stephan Ewen wrote: > Hi all! > > Happy to announce that over the last few weeks, several people in the > community joined in new roles: > > - Konstantin Knauf joined as a committer.

Re: [DISCUSS] Change default planner to blink planner in 1.11

2020-03-31 Thread Jingsong Li
+1 In 1.10, we have set default planner for SQL Client to Blink planner[1]. Looks good. [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Set-default-planner-for-SQL-Client-to-Blink-planner-in-1-10-release-td36379.html Best, Jingsong Lee On Wed, Apr 1, 2020 at 11:39 AM

Re: [VOTE] FLIP-113: Supports Dynamic Table Options for Flink SQL

2020-03-31 Thread Jingsong Li
Hi Danny, If I haven't missed anything, I don't know how to support the dynamic options of format. Connector don't know format information in TableFactory before obtains real properties, so it can not list any format `supportedHintOptions`. And I think it is important for support format dynamic o

Re: [DISCUSS] FLIP-110: Support LIKE clause in CREATE TABLE

2020-03-31 Thread Jingsong Li
QL standard, I think it's fine for vendors to > >>> define > >>> theire semantics. > >>> > >>>> Standard also allows declaring the clause after the schema part. We > >>>> can > >>> also do it. > >>> Is that true? I didn't find it in SQL st

Re: [ANNOUNCE] Flink on Zeppelin (Zeppelin 0.9 is released)

2020-03-30 Thread Jingsong Li
Thanks Jeff very much, that is very impressive. Zeppelin is very convenient development platform. Best, Jingsong Lee On Tue, Mar 31, 2020 at 11:58 AM Zhijiang wrote: > > Thanks for the continuous efforts for engaging in Flink ecosystem Jeff! > Glad to see the progressive achievement. Wish more

Re: [VOTE] FLIP-95: New TableSource and TableSink interfaces

2020-03-30 Thread Jingsong Li
+1 Best, Jingsong Lee On Mon, Mar 30, 2020 at 4:41 PM Kurt Young wrote: > +1 > > Best, > Kurt > > > On Mon, Mar 30, 2020 at 4:08 PM Benchao Li wrote: > > > +1 (non-binding) > > > > Jark Wu 于2020年3月30日周一 下午3:57写道: > > > > > +1 from my side. > > > > > > Thanks Timo for driving this. > > > > > >

Re: [DISCUSS] FLIP-122: New Connector Property Keys for New Factory

2020-03-30 Thread Jingsong Li
Thanks Jark for the proposal. +1 to the general idea. For "version", what about "kafka.version"? It is obvious to know its meaning. And I'd like to start a new topic: Should we need to explicitly separate source from sink? With the development of batch and streaming, more and more connectors hav

Re: [VOTE] FLIP-115: Filesystem connector in Table

2020-03-29 Thread Jingsong Li
; Best, > Jark > > On Mon, 30 Mar 2020 at 11:32, Jingsong Li wrote: > > > +1 (binding) > > > > Best, > > Jingsong Lee > > > > On Sun, Mar 29, 2020 at 10:37 PM Benchao Li wrote: > > > > > +1 (non-binding) > > > > > >

Re: [VOTE] FLIP-115: Filesystem connector in Table

2020-03-29 Thread Jingsong Li
t; > > 在 2020年3月28日,16:25,Kurt Young 写道: > > > > > > +1 > > > > > > Best, > > > Kurt > > > > > > > > > On Fri, Mar 27, 2020 at 10:51 AM Jingsong Li > > wrote: > > > > > >> Hi eve

[VOTE] FLIP-115: Filesystem connector in Table

2020-03-26 Thread Jingsong Li
Hi everyone, I'd like to start the vote of FLIP-115 [1], which introduce Filesystem table factory in table. This FLIP is discussed in the thread[2]. The vote will be open for at least 72 hours. Unless there is an objection, I will try to close it by March 30, 2020 03:00 UTC if we have received su

Re: [DISCUSS]FLIP-113: Support SQL and planner hints

2020-03-26 Thread Jingsong Li
; > > > > >> Best, > > > > >> Jark > > > > >> > > > > >> [1]: https://kafka.apache.org/documentation/#consumerconfigs > > > > >> > > > > >> On Wed, 18 Mar 2020 at 21:44, Danny Chan > >

Re: [DISCUSS] FLIP-115: Filesystem connector in Table

2020-03-25 Thread Jingsong Li
treamingFileSink might not > support currently. I am very glad to see that you all agree that improving > the StreamingFileSink architecture for these new cases. > > Best, > Guowei > > > Jingsong Li 于2020年3月19日周四 上午12:19写道: > > > Hi Stephan & Kostas & Piotr

Re: [DISCUSS] FLIP-95: New TableSource and TableSink interfaces

2020-03-24 Thread Jingsong Li
+1. Thanks Timo for the design doc. We can also consider @Experimental too. But I am +1 to @PublicEvolving, we should be confident in the current change. Best, Jingsong Lee On Tue, Mar 24, 2020 at 4:30 PM Timo Walther wrote: > @Becket: We totally agree that we don't need table specific connect

Re: [DISCUSS] Introduce TableFactory for StatefulSequenceSource

2020-03-24 Thread Jingsong Li
tackle them for the sake of > simplifying user experience to the extreme. Providing the above handy > source and sink implementations already offer users a ton of immediate > value. > > > On Mon, Mar 23, 2020 at 20:20 Jingsong Li wrote: > > > Hi Benchao, > > > &g

Re: [DISCUSS] Introduce TableFactory for StatefulSequenceSource

2020-03-23 Thread Jingsong Li
nerally, it's a very good proposal. > > > > About data gen source, do you think we need to add more columns with > > various types? > > > > About print sink, do we need to specify the schema? > > > > Jingsong Li 于2020年3月23日周一 下午1:51写道: > > > &

Re: [DISCUSS] Introduce TableFactory for StatefulSequenceSource

2020-03-22 Thread Jingsong Li
nt data in row format in console > >> - purposes > >> - make it easier to test Flink SQL job e2e in IDE > >> - test Flink pipeline and ensure output data format/value is > >> correct > >> 3. no output data sink > >>

[DISCUSS] Introduce TableFactory for StatefulSequenceSource

2020-03-19 Thread Jingsong Li
Hi all, I heard some users complain that table is difficult to test. Now with SQL client, users are more and more inclined to use it to test rather than program. The most common example is Kafka source. If users need to test their SQL output and checkpoint, they need to: - 1.Launch a Kafka standa

Re: [DISCUSS] FLIP-115: Filesystem connector in Table

2020-03-18 Thread Jingsong Li
eviating from > >> this now, we only put more stress on other teams in the future. When > >> the users start using a given API, with high probability, they will > >> ask (and it is totally reasonable) consistent behaviour from all the > >> other APIs that ship wit

Re: [DISCUSS] (FLINK-16648) Clean up the input parameter StreamQueryConfig of the API in StreamTableEnvironment

2020-03-18 Thread Jingsong Li
Hi Jinhai, thanks for driving. +1 to remove, I think we can remove StreamQueryConfig too. since we have deprecated StreamQueryConfig two versions. Remember to record it in release notes of issue. Best, Jingsong Lee On Wed, Mar 18, 2020 at 5:58 PM jinhai wang wrote: > Hi Devs > > I would like

Re: [DISCUSS]FLIP-113: Support SQL and planner hints

2020-03-17 Thread Jingsong Li
Hi, I am thinking we can provide hints to *table* related instances. - TableFormatFactory: of cause we need hints support, there are many format options in DDL too. - catalog and module: I don't know, maybe in future we can provide some hints for them. Best, Jingsong Lee On Wed, Mar 18, 2020 at

Re: [DISCUSS] FLIP-115: Filesystem connector in Table

2020-03-17 Thread Jingsong Li
would make it much easier for users > to > > understand the system. > > Especially when it comes to consistent behavior across external systems. > > Having a different file sink in Table API and DataStream API means that > > DataStream can write correctly to S3 while Table API

Re: [DISCUSS] FLIP-115: Filesystem connector in Table

2020-03-16 Thread Jingsong Li
wn set of limitations, quirks and features. > Especially that we have on our long term roadmap and wish list to unify > such kind of operators. > > Piotrek > > [1] https://issues.apache.org/jira/browse/FLINK-11499 < > https://issues.apache.org/jira/browse/FLINK-11499> > &

Re: [DISCUSS] FLIP-115: Filesystem connector in Table

2020-03-15 Thread Jingsong Li
e On Mon, Mar 16, 2020 at 12:01 PM Jingsong Li wrote: > Thanks Piotr and Yun for involving. > > Hi Piotr and Yun, for implementation, > > FLINK-14254 [1] introduce batch sink table world, it deals with partitions > thing, metastore thing and etc.. And it just reuse

Re: FLIP-117: HBase catalog

2020-03-15 Thread Jingsong Li
Thanks Flavio for driving. Personally I am +1 for integrating HBase tables. I start a new topic for discussion. It is related but not the core of this FLIP. In the FLIP, I can see: - Does HBase support the concept of partitions..? I don't think so.. - Does HBase support functions? I don't think so

Re: [DISCUSS] FLIP-115: Filesystem connector in Table

2020-03-15 Thread Jingsong Li
s for FLIP-115. It is really useful feature for platform developers > > who manage hundreds of Flink to Hive jobs in production. > > > I think we need add 'connector.sink.username' for UserGroupInformation when > > data is written to HDFS > > > > > > 在 2020/3

[DISCUSS] FLIP-115: Filesystem connector in Table

2020-03-13 Thread Jingsong Li
Hi everyone, I'd like to start a discussion about FLIP-115 Filesystem connector in Table [1]. This FLIP will bring: - Introduce Filesystem table factory in table, support csv/parquet/orc/json/avro formats. - Introduce streaming filesystem/hive sink in table CC to user mail list, if you have any u

Re: [DISCUSS] Drop Bucketing Sink

2020-03-12 Thread Jingsong Li
Hi Robert, +1 to drop it but maybe not 1.11. ORC has not been supported on StreamingFileSink. I have seen lots of users run ORC in the bucketing sink. Best, Jingsong Lee On Fri, Mar 13, 2020 at 1:11 AM Seth Wiesman wrote: > Sorry, I meant FLIP-46. > > Seth > > On Thu, Mar 12, 2020 at 11:52 AM

Re: [DISCUSS] Releasing Flink 1.10.1

2020-03-11 Thread Jingsong Li
Thanks for driving. Yu. +1 for starting the 1.10.1 release. Some issues are very important, Users are looking forward to them. Best, Jingsong Lee On Wed, Mar 11, 2020 at 2:52 PM Yangze Guo wrote: > Thanks for driving this release, Yu! > > +1 for starting the 1.10.1 release cycle. > > Best, > Y

Re: [DISCUSS]FLIP-113: Support SQL and planner hints

2020-03-09 Thread Jingsong Li
Hi Danny, +1 for table hints, thanks for driving. I took a look to FLIP, most of content are talking about query hints. It is hard to discussion and voting. So +1 to split it as Jark said. Another thing is configuration that suitable to config with table hints: "connector.path" and "connector.top

Re: [DISCUSS] Introduce flink-connector-hive-xx modules

2020-03-05 Thread Jingsong Li
a name including the > lowest Hive version it supports. > > What do you think? > > > > On Wed, Mar 4, 2020 at 11:14 PM Jingsong Li > wrote: > > > Hi Bowen, thanks for your reply. > > > > > will there be a base module like "flink-connector-hive-base

Re: [DISCUSS] Disable "Squash and merge" button for Flink repository on GitHub

2020-03-05 Thread Jingsong Li
h > >>>>> in > >>>>>> the repository(according to github rules). > >>>>>> > >>>>>> If we only left "merge and commits" button, it will > >>>>>> against requiring > >> a >

Re: [DISCUSS] Disable "Squash and merge" button for Flink repository on GitHub

2020-03-05 Thread Jingsong Li
Thanks for deep investigation. +1 to disable "Squash and merge" button now. But I think this is a very serious problem, It affects too many GitHub workers. Github should deal with it quickly? Best, Jingsong Lee On Thu, Mar 5, 2020 at 7:21 PM Xingbo Huang wrote: > Hi Jark, > > Thanks for bringi

Re: [DISCUSS] Introduce flink-connector-hive-xx modules

2020-03-04 Thread Jingsong Li
1.0.0 - 1.2.2, the module name can be "flink-connector-hive-1.0" > rather than "flink-connector-hive-1.2" > > > On Wed, Mar 4, 2020 at 10:20 PM Jingsong Li > wrote: > > > Thanks Bowen for involving. > > > > > why you proposed segregating

Re: [DISCUSS] Introduce flink-connector-hive-xx modules

2020-03-04 Thread Jingsong Li
Thanks Bowen for involving. > why you proposed segregating hive versions into the 5 ranges above? & what different Hive features are supported in the 5 ranges? For only higher client dependencies version support lower hive metastore versions: - Hive 1.0.0 - 1.2.2, thrift change is OK, only hive d

Re: [DISCUSS] FLIP-110: Support LIKE clause in CREATE TABLE

2020-03-04 Thread Jingsong Li
Thanks Dawid for starting this discussion. I like the "LIKE". 1.For "INHERITS", I think this is a good feature too, yes, ALTER TABLE will propagate any changes in column data definitions and check constraints down the inheritance hierarchy. A inherits B, A and B share every things, they have the

Re: Flink dev blog

2020-03-03 Thread Jingsong Li
+1 for this proposal. I have a lot of desired topics in table and batch. I also second Seth and Stephan 's comment separate this in a clear way. Have concerns that maybe easy to confuse new users. If I am a beginner and find a bunch of deep documents, I need to further distinguish which is effecti

Re: TIME/TIMESTAMP parse in Flink TABLE/SQL API

2020-02-26 Thread Jingsong Li
in a topic. > > Best, > Jark > > On Wed, 26 Feb 2020 at 22:20, Jingsong Li wrote: > >> Thanks all for your discussion. >> >> Hi Dawid, >> >> +1 to apply the logic of parsing a SQL timestamp literal. >> >> I don't fully understand the

Re: TIME/TIMESTAMP parse in Flink TABLE/SQL API

2020-02-26 Thread Jingsong Li
Thanks all for your discussion. Hi Dawid, +1 to apply the logic of parsing a SQL timestamp literal. I don't fully understand the matrix your list. Should this be the semantics of SQL cast? Do you mean this is implicit cast in JSON parser? I doubt that because these implicit casts are not support

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

2020-02-21 Thread Jingsong Li
Thanks everyone~ It's my pleasure to be part of the community. I hope I can make a better contribution in future. Best, Jingsong Lee On Fri, Feb 21, 2020 at 2:48 PM Hequn Cheng wrote: > Congratulations Jingsong! Well deserved. > > Best, > Hequn > > On Fri, Feb 21, 2020 at 2:42 PM Yang Wang wr

Re: [DISCUSS] Improve TableFactory

2020-02-21 Thread Jingsong Li
hink we need a new VOTE for this, I just want to make this > discussion more publicly. > What do you think? > > Best, > Jark > > On Wed, 5 Feb 2020 at 16:05, Rui Li wrote: > > > +1, thanks for the efforts. > > > > On Wed, Feb 5, 2020 at 4:00 PM Jingson

Re: [DISCUSS] Support "create [temporary] table" and "create [temporary] view" statements

2020-02-20 Thread Jingsong Li
Hi Caizhi, thanks for starting this discussion. There is a FLIP-71 [1] to describe the whole story of view. Sql-cli now implements a wrong way, and a separate way, which should be deprecated and unified to TableEnvironment. > Shall we make it clear and support create table / create temporary tab

Re: [DISCUSS] Kicking off the 1.11 release cycle

2020-02-20 Thread Jingsong Li
Thanks Stephan for the kicking off. Thanks Piotr and Zhijiang for volunteering. +1 for aiming with the feature freeze date for end of April. We should return to the quick release model. (3 months) Best, Jingsong Lee On Thu, Feb 20, 2020 at 6:04 PM Zhu Zhu wrote: > Thanks Piotr and Zhijiang for

Re: [DISCUSS] Improvements on FLIP Process

2020-02-18 Thread Jingsong Li
Hi all, thanks for launching this discussion. About eliminating Google Docs. I agree with Zhijiang, I share my concern about it. If the FLIP Driver is a Flink newer or the FLIP is very big and complicated. His/Her design maybe need change many many things, in this situation, Google doc is good to

Re: [DISCUSS][TABLE] Issue with package structure in the Table API

2020-02-17 Thread Jingsong Li
Thanks for bringing this discussion. +1 to peforming this big change as early as possible. You solved my question, why we need "_root_". Yes, I don't like this import too. And it is very strange that expressionDsl is in api, but can only work in api.scala. (Because scala extends ImplicitExpressi

Re: [ANNOUNCE] New Documentation Style Guide

2020-02-15 Thread Jingsong Li
Thank for the great work, In 1.10, I have modified and reviewed some documents. In that process, sometimes there is some confusion, how to write is the standard. How to write is correct to the users. Docs style now tells me. Learned a lot. Best, Jingsong Lee On Sat, Feb 15, 2020 at 10:00 PM Dian

Re: [DISCUSS] FLIP-84: Improve & Refactor execute/sqlQuery/sqlUpdate APIS of TableEnvironment

2020-02-15 Thread Jingsong Li
> other statement in a single batch. If that happens, the name > "Inserts" > > > will > > > > be weird. > > > > > > > > Best, > > > > Kurt > > > > > > > > > > > > On Thu, Feb 13, 2020 at 4:03 PM J

Re: [DISCUSS] FLIP-84: Improve & Refactor execute/sqlQuery/sqlUpdate APIS of TableEnvironment

2020-02-12 Thread Jingsong Li
Hi Godfrey, Thanks for updating. +1 sketchy. I have no idea to change "sqlQuery" to "fromQuery", I think "sqlQuery" is OK, It's not that confusing with return values. Can we change the "DmlBatch" to "Inserts"? I don't see any other needs. "Dml" seems a little weird. It is better to support "Ins

Re: [VOTE] Support scalar vectorized Python UDF in PyFlink

2020-02-12 Thread Jingsong Li
+1 (non-binding) Thanks Dian for driving. Best, Jingsong Lee On Thu, Feb 13, 2020 at 11:45 AM jincheng sun wrote: > +1 (binding) > > Best, > Jincheng > > > Dian Fu 于2020年2月12日周三 下午1:31写道: > > > Hi all, > > > > I'd like to start the vote of FLIP-97[1] which is discussed and reached > > consensu

Re: [ANNOUNCE] Apache Flink 1.10.0 released

2020-02-12 Thread Jingsong Li
Congratulations! Great work. Best, Jingsong Lee On Wed, Feb 12, 2020 at 11:05 PM Leonard Xu wrote: > Great news! > Thanks everyone involved ! > Thanks Gary and Yu for being the release manager ! > > Best, > Leonard Xu > > 在 2020年2月12日,23:02,Stephan Ewen 写道: > > Congrats to us all. > > A big pi

<    1   2   3   4   5   6   7   >