Hi Jingsong, Thank you for bringing this discussion. Since I don't have much experience of Flink table/SQL, I'll ask some questions from runtime or engine perspective.
> ... where we describe how to partition support in flink and how to integrate to hive partition. FLIP-27 [1] introduces "partition" concept officially. The changes of FLIP-27 are not only about source interface but also about the whole infrastructure. Have you ever thought how to integrate your proposal with these changes? Or you just want to support "partition" in table layer, there will be no requirement of underlying infrastructure? I have seen a discussion [2] that seems be a requirement of infrastructure to support your proposal. So I have some concerns there might be some conflicts between this proposal and FLIP-27. 1. https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface 2. http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-notifyOnMaster-for-notifyCheckpointComplete-td32769.html Thanks, Biao /'bɪ.aʊ/ On Fri, 6 Sep 2019 at 13:22, JingsongLee <lzljs3620...@aliyun.com.invalid> wrote: > Hi everyone, thank you for your comments. Mail name was updated > and streaming-related concepts were added. > > We would like to start a discussion thread on "FLIP-63: Rework table > partition support"(Design doc: [1]), where we describe how to partition > support in flink and how to integrate to hive partition. > > This FLIP addresses: > - Introduce whole story about partition support. > - Introduce and discuss DDL of partition support. > - Introduce static and dynamic partition insert. > - Introduce partition pruning > - Introduce dynamic partition implementation > - Introduce FileFormatSink to deal with streaming exactly-once and > partition-related logic. > > Details can be seen in the design document. > Looking forward to your feedbacks. Thank you. > > [1] > https://docs.google.com/document/d/15R3vZ1R_pAHcvJkRx_CWleXgl08WL3k_ZpnWSdzP7GY/edit?usp=sharing > > Best, > Jingsong Lee
