Hi Radu,

thanks for continuing the discussion we had at the conference here. Your proposals are all valid. If you have a look at the inital design document [1] for Table API/SQL we plan to add a SQL client at some point, but first we should focus on extending the set of supported operations. A first step regarding windows and aggregations on streams can be found in the current FLIP-11 [2]. However, it only describes the Table API so far. How Stream SQL should extactly look like is still up for discussion (together with the Calcite guys). In a long term view the Table API could become a DataSet++ or DataStream++. We could add support for UDFs and operations such as map/reduce. If customization/replacement of existing rules is required we can add a Jira issue for that.

The development just started so there is a lot to improve and to add. New contibutions, discussions on certain features and design documents are always welcome.
Btw. this discussion should actually be continued on the dev mailing list.


[1] https://docs.google.com/document/d/1TLayJNOTBle_-m1rQfgA6Ouj1oYsfqRjPcp1h2TVqdI/edit#heading=h.4vdi2v1tlg8h [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-11%3A+Table+API+Stream+Aggregations

Am 14/09/16 um 15:07 schrieb Deepak Sharma:

Thanks Greg .
I will start picking some of them.


On 14 Sep 2016 6:31 pm, "Greg Hogan" <c...@greghogan.com <mailto:c...@greghogan.com>> wrote:

    Hi Deepak,

    There are many open tickets for Flink's SQL API. Documentation is



    On Wed, Sep 14, 2016 at 12:27 PM, Deepak Sharma
    <deepakmc...@gmail.com <mailto:deepakmc...@gmail.com>> wrote:

        Yes.I agree to having SQL for Flink.
        I can take up some tasks as well once this starts.


        On Wed, Sep 14, 2016 at 3:47 PM, Radu Tudoran
        <radu.tudo...@huawei.com <mailto:radu.tudo...@huawei.com>> wrote:


            As a follow up to multiple discussions that happened
            during Flink Forward about how SQL should be supported by
            Flink, I was thinking to make a couple of proposals.

            Disclaimer: I do not claim I have managed to synthesized
            all the discussions and probably a great deal of things
            are still missing

            *Why supporting SQL for Flink?*

            -A goal to support SQL for Flink should be to enable
            larger adoption of Flink – particularly for data
            scientists / data engineers who might not want/know how to
            program against the existing APIs

            -The main implication as I see from this is that SQL
            should serve as a translation tool of the data processing
            processing flow to a stream topology that will be executed
            by Flink

            -This would require to support rather soon an SQL client
            for Flink

            *How many features should be supported?*

            -In order to enable a (close to ) full benefit of the
            processing capabilities of Flink, I believe most of the
            processing types should be supported – this includes all
            different types of windows, aggregations, transformations,

            -I would propose that UDFs should also be supported such
            that one can easily add more complex computation if needed

            -In the spirit of the extensibility that Flink supports
            for the operators, functions… such custom operators should
            be supported to replace the default implementations of the
            SQL logical operators

            *How much customization should be enabled?*

            -Regarding customization this could be provided by
            configuration files. Such a configuration can cover the
            policies for how the triggers, evictors, parallelization …
             will be done for the specific translation of the SQL
            query into Flink code

            -In order to support the integration of custom operators
            for specific SQL logical operators, the users should be
            enabled also to provide translation RULES that will
            replace the default ones  (e.g. if a user want to define
            their own CUSTOM_TABLE_SCAN, it should be able to provide
            something like
            configuration.replaceRule(DataStreamScanRule.INSTANCE ,
            CUSTOM_TABLE_SCAN_Rule.INSTANCE) – or if the selection of
            the new translation rule can be handled from the cost than
            simply configuration.addRule( CUSTOM_TABLE_SCAN_Rule.INSTANCE)

            What do you think?

            Dr. Radu Tudoran

            Senior Research Engineer - Big Data Expert

            IT R&D Division


            HUAWEI TECHNOLOGIES Duesseldorf GmbH

            European Research Center

            Riesstrasse 25, 80992 München

            E-mail: _radu.tudo...@huawei.com

            Mobile: +49 15209084330 <tel:%2B49%2015209084330>

            Telephone: +49 891588344173 <tel:%2B49%20891588344173>

            HUAWEI TECHNOLOGIES Duesseldorf GmbH
            Hansaallee 205, 40549 Düsseldorf, Germany, www.huawei.com
            Registered Office: Düsseldorf, Register Court Düsseldorf,
            HRB 56063,
            Managing Director: Bo PENG, Wanzhou MENG, Lifang CHEN
            Sitz der Gesellschaft: Düsseldorf, Amtsgericht Düsseldorf,
            HRB 56063,
            Geschäftsführer: Bo PENG, Wanzhou MENG, Lifang CHEN

            This e-mail and its attachments contain confidential
            information from HUAWEI, which is intended only for the
            person or entity whose address is listed above. Any use of
            the information contained herein in any way (including,
            but not limited to, total or partial disclosure,
            reproduction, or dissemination) by persons other than the
            intended recipient(s) is prohibited. If you receive this
            e-mail in error, please notify the sender by phone or
            email immediately and delete it!

-- Thanks
        www.bigdatabig.com <http://www.bigdatabig.com>
        www.keosha.net <http://www.keosha.net>

Freundliche Grüße / Kind Regards

Timo Walther

Follow me: @twalthr

Reply via email to