Beam and HerdDB are now on “powered by”: 
https://calcite.apache.org/docs/powered_by.html 
<https://calcite.apache.org/docs/powered_by.html>

> On Apr 12, 2018, at 5:33 AM, Michael Mior <mm...@uwaterloo.ca> wrote:
> 
> FYI, I talked to Julian earlier this week and he will be adding Beam to the
> powered by page since he has a doc for generating the image with the logos.
> 
> --
> Michael Mior
> mm...@apache.org
> 
> 2018-04-12 4:06 GMT-04:00 Shuyi Chen <suez1...@gmail.com>:
> 
>> @Andrew, great to see BEAM is also using Calcite streaming SQL. Maybe you
>> can help adding an entry in the Calcite powered_by page
>> <https://calcite.apache.org/docs/powered_by.html> for BEAM  by editing
>> site/_docs/powered_by.md
>> <https://github.com/apache/calcite/pull/657#diff-
>> 1aa810bc92051b46555ee4caf24bd6c3>.
>> Also,
>> can you explain a bit more on how you plan to use streaming SQL to
>> transform
>> arbitrary JSON objects with and w/o the AS STRUCT syntax?
>> For adding DDL in BEAM, please take a look at the server module and the
>> TYPE DDL (https://issues.apache.org/jira/browse/CALCITE-2045) I am adding.
>> Let me know if you have any comments and need any help.
>> 
>> @Rong, I think it's better to conform with the ROW SQL standard, and add
>> new grammar to handle named struct construction.
>> This should work with the TYPE DDL, and we should be able to CAST the
>> created STRUCT as a custom type defined using DDL.
>> 
>> @Julian, thanks for the suggestions. I think the AS STRUCT addition should
>> do it.
>> 
>> Shuyi
>> 
>> On Mon, Apr 9, 2018 at 4:36 AM, Julian Hyde <jhyde.apa...@gmail.com>
>> wrote:
>> 
>>> For what it’s worth, ROW is standard SQL. If it does what you need, we
>>> should use it.
>>> 
>>> Reading your case quickly, I perceived that you needed a concise way to
>>> assign field names, and AS STRUCT seemed to do that.
>>> 
>>> But staying within the standard is always preferred. BigQuery isn’t
>> always
>>> good at that.
>>> 
>>> Julian
>>> 
>>>> On Apr 5, 2018, at 10:18, Rong Rong <walter...@gmail.com> wrote:
>>>> 
>>>> Thanks for the fantastic proposal @shuyi. I think the STRUCT idea is
>>> great
>>>> considering ROW is not standard SQL either. As a user of calcite I
>> have a
>>>> couple questions.
>>>> 
>>>> Since ROW constructor is so similar with STRUCT, would it be a good
>> idea
>>> to
>>>> consolidate the two syntax? Or have a clear distinction between?
>>>> 
>>>> Whats the relationship going forward with DDL, for example
>> CALCITE-2045.
>>>> DDL seems more flexible in terms of defining the structure not just on
>>>> field names but also field types. Maybe @andrew can share more on the
>> use
>>>> cases on calcite on beam steaming integration?
>>>> 
>>>> Thanks,
>>>> Rong
>>>> 
>>>> 
>>>> On Thu, Apr 5, 2018, 10:05 AM Andrew Pilloud
>> <apill...@google.com.invalid
>>>> 
>>>> wrote:
>>>> 
>>>>> As a user of Calcite working on adding streaming SQL to Apache Beam
>> this
>>>>> sounds like a fantastic proposal. Our initial goal is to be able to
>> run
>>> SQL
>>>>> queries that transform arbitrary JSON objects. Without this syntax
>>> objects
>>>>> must be flattened when they pass through the transform. Is this
>>> something
>>>>> that might make it into 1.17?
>>>>> 
>>>>> We have also had some discussion about adding DDL to Beam so a user
>> can
>>>>> describe the schema of a stream of JSON in pure SQL. Our current
>> though
>>> is
>>>>> to use Big Query compatible STRUCT and ARRAY syntax. Big Query is a
>>> popular
>>>>> sink for our users. Syntax compatible with Big Query would be a big
>> plus
>>>>> for us.
>>>>> 
>>>>> Andrew
>>>>> 
>>>>>> On Thu, Apr 5, 2018 at 12:43 AM Shuyi Chen <suez1...@gmail.com>
>> wrote:
>>>>>> 
>>>>>> @Michael, @Albert, yes, I dont think it is SQL standard. But I think
>>> it's
>>>>>> very useful in the context of streaming SQL, e.g. Flink SQL, where
>> the
>>>>>> sinks can be a database or endpoints with defined protobuf/thrift
>>> schema.
>>>>>> They usually have complex structure. Supporting complex structure in
>>> SQL
>>>>>> output will make it much easier to write to different sinks with
>>>>> predefined
>>>>>> schemas in a unified way,
>>>>>> 
>>>>>> @julian, that's great suggestion, I think instead of extending the
>> ROW
>>>>>> constructor, which is not SQL standard, adding a new extension might
>> be
>>>>> the
>>>>>> right way to go. Looking at the STRUCT big query syntax, we can
>>> implement
>>>>>> something like the following:
>>>>>> 
>>>>>> SELECT STRUCT(a as first_name, b as last_name, STRUCT(c as zip code,
>> d
>>> as
>>>>>> street, e as state) as address) as record FROM example_table
>>>>>> 
>>>>>> On Wed, Apr 4, 2018 at 5:51 PM, Julian Hyde <jhyde.apa...@gmail.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> If I recall correctly, Google BigQuery has SELECT AS STRUCT. It’s
>> not
>>>>>>> standard, but if it does what you need we could consider adopting
>> that
>>>>>>> syntax.
>>>>>>> 
>>>>>>> Julian
>>>>>>> 
>>>>>>>> On Apr 4, 2018, at 10:23 AM, Albert <zinki...@gmail.com> wrote:
>>>>>>>> 
>>>>>>>> if it is not SQL standard, it's just a matter of categorizing it to
>>>>>> some
>>>>>>>> dialect ?
>>>>>>>> 
>>>>>>>>> On Wed, Apr 4, 2018 at 10:19 AM, Michael Mior <mm...@uwaterloo.ca
>>> 
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> Apologies for my silence. I don't really have thoughts on the
>> matter
>>>>>> at
>>>>>>>>> this point. It might be helpful if you can give an example of what
>>>>>>> you're
>>>>>>>>> proposing. Unless I'm missing something (very possible), it's not
>>>>> part
>>>>>>> of
>>>>>>>>> the SQL standard.
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> Michael Mior
>>>>>>>>> mm...@apache.org
>>>>>>>>> 
>>>>>>>>> 2018-04-03 18:48 GMT-04:00 Shuyi Chen <suez1...@gmail.com>:
>>>>>>>>> 
>>>>>>>>>> Friendly ping, any thoughts? Much appreciated.
>>>>>>>>>> 
>>>>>>>>>> Shuyi
>>>>>>>>>> 
>>>>>>>>>>> On Tue, Mar 27, 2018 at 11:59 PM, Shuyi Chen <
>> suez1...@gmail.com>
>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Hi community,
>>>>>>>>>>> 
>>>>>>>>>>> I am thinking of adding the following support in Calcite to
>>>>> support
>>>>>>>>> named
>>>>>>>>>>> row construction, e.g.
>>>>>>>>>>> 
>>>>>>>>>>> SELECT (a as first_name, b as last_name, (c as zip code, d as
>>>>>> street,
>>>>>>> e
>>>>>>>>>> as
>>>>>>>>>>> state) as address) as record FROM example_table
>>>>>>>>>>> 
>>>>>>>>>>> The output will be struct with field names specified in the SQL.
>>>>> The
>>>>>>>>>> usage
>>>>>>>>>>> scenario is that say, in streaming SQL, the downstream sink's
>>>>> schema
>>>>>>>>> can
>>>>>>>>>>> not be changed, so we will need to use SQL to construct a struct
>>>>>> with
>>>>>>>>> the
>>>>>>>>>>> proper naming according to the schema in order to write to the
>>>>>>>>> downstream
>>>>>>>>>>> sinks. Thanks a lot.
>>>>>>>>>>> 
>>>>>>>>>>> Shuyi
>>>>>>>>>>> 
>>>>>>>>>>> --
>>>>>>>>>>> "So you have to trust that the dots will somehow connect in your
>>>>>>>>> future."
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> "So you have to trust that the dots will somehow connect in your
>>>>>>> future."
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> ~~~~~~~~~~~~~~~
>>>>>>>> no mistakes
>>>>>>>> ~~~~~~~~~~~~~~~~~~
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> "So you have to trust that the dots will somehow connect in your
>>> future."
>>>>>> 
>>>>> 
>>> 
>> 
>> 
>> 
>> --
>> "So you have to trust that the dots will somehow connect in your future."
>> 

Reply via email to