Re: Release Today/Tomorrow

[Chris Kellogg]

It's a little more involved then i thought. We could do a release and then
follow up with another one soon.

On Mon, Feb 26, 2018 at 8:50 PM, Karthik Ramasamy 
wrote:

> We can do a release. Chris wanted to get a PR in. Chris - any update?
>
> On Mon, Feb 26, 2018 at 11:38 PM Jerry Peng 
> wrote:
>
> > We haven't done a release in a while.  Should we do a release today or
> > tomorrow?
> >
> > Best,
> >
> > Jerry
> >
>

Re: Release Today/Tomorrow

[Sanjeev Kulkarni]

+1

On Mon, Feb 26, 2018 at 8:38 PM, Jerry Peng 
wrote:

> We haven't done a release in a while.  Should we do a release today or
> tomorrow?
>
> Best,
>
> Jerry
>

Release Today/Tomorrow

[Jerry Peng]

We haven't done a release in a while.  Should we do a release today or tomorrow?

Best,

Jerry

Re: Proposing Changes To Heron

[Yaliang Wang]

Josh,

Totally agree with your concern. I was bringing that idea into conversation and 
thought that as a back up solution. Since Heron is getting more and more 
popular, it would be really nice to have SQL support. I think having a built-in 
Heron SQL can shorten the development iteration since we will have less concern 
of abstraction and generalization in implementation.

Best,
Yaliang

> On Feb 26, 2018, at 7:17 PM, Josh Fischer  wrote:
> 
> Yaliang,
> 
> I think this is a fantastic idea and I agree about the code maintenance
> being a cost.   I have a concern that creating a smaller project may get
> abandoned, especially if it had a smaller following.   One of the nice
> things about Heron is the large community and list of core contributors
> behind it.  But, I don't want to abandon this idea.  I think, for me at
> least, that it would make sense to get Storm SQL running in Heron and take
> what we learned from that experience and apply it to a third part project
> if there is a need/demand for it.  What do you think?
> 
> -Josh
> 
> On Mon, Feb 26, 2018 at 6:51 PM, Yaliang Wang 
> wrote:
> 
>> Sounds like a very great feature to have. A question I have: will it be
>> feasible to start a separate project to support SQL on Heron-like streaming?
>> 
>> - I’m imaging that there will be a lot code similar/same to Storm SQL.
>> - Only the last step of the three steps(parse sql -> logical/physical plan
>> -> heron topology) you mentioned is specified for Heron. The first two
>> steps can be shared for other heron-like streaming vendors.
>> - The native support for SQL inside the Heron project will give extra
>> advertising/marketing bonus but with an increase of the code maintenance
>> cost, especially, if it requires APIs that not very popular and may be
>> changed over time. However, a separate project can target a specific
>> version of Heron.
>> 
>> Best,
>> Yaliang
>> 
>>> On Feb 26, 2018, at 12:48 PM, Eren Avsarogullari <
>> erenavsarogull...@gmail.com> wrote:
>>> 
>>> +1 for Heron SQL Support. Thanks Josh.
>>> 
>>> On 26 February 2018 at 18:42, Karthik Ramasamy 
>> wrote:
>>> 
 Thanks Josh for initiating this. It will be a great feature to add for
 Heron.
 
 cheers
 /karthik
 
> On Feb 26, 2018, at 11:11 AM, Josh Fischer 
>> wrote:
> 
> Jerry,
> 
> Great point.  Lets keep things simple for the migration to make sure
>> the
> implementation is correct.  Then we can modify from there.
> 
> On Sun, Feb 25, 2018 at 11:28 PM, Jerry Peng <
 jerry.boyang.p...@gmail.com>
> wrote:
> 
>> Thanks Josh for taking the initiative to get this start!  SQL on Heron
>> will be a great feature! The plan sounds great to me.  Lets first get
>> an initial version of the Heron SQL out and then we can worry about
>> custom / user defined sources and sinks.  We can even start talking
>> about UDFs (User defined functions) at that point!
>> 
>> Best,
>> 
>> Jerry
>> 
>> On Sun, Feb 25, 2018 at 9:05 PM, Josh Fischer 
 wrote:
>>> Please see this google drive link for adding comments.  I will copy
>> and
>>> paste the drive doc below as well.
>>> 
>>> https://docs.google.com/document/d/1PxLCyR_H-
>> mOgPjyFj3DhWXryKW21CH2zFWwzTnqjfEA/edit?usp=sharing
>>> 
>>> 
>>> Proposal Below
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> *I am writing this document to propose changes and to start
 conversations
>>> on adding functionality similar to Storm SQL to Heron.  We would call
 it
>>> Heron SQL.  After reviewing how the code is structured in Storm I
>> have
>> some
>>> suggestions and questions relating to the implementation into the
>> Heron
>>> code base. - High Level Overview Of Code Workflow (Keeping Similar to
>>> Storm)- We would parse the sql with calcite to create the logical and
>>> physical plans- We would then convert the logical and physical plans
 to a
>>> Heron Topology- We would then submit the Heron Topology into the
>> Heron
>>> System - Some thoughts on code structure and overall functionality- I
>> think
>>> we should place the Heron SQL code base as a top level directory in
>> the
>>> repo. - I will have to add the command “sql” to the Heron command
>> line
>> code
>>> in python.- As a first pass implementation users  can interact with
 Heron
>>> SQL via the following command - heron sql  -
 We
>>> will also support the explain command for displaying the query plan,
 this
>>> will not deploy the topology- heron sql  --explain- After
>> the
>>> first pass implementation is working smoothly, we can then add an
>>> interactive command line interface to accept sql on the fly by
>> omitting
>> the
>>> sql file argument- Heron sql - 

Re: Proposing Changes To Heron

[Karthik Ramasamy]

Thanks Josh for initiating this. It will be a great feature to add for Heron.

cheers
/karthik

> On Feb 26, 2018, at 11:11 AM, Josh Fischer  wrote:
> 
> Jerry,
> 
> Great point.  Lets keep things simple for the migration to make sure the
> implementation is correct.  Then we can modify from there.
> 
> On Sun, Feb 25, 2018 at 11:28 PM, Jerry Peng 
> wrote:
> 
>> Thanks Josh for taking the initiative to get this start!  SQL on Heron
>> will be a great feature! The plan sounds great to me.  Lets first get
>> an initial version of the Heron SQL out and then we can worry about
>> custom / user defined sources and sinks.  We can even start talking
>> about UDFs (User defined functions) at that point!
>> 
>> Best,
>> 
>> Jerry
>> 
>> On Sun, Feb 25, 2018 at 9:05 PM, Josh Fischer  wrote:
>>> Please see this google drive link for adding comments.  I will copy and
>>> paste the drive doc below as well.
>>> 
>>> https://docs.google.com/document/d/1PxLCyR_H-
>> mOgPjyFj3DhWXryKW21CH2zFWwzTnqjfEA/edit?usp=sharing
>>> 
>>> 
>>> Proposal Below
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> *I am writing this document to propose changes and to start conversations
>>> on adding functionality similar to Storm SQL to Heron.  We would call it
>>> Heron SQL.  After reviewing how the code is structured in Storm I have
>> some
>>> suggestions and questions relating to the implementation into the Heron
>>> code base. - High Level Overview Of Code Workflow (Keeping Similar to
>>> Storm)- We would parse the sql with calcite to create the logical and
>>> physical plans- We would then convert the logical and physical plans to a
>>> Heron Topology- We would then submit the Heron Topology into the Heron
>>> System - Some thoughts on code structure and overall functionality- I
>> think
>>> we should place the Heron SQL code base as a top level directory in the
>>> repo. - I will have to add the command “sql” to the Heron command line
>> code
>>> in python.- As a first pass implementation users  can interact with Heron
>>> SQL via the following command - heron sql  - We
>>> will also support the explain command for displaying the query plan, this
>>> will not deploy the topology- heron sql  --explain- After the
>>> first pass implementation is working smoothly, we can then add an
>>> interactive command line interface to accept sql on the fly by omitting
>> the
>>> sql file argument- Heron sql - We would support all of the
>>> existing functionality in Storm SQL today with the exception of being
>>> dependent on trident.  We would use Storm SQL as a way to deploy
>> topologies
>>> into Heron.  Similar to how you deploy topologies with the Streamlet,
>>> Topology, and ECO APIs- Questions- Do we see any issue with this plan to
>>> implement?- I believe we would have to supply an external jar at times to
>>> connect to external data sources, such as reuse of kafka libraries or
>>> database drivers.  I see that Storm has few external connectors for
>> mongo,
>>> kafka, redis and hdfs.  Do we want to limit users to what we decide to
>>> build as connectors or do we want to give them the ability to load
>> external
>>> jars at submit time? I don’t think heron offers the ability to pass extra
>>> jars to via the “--jars” or “--artifacts” flags like Storm does today.
>>> Would this be the correct way to pull in external jars?  Does anyone
>> have a
>>> different idea?  I’m thinking that this might be a v2 feature after we
>> get
>>> Heron sql working well.  Ideas, thoughts or concerns?- Is there anything
>> I
>>> missed?*
>> 

Re: Proposing Changes To Heron

[Josh Fischer]

Jerry,

Great point.  Lets keep things simple for the migration to make sure the
implementation is correct.  Then we can modify from there.

On Sun, Feb 25, 2018 at 11:28 PM, Jerry Peng 
wrote:

> Thanks Josh for taking the initiative to get this start!  SQL on Heron
> will be a great feature! The plan sounds great to me.  Lets first get
> an initial version of the Heron SQL out and then we can worry about
> custom / user defined sources and sinks.  We can even start talking
> about UDFs (User defined functions) at that point!
>
> Best,
>
> Jerry
>
> On Sun, Feb 25, 2018 at 9:05 PM, Josh Fischer  wrote:
> > Please see this google drive link for adding comments.  I will copy and
> > paste the drive doc below as well.
> >
> > https://docs.google.com/document/d/1PxLCyR_H-
> mOgPjyFj3DhWXryKW21CH2zFWwzTnqjfEA/edit?usp=sharing
> >
> >
> > Proposal Below
> >
> >
> >
> >
> >
> >
> >
> > *I am writing this document to propose changes and to start conversations
> > on adding functionality similar to Storm SQL to Heron.  We would call it
> > Heron SQL.  After reviewing how the code is structured in Storm I have
> some
> > suggestions and questions relating to the implementation into the Heron
> > code base. - High Level Overview Of Code Workflow (Keeping Similar to
> > Storm)- We would parse the sql with calcite to create the logical and
> > physical plans- We would then convert the logical and physical plans to a
> > Heron Topology- We would then submit the Heron Topology into the Heron
> > System - Some thoughts on code structure and overall functionality- I
> think
> >  we should place the Heron SQL code base as a top level directory in the
> > repo. - I will have to add the command “sql” to the Heron command line
> code
> > in python.- As a first pass implementation users  can interact with Heron
> > SQL via the following command - heron sql  - We
> > will also support the explain command for displaying the query plan, this
> > will not deploy the topology- heron sql  --explain- After the
> > first pass implementation is working smoothly, we can then add an
> > interactive command line interface to accept sql on the fly by omitting
> the
> > sql file argument- Heron sql - We would support all of the
> > existing functionality in Storm SQL today with the exception of being
> > dependent on trident.  We would use Storm SQL as a way to deploy
> topologies
> > into Heron.  Similar to how you deploy topologies with the Streamlet,
> > Topology, and ECO APIs- Questions- Do we see any issue with this plan to
> > implement?- I believe we would have to supply an external jar at times to
> > connect to external data sources, such as reuse of kafka libraries or
> > database drivers.  I see that Storm has few external connectors for
> mongo,
> > kafka, redis and hdfs.  Do we want to limit users to what we decide to
> > build as connectors or do we want to give them the ability to load
> external
> > jars at submit time? I don’t think heron offers the ability to pass extra
> > jars to via the “--jars” or “--artifacts” flags like Storm does today.
> > Would this be the correct way to pull in external jars?  Does anyone
> have a
> > different idea?  I’m thinking that this might be a v2 feature after we
> get
> > Heron sql working well.  Ideas, thoughts or concerns?- Is there anything
> I
> > missed?*
>

Re: Proposing Changes To Heron

[Ning Wang]

+1 for Heron SQL

On Sun, Feb 25, 2018 at 9:28 PM, Jerry Peng 
wrote:

> Thanks Josh for taking the initiative to get this start!  SQL on Heron
> will be a great feature! The plan sounds great to me.  Lets first get
> an initial version of the Heron SQL out and then we can worry about
> custom / user defined sources and sinks.  We can even start talking
> about UDFs (User defined functions) at that point!
>
> Best,
>
> Jerry
>
> On Sun, Feb 25, 2018 at 9:05 PM, Josh Fischer  wrote:
> > Please see this google drive link for adding comments.  I will copy and
> > paste the drive doc below as well.
> >
> > https://docs.google.com/document/d/1PxLCyR_H-
> mOgPjyFj3DhWXryKW21CH2zFWwzTnqjfEA/edit?usp=sharing
> >
> >
> > Proposal Below
> >
> >
> >
> >
> >
> >
> >
> > *I am writing this document to propose changes and to start conversations
> > on adding functionality similar to Storm SQL to Heron.  We would call it
> > Heron SQL.  After reviewing how the code is structured in Storm I have
> some
> > suggestions and questions relating to the implementation into the Heron
> > code base. - High Level Overview Of Code Workflow (Keeping Similar to
> > Storm)- We would parse the sql with calcite to create the logical and
> > physical plans- We would then convert the logical and physical plans to a
> > Heron Topology- We would then submit the Heron Topology into the Heron
> > System - Some thoughts on code structure and overall functionality- I
> think
> >  we should place the Heron SQL code base as a top level directory in the
> > repo. - I will have to add the command “sql” to the Heron command line
> code
> > in python.- As a first pass implementation users  can interact with Heron
> > SQL via the following command - heron sql  - We
> > will also support the explain command for displaying the query plan, this
> > will not deploy the topology- heron sql  --explain- After the
> > first pass implementation is working smoothly, we can then add an
> > interactive command line interface to accept sql on the fly by omitting
> the
> > sql file argument- Heron sql - We would support all of the
> > existing functionality in Storm SQL today with the exception of being
> > dependent on trident.  We would use Storm SQL as a way to deploy
> topologies
> > into Heron.  Similar to how you deploy topologies with the Streamlet,
> > Topology, and ECO APIs- Questions- Do we see any issue with this plan to
> > implement?- I believe we would have to supply an external jar at times to
> > connect to external data sources, such as reuse of kafka libraries or
> > database drivers.  I see that Storm has few external connectors for
> mongo,
> > kafka, redis and hdfs.  Do we want to limit users to what we decide to
> > build as connectors or do we want to give them the ability to load
> external
> > jars at submit time? I don’t think heron offers the ability to pass extra
> > jars to via the “--jars” or “--artifacts” flags like Storm does today.
> > Would this be the correct way to pull in external jars?  Does anyone
> have a
> > different idea?  I’m thinking that this might be a v2 feature after we
> get
> > Heron sql working well.  Ideas, thoughts or concerns?- Is there anything
> I
> > missed?*
>