Re: [Propose] Create a new HAWQ roadmap page

[Wen Lin]

Hi,

Glad to see this roadmap page is created. I am wondering if some
enhancements for Hadoop YARN related can be added into this roadmap.
1. support "YARN label based scheduling" in HAWQ and libyarn,
https://issues.apache.org/jira/browse/HAWQ-742
2. support Fair Scheduler in HAWQ and libyarn;
3. support resource preemption with Capacity Scheduler in HAWQ and libyarn,
http://hortonworks.com/blog/better-slas-via-resource-preemption-in-yarns-capacityscheduler/

It seems no jira is created for number 2 and 3.

Thanks!

On Thu, Jul 14, 2016 at 8:54 AM, Lei Chang  wrote:

> Added to the page:
> https://cwiki.apache.org/confluence/display/HAWQ/HAWQ+Roadmap
>
> It can be incrementally updated.
>
> Cheers
> Lei
>
>
>
>
> On Wed, Jun 29, 2016 at 5:23 PM, Lei Chang  wrote:
>
> >
> > I classified the items into the following categories. appreciate your
> > comments.
> >
> > Cloud related:
> > [HAWQ-308] - S3 Integration
> > [HAWQ-310] - Snapshot support
> >
> > Data Management Functionality Enhancement
> > [HAWQ-786] - Framework to support pluggable formats and file systems
> > [HAWQ-864] - Support ORC as a native file format
> > [HAWQ-150] - External tables can be designated for both READ and WRITE
> > [HAWQ-304] - Support update and delete on non-heap tables
> > [HAWQ-401] - json type support
> > [HAWQ-319] - REST API for HAWQ
> > [HAWQ-312] - Multiple active master support
> >
> > Performance enhancement
> > [HAWQ-303] - Index support for non-heap tables
> >
> > Languages & Analytics
> > [HAWQ-321] - Support plpython3u
> >
> > Ecosystem:
> > [HAWQ-256] - Integrate Security with Apache Ranger
> > [HAWQ-29] - Refactor HAWQ InputFormat to support Spark/Scala
> >
> > Management & Build
> > [HAWQ-8] - Installing the HAWQ Software thru the Apache Ambari
> > [HAWQ-311] - Data Transfer tool
> > [HAWQ-326] - Support RPM build for HAWQ
> >
> > Cheers
> > Lei
> >
> >
> >
> >
> > On Fri, Jun 24, 2016 at 5:10 PM, Lei Chang  wrote:
> >
> >>
> >> Nice, I created a page and we can discuss the items and put them on the
> >> page.
> >>
> >> For the items, I think it makes sense to add at least items in the jira
> >> roadmap panel, here are some major ones I extracted from the panel.
> looks
> >> better to classify them into categories.
> >>
> >> [HAWQ-786] - Framework to support pluggable formats and file systems
> >> [HAWQ-864] - Support ORC as a native file format
> >> [HAWQ-308] - S3 Integration
> >> [HAWQ-256] - Integrate Security with Apache Ranger
> >> [HAWQ-150] - External tables can be designated for both READ and WRITE
> >> [HAWQ-303] - Index support for non-heap tables
> >> [HAWQ-304] - Support update and delete on non-heap tables
> >> [HAWQ-310] - Snapshot support
> >> [HAWQ-312] - Multiple active master support
> >> [HAWQ-319] - REST API for HAWQ
> >> [HAWQ-321] - Support plpython3u
> >> [HAWQ-29] - Refactor HAWQ InputFormat to support Spark/Scala
> >> [HAWQ-311] - Data Transfer tool
> >> [HAWQ-326] - Support RPM build for HAWQ
> >> [HAWQ-8] - Installing the HAWQ Software thru the Apache Ambari
> >> [HAWQ-752] - build pxf compatible with Apache Hadoop
> >> [HAWQ-401] - json type support
> >>
> >> Cheers
> >> Lei
> >>
> >>
> >>
> >> On Thu, Jun 23, 2016 at 11:23 PM, Vineet Goel 
> wrote:
> >>
> >>> +1 too
> >>>
> >>> I can help start a draft on the wiki based on historical user requests
> >>> and
> >>> trends in the ecosystem. And of course, the roadmap is a living and
> >>> breathing document which will continue to evolve over time based on
> >>> continuous feedback, and more.
> >>>
> >>> -Vineet
> >>>
> >>>
> >>> On Thu, Jun 23, 2016 at 8:18 AM, Kavinder Dhaliwal <
> kdhali...@pivotal.io
> >>> >
> >>> wrote:
> >>>
> >>> > +1 I'm in favor of this. The Zeppelin roadmap is very community
> driven
> >>> and
> >>> > having something similar for HAWQ will go a long way to getting more
> >>> > feedback about the overall direction and goals of HAWQ.
> >>> >
> >>> > On Thu, Jun 23, 2016 at 2:02 AM, Lei Chang 
> >>> wrote:
> >>> >
> >>> > > Hi Guys,
> >>> > >
> >>> > > I noticed there are a lot of requests about hawq roadmaps coming
> >>> from the
> >>> > > offline hawq activities (meetup et al).
> >>> > >
> >>> > > Although we have the list of backlog JIRAs on our JIRA page
> >>> > > <
> >>> > >
> >>> >
> >>>
> https://issues.apache.org/jira/browse/HAWQ/?selectedTab=com.atlassian.jira.jira-projects-plugin:roadmap-panel
> >>> > > >.
> >>> > > But it does not give a high level description. A good example from
> >>> other
> >>> > > communities is here:
> >>> > >
> >>> https://cwiki.apache.org/confluence/display/ZEPPELIN/Zeppelin+Roadmap
> >>> > >
> >>> > > So I am proposing we have a similar HAWQ Roadmap page maintained on
> >>> our
> >>> > > wiki page.
> >>> > >
> >>> > > Thoughts?
> >>> > >
> >>> > > Cheers
> >>> > > Lei
> >>> > >
> >>> >
> >>>
> >>
> >>
> >
>

Re: [Propose] Create a new HAWQ roadmap page

[Lei Chang]

Added to the page:
https://cwiki.apache.org/confluence/display/HAWQ/HAWQ+Roadmap

It can be incrementally updated.

Cheers
Lei




On Wed, Jun 29, 2016 at 5:23 PM, Lei Chang  wrote:

>
> I classified the items into the following categories. appreciate your
> comments.
>
> Cloud related:
> [HAWQ-308] - S3 Integration
> [HAWQ-310] - Snapshot support
>
> Data Management Functionality Enhancement
> [HAWQ-786] - Framework to support pluggable formats and file systems
> [HAWQ-864] - Support ORC as a native file format
> [HAWQ-150] - External tables can be designated for both READ and WRITE
> [HAWQ-304] - Support update and delete on non-heap tables
> [HAWQ-401] - json type support
> [HAWQ-319] - REST API for HAWQ
> [HAWQ-312] - Multiple active master support
>
> Performance enhancement
> [HAWQ-303] - Index support for non-heap tables
>
> Languages & Analytics
> [HAWQ-321] - Support plpython3u
>
> Ecosystem:
> [HAWQ-256] - Integrate Security with Apache Ranger
> [HAWQ-29] - Refactor HAWQ InputFormat to support Spark/Scala
>
> Management & Build
> [HAWQ-8] - Installing the HAWQ Software thru the Apache Ambari
> [HAWQ-311] - Data Transfer tool
> [HAWQ-326] - Support RPM build for HAWQ
>
> Cheers
> Lei
>
>
>
>
> On Fri, Jun 24, 2016 at 5:10 PM, Lei Chang  wrote:
>
>>
>> Nice, I created a page and we can discuss the items and put them on the
>> page.
>>
>> For the items, I think it makes sense to add at least items in the jira
>> roadmap panel, here are some major ones I extracted from the panel. looks
>> better to classify them into categories.
>>
>> [HAWQ-786] - Framework to support pluggable formats and file systems
>> [HAWQ-864] - Support ORC as a native file format
>> [HAWQ-308] - S3 Integration
>> [HAWQ-256] - Integrate Security with Apache Ranger
>> [HAWQ-150] - External tables can be designated for both READ and WRITE
>> [HAWQ-303] - Index support for non-heap tables
>> [HAWQ-304] - Support update and delete on non-heap tables
>> [HAWQ-310] - Snapshot support
>> [HAWQ-312] - Multiple active master support
>> [HAWQ-319] - REST API for HAWQ
>> [HAWQ-321] - Support plpython3u
>> [HAWQ-29] - Refactor HAWQ InputFormat to support Spark/Scala
>> [HAWQ-311] - Data Transfer tool
>> [HAWQ-326] - Support RPM build for HAWQ
>> [HAWQ-8] - Installing the HAWQ Software thru the Apache Ambari
>> [HAWQ-752] - build pxf compatible with Apache Hadoop
>> [HAWQ-401] - json type support
>>
>> Cheers
>> Lei
>>
>>
>>
>> On Thu, Jun 23, 2016 at 11:23 PM, Vineet Goel  wrote:
>>
>>> +1 too
>>>
>>> I can help start a draft on the wiki based on historical user requests
>>> and
>>> trends in the ecosystem. And of course, the roadmap is a living and
>>> breathing document which will continue to evolve over time based on
>>> continuous feedback, and more.
>>>
>>> -Vineet
>>>
>>>
>>> On Thu, Jun 23, 2016 at 8:18 AM, Kavinder Dhaliwal >> >
>>> wrote:
>>>
>>> > +1 I'm in favor of this. The Zeppelin roadmap is very community driven
>>> and
>>> > having something similar for HAWQ will go a long way to getting more
>>> > feedback about the overall direction and goals of HAWQ.
>>> >
>>> > On Thu, Jun 23, 2016 at 2:02 AM, Lei Chang 
>>> wrote:
>>> >
>>> > > Hi Guys,
>>> > >
>>> > > I noticed there are a lot of requests about hawq roadmaps coming
>>> from the
>>> > > offline hawq activities (meetup et al).
>>> > >
>>> > > Although we have the list of backlog JIRAs on our JIRA page
>>> > > <
>>> > >
>>> >
>>> https://issues.apache.org/jira/browse/HAWQ/?selectedTab=com.atlassian.jira.jira-projects-plugin:roadmap-panel
>>> > > >.
>>> > > But it does not give a high level description. A good example from
>>> other
>>> > > communities is here:
>>> > >
>>> https://cwiki.apache.org/confluence/display/ZEPPELIN/Zeppelin+Roadmap
>>> > >
>>> > > So I am proposing we have a similar HAWQ Roadmap page maintained on
>>> our
>>> > > wiki page.
>>> > >
>>> > > Thoughts?
>>> > >
>>> > > Cheers
>>> > > Lei
>>> > >
>>> >
>>>
>>
>>
>

Re: [Propose] Create a new HAWQ roadmap page

[Lei Chang]

I classified the items into the following categories. appreciate your
comments.

Cloud related:
[HAWQ-308] - S3 Integration
[HAWQ-310] - Snapshot support

Data Management Functionality Enhancement
[HAWQ-786] - Framework to support pluggable formats and file systems
[HAWQ-864] - Support ORC as a native file format
[HAWQ-150] - External tables can be designated for both READ and WRITE
[HAWQ-304] - Support update and delete on non-heap tables
[HAWQ-401] - json type support
[HAWQ-319] - REST API for HAWQ
[HAWQ-312] - Multiple active master support

Performance enhancement
[HAWQ-303] - Index support for non-heap tables

Languages & Analytics
[HAWQ-321] - Support plpython3u

Ecosystem:
[HAWQ-256] - Integrate Security with Apache Ranger
[HAWQ-29] - Refactor HAWQ InputFormat to support Spark/Scala

Management & Build
[HAWQ-8] - Installing the HAWQ Software thru the Apache Ambari
[HAWQ-311] - Data Transfer tool
[HAWQ-326] - Support RPM build for HAWQ

Cheers
Lei




On Fri, Jun 24, 2016 at 5:10 PM, Lei Chang  wrote:

>
> Nice, I created a page and we can discuss the items and put them on the
> page.
>
> For the items, I think it makes sense to add at least items in the jira
> roadmap panel, here are some major ones I extracted from the panel. looks
> better to classify them into categories.
>
> [HAWQ-786] - Framework to support pluggable formats and file systems
> [HAWQ-864] - Support ORC as a native file format
> [HAWQ-308] - S3 Integration
> [HAWQ-256] - Integrate Security with Apache Ranger
> [HAWQ-150] - External tables can be designated for both READ and WRITE
> [HAWQ-303] - Index support for non-heap tables
> [HAWQ-304] - Support update and delete on non-heap tables
> [HAWQ-310] - Snapshot support
> [HAWQ-312] - Multiple active master support
> [HAWQ-319] - REST API for HAWQ
> [HAWQ-321] - Support plpython3u
> [HAWQ-29] - Refactor HAWQ InputFormat to support Spark/Scala
> [HAWQ-311] - Data Transfer tool
> [HAWQ-326] - Support RPM build for HAWQ
> [HAWQ-8] - Installing the HAWQ Software thru the Apache Ambari
> [HAWQ-752] - build pxf compatible with Apache Hadoop
> [HAWQ-401] - json type support
>
> Cheers
> Lei
>
>
>
> On Thu, Jun 23, 2016 at 11:23 PM, Vineet Goel  wrote:
>
>> +1 too
>>
>> I can help start a draft on the wiki based on historical user requests and
>> trends in the ecosystem. And of course, the roadmap is a living and
>> breathing document which will continue to evolve over time based on
>> continuous feedback, and more.
>>
>> -Vineet
>>
>>
>> On Thu, Jun 23, 2016 at 8:18 AM, Kavinder Dhaliwal 
>> wrote:
>>
>> > +1 I'm in favor of this. The Zeppelin roadmap is very community driven
>> and
>> > having something similar for HAWQ will go a long way to getting more
>> > feedback about the overall direction and goals of HAWQ.
>> >
>> > On Thu, Jun 23, 2016 at 2:02 AM, Lei Chang 
>> wrote:
>> >
>> > > Hi Guys,
>> > >
>> > > I noticed there are a lot of requests about hawq roadmaps coming from
>> the
>> > > offline hawq activities (meetup et al).
>> > >
>> > > Although we have the list of backlog JIRAs on our JIRA page
>> > > <
>> > >
>> >
>> https://issues.apache.org/jira/browse/HAWQ/?selectedTab=com.atlassian.jira.jira-projects-plugin:roadmap-panel
>> > > >.
>> > > But it does not give a high level description. A good example from
>> other
>> > > communities is here:
>> > > https://cwiki.apache.org/confluence/display/ZEPPELIN/Zeppelin+Roadmap
>> > >
>> > > So I am proposing we have a similar HAWQ Roadmap page maintained on
>> our
>> > > wiki page.
>> > >
>> > > Thoughts?
>> > >
>> > > Cheers
>> > > Lei
>> > >
>> >
>>
>
>

Re: [Propose] Create a new HAWQ roadmap page

[Lei Chang]

Nice, I created a page and we can discuss the items and put them on the
page.

For the items, I think it makes sense to add at least items in the jira
roadmap panel, here are some major ones I extracted from the panel. looks
better to classify them into categories.

[HAWQ-786] - Framework to support pluggable formats and file systems
[HAWQ-864] - Support ORC as a native file format
[HAWQ-308] - S3 Integration
[HAWQ-256] - Integrate Security with Apache Ranger
[HAWQ-150] - External tables can be designated for both READ and WRITE
[HAWQ-303] - Index support for non-heap tables
[HAWQ-304] - Support update and delete on non-heap tables
[HAWQ-310] - Snapshot support
[HAWQ-312] - Multiple active master support
[HAWQ-319] - REST API for HAWQ
[HAWQ-321] - Support plpython3u
[HAWQ-29] - Refactor HAWQ InputFormat to support Spark/Scala
[HAWQ-311] - Data Transfer tool
[HAWQ-326] - Support RPM build for HAWQ
[HAWQ-8] - Installing the HAWQ Software thru the Apache Ambari
[HAWQ-752] - build pxf compatible with Apache Hadoop
[HAWQ-401] - json type support

Cheers
Lei



On Thu, Jun 23, 2016 at 11:23 PM, Vineet Goel  wrote:

> +1 too
>
> I can help start a draft on the wiki based on historical user requests and
> trends in the ecosystem. And of course, the roadmap is a living and
> breathing document which will continue to evolve over time based on
> continuous feedback, and more.
>
> -Vineet
>
>
> On Thu, Jun 23, 2016 at 8:18 AM, Kavinder Dhaliwal 
> wrote:
>
> > +1 I'm in favor of this. The Zeppelin roadmap is very community driven
> and
> > having something similar for HAWQ will go a long way to getting more
> > feedback about the overall direction and goals of HAWQ.
> >
> > On Thu, Jun 23, 2016 at 2:02 AM, Lei Chang  wrote:
> >
> > > Hi Guys,
> > >
> > > I noticed there are a lot of requests about hawq roadmaps coming from
> the
> > > offline hawq activities (meetup et al).
> > >
> > > Although we have the list of backlog JIRAs on our JIRA page
> > > <
> > >
> >
> https://issues.apache.org/jira/browse/HAWQ/?selectedTab=com.atlassian.jira.jira-projects-plugin:roadmap-panel
> > > >.
> > > But it does not give a high level description. A good example from
> other
> > > communities is here:
> > > https://cwiki.apache.org/confluence/display/ZEPPELIN/Zeppelin+Roadmap
> > >
> > > So I am proposing we have a similar HAWQ Roadmap page maintained on our
> > > wiki page.
> > >
> > > Thoughts?
> > >
> > > Cheers
> > > Lei
> > >
> >
>

Re: [Propose] Create a new HAWQ roadmap page

[Vineet Goel]

+1 too

I can help start a draft on the wiki based on historical user requests and
trends in the ecosystem. And of course, the roadmap is a living and
breathing document which will continue to evolve over time based on
continuous feedback, and more.

-Vineet


On Thu, Jun 23, 2016 at 8:18 AM, Kavinder Dhaliwal 
wrote:

> +1 I'm in favor of this. The Zeppelin roadmap is very community driven and
> having something similar for HAWQ will go a long way to getting more
> feedback about the overall direction and goals of HAWQ.
>
> On Thu, Jun 23, 2016 at 2:02 AM, Lei Chang  wrote:
>
> > Hi Guys,
> >
> > I noticed there are a lot of requests about hawq roadmaps coming from the
> > offline hawq activities (meetup et al).
> >
> > Although we have the list of backlog JIRAs on our JIRA page
> > <
> >
> https://issues.apache.org/jira/browse/HAWQ/?selectedTab=com.atlassian.jira.jira-projects-plugin:roadmap-panel
> > >.
> > But it does not give a high level description. A good example from other
> > communities is here:
> > https://cwiki.apache.org/confluence/display/ZEPPELIN/Zeppelin+Roadmap
> >
> > So I am proposing we have a similar HAWQ Roadmap page maintained on our
> > wiki page.
> >
> > Thoughts?
> >
> > Cheers
> > Lei
> >
>

Re: [Propose] Create a new HAWQ roadmap page

[Kavinder Dhaliwal]

+1 I'm in favor of this. The Zeppelin roadmap is very community driven and
having something similar for HAWQ will go a long way to getting more
feedback about the overall direction and goals of HAWQ.

On Thu, Jun 23, 2016 at 2:02 AM, Lei Chang  wrote:

> Hi Guys,
>
> I noticed there are a lot of requests about hawq roadmaps coming from the
> offline hawq activities (meetup et al).
>
> Although we have the list of backlog JIRAs on our JIRA page
> <
> https://issues.apache.org/jira/browse/HAWQ/?selectedTab=com.atlassian.jira.jira-projects-plugin:roadmap-panel
> >.
> But it does not give a high level description. A good example from other
> communities is here:
> https://cwiki.apache.org/confluence/display/ZEPPELIN/Zeppelin+Roadmap
>
> So I am proposing we have a similar HAWQ Roadmap page maintained on our
> wiki page.
>
> Thoughts?
>
> Cheers
> Lei
>

[Propose] Create a new HAWQ roadmap page

[Lei Chang]

Hi Guys,

I noticed there are a lot of requests about hawq roadmaps coming from the
offline hawq activities (meetup et al).

Although we have the list of backlog JIRAs on our JIRA page
.
But it does not give a high level description. A good example from other
communities is here:
https://cwiki.apache.org/confluence/display/ZEPPELIN/Zeppelin+Roadmap

So I am proposing we have a similar HAWQ Roadmap page maintained on our
wiki page.

Thoughts?

Cheers
Lei