Re: [VOTE] HAWQ 2.0.0-incubating Release

[Lei Chang]

Good discussion. since it is the first hawq release, all the release
related processes are not discussed on this mailing list.

I think it is a good time to have an open discussion around it.

Cheers
Lei


On Thu, Jul 7, 2016 at 10:36 AM, Vineet Goel  wrote:

> Radar,
>
> I understand that we have a clean slate right now in terms of releases, so
> it's not a huge issue re-creating the branch. However, going forward, we
> should follow the process to be able to do proper release management and
> change control.
>
> So, should we make an exception for this one time in the spirit of getting
> our first release out asap?
> I think Goden, as the release manager should make that call.
>
> Thanks
> -Vineet
>
>
>
> On Wed, Jul 6, 2016 at 7:02 PM, Radar Da lei  wrote:
>
> > Hi Vineet,
> >
> > Merge or re-create the branch is almost the same for our current status.
> >
> > Re-create the branch can make all our good changes in, and it's more
> clear,
> > users can easy find out where it cut out, just following one version
> change
> > commit.
> >
> > Anyway, I'm fine with each way. Thanks.
> >
> > Regards,
> > Radar
> >
> > On Thu, Jul 7, 2016 at 9:50 AM, Vineet Goel  wrote:
> >
> > > Radar,
> > >
> > > The whole point of cutting a branch was to stabilize it for the first
> > > release. Why would we discard it and start a new one from master? Any
> > > changes since the 2.0.0-incubating branch needs to go in the next
> release
> > > scope (unless selective back-porting is necessary to meet the first
> > release
> > > scope).
> > >
> > > Thanks
> > > -Vineet
> > >
> > >
> > > On Wed, Jul 6, 2016 at 6:39 PM, Radar Da lei  wrote:
> > >
> > > > Since we have a lot of commits for this release after creating the
> > > branch.
> > > > I will re-create the branch. So the first blocking issue is gone.
> > Nothing
> > > > need to be merged.
> > > >
> > > > Any concerns, please let me know. Thanks.
> > > >
> > > > Regards,
> > > > Radar
> > > >
> > > > On Thu, Jul 7, 2016 at 2:04 AM, Ting(Goden) Yao 
> > wrote:
> > > >
> > > > > So we have 2 blocking issues:
> > > > > https://issues.apache.org/jira/browse/HAWQ-867  (need to port this
> > to
> > > > > incubating branch)
> > > > > https://issues.apache.org/jira/browse/HAWQ-892  (version naming
> > issue)
> > > > > Anyone can help to resolve them, please reply ASAP.
> > > > > -Goden
> > > > >
> > > > > On Tue, Jul 5, 2016 at 5:59 PM Ting(Goden) Yao 
> > > wrote:
> > > > >
> > > > > > We found an issue during discussion:
> > > > > > https://issues.apache.org/jira/browse/HAWQ-892
> > > > > > We need to contain "incubating" in the version and hawq --version
> > > > > command.
> > > > > >
> > > > > > Thanks
> > > > > > -Goden
> > > > > >
> > > > > > On Tue, Jul 5, 2016 at 10:24 AM Ting(Goden) Yao  >
> > > > wrote:
> > > > > >
> > > > > >> @Roman - for this JIRA, per Lei's suggestion, it's not blocking
> > this
> > > > > >> release (as no IP issues) , so I'll update the release info in
> the
> > > > JIRA.
> > > > > >> HAWQ-783 
> Remove
> > > > > quicklz
> > > > > >> in medadata
> > > > > >>
> > > > > >> On Tue, Jul 5, 2016 at 10:22 AM Ting(Goden) Yao <
> t...@pivotal.io>
> > > > > wrote:
> > > > > >>
> > > > > >>> Thanks Guo, for your info. This JIRA was not marked in this
> > > release.
> > > > so
> > > > > >>> next time, if you think this is required, please mark it for
> the
> > > > > release
> > > > > >>> and contact release manager ASAP.
> > > > > >>> I'm checking this JIRA for details but I see it is closed
> > already.
> > > > > >>>
> > > > > >>> On Tue, Jul 5, 2016 at 10:20 AM Roman Shaposhnik <
> > > > ro...@shaposhnik.org
> > > > > >
> > > > > >>> wrote:
> > > > > >>>
> > > > >  On Tue, Jul 5, 2016 at 2:25 AM, Guo Gang 
> > > wrote:
> > > > >  > Goden, It seems that we need all of the features could be
> > > compiled
> > > > > in
> > > > >  > source tarball (i.e. non-git workspace) before the release,
> > but
> > > > this
> > > > >  seems
> > > > >  > to be not the case in the tarball. See JIRA HAWQ-867
> (Replace
> > > the
> > > > >  > git-submobule mechanism with git-clone). I think we need a
> new
> > > > > source
> > > > >  > version.
> > > > > 
> > > > >  Guo, Goden, it would be great if you guys could help all of us
> > > > follow
> > > > >  this
> > > > >  release train by filing blocking JIRAs for the
> 2.0.0-incubating
> > > > >  release. The
> > > > >  way it currently stands, we seem to only have 1 blocking JIRA:
> > > > > 
> > > > > 
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/HAWQ/fixforversion/12334000/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
> > > > > 
> > > > >  Thanks,
> > > > >  Roman.
> > > > > 
> > > > > >>>
> > > > >
> > > >
> > >
> 

Re: [Propose] New PXF profile optimized for ORC (predicate pushdown)

[Hubert Zhang]

+1 for lazy reader, It can save a lot of decompression and
deserialization(CPU bound) time.

On Wed, Jul 6, 2016 at 7:00 AM, Roman Shaposhnik 
wrote:

> On Tue, Jul 5, 2016 at 12:01 PM, Shivram Mani 
> wrote:
> > I've created the following jira HAWQ-866
> >  which is focussed on
> > improving/enhancing the existing PXF profile to read ORC files. The goal
> is
> > to make use of the underlying ORC reader's capability of supporting
> > predicate push-down among others.
> >
> > Presto has also contributed an alternative ORC reader which provides both
> > predicate push down and Lazy reads
> >
> https://code.facebook.com/posts/370832626374903/even-faster-data-at-the-speed-of-presto-orc/
> > .
> >
> > Will be evaluating both the options as part of this effort.
>
> Great to see this effort! Do you plan to come up with any kind of
> benchmark to
> be able to compare the native ORC reader vs. PXF ORC reader performance
> and capabilities?
>
> Or does it really all just boil down to TPC?
>
> Thanks,
> Roman.
>



-- 
Thanks

Hubert Zhang

Re: [VOTE] HAWQ 2.0.0-incubating Release

[Vineet Goel]

Radar,

I understand that we have a clean slate right now in terms of releases, so
it's not a huge issue re-creating the branch. However, going forward, we
should follow the process to be able to do proper release management and
change control.

So, should we make an exception for this one time in the spirit of getting
our first release out asap?
I think Goden, as the release manager should make that call.

Thanks
-Vineet



On Wed, Jul 6, 2016 at 7:02 PM, Radar Da lei  wrote:

> Hi Vineet,
>
> Merge or re-create the branch is almost the same for our current status.
>
> Re-create the branch can make all our good changes in, and it's more clear,
> users can easy find out where it cut out, just following one version change
> commit.
>
> Anyway, I'm fine with each way. Thanks.
>
> Regards,
> Radar
>
> On Thu, Jul 7, 2016 at 9:50 AM, Vineet Goel  wrote:
>
> > Radar,
> >
> > The whole point of cutting a branch was to stabilize it for the first
> > release. Why would we discard it and start a new one from master? Any
> > changes since the 2.0.0-incubating branch needs to go in the next release
> > scope (unless selective back-porting is necessary to meet the first
> release
> > scope).
> >
> > Thanks
> > -Vineet
> >
> >
> > On Wed, Jul 6, 2016 at 6:39 PM, Radar Da lei  wrote:
> >
> > > Since we have a lot of commits for this release after creating the
> > branch.
> > > I will re-create the branch. So the first blocking issue is gone.
> Nothing
> > > need to be merged.
> > >
> > > Any concerns, please let me know. Thanks.
> > >
> > > Regards,
> > > Radar
> > >
> > > On Thu, Jul 7, 2016 at 2:04 AM, Ting(Goden) Yao 
> wrote:
> > >
> > > > So we have 2 blocking issues:
> > > > https://issues.apache.org/jira/browse/HAWQ-867  (need to port this
> to
> > > > incubating branch)
> > > > https://issues.apache.org/jira/browse/HAWQ-892  (version naming
> issue)
> > > > Anyone can help to resolve them, please reply ASAP.
> > > > -Goden
> > > >
> > > > On Tue, Jul 5, 2016 at 5:59 PM Ting(Goden) Yao 
> > wrote:
> > > >
> > > > > We found an issue during discussion:
> > > > > https://issues.apache.org/jira/browse/HAWQ-892
> > > > > We need to contain "incubating" in the version and hawq --version
> > > > command.
> > > > >
> > > > > Thanks
> > > > > -Goden
> > > > >
> > > > > On Tue, Jul 5, 2016 at 10:24 AM Ting(Goden) Yao 
> > > wrote:
> > > > >
> > > > >> @Roman - for this JIRA, per Lei's suggestion, it's not blocking
> this
> > > > >> release (as no IP issues) , so I'll update the release info in the
> > > JIRA.
> > > > >> HAWQ-783  Remove
> > > > quicklz
> > > > >> in medadata
> > > > >>
> > > > >> On Tue, Jul 5, 2016 at 10:22 AM Ting(Goden) Yao 
> > > > wrote:
> > > > >>
> > > > >>> Thanks Guo, for your info. This JIRA was not marked in this
> > release.
> > > so
> > > > >>> next time, if you think this is required, please mark it for the
> > > > release
> > > > >>> and contact release manager ASAP.
> > > > >>> I'm checking this JIRA for details but I see it is closed
> already.
> > > > >>>
> > > > >>> On Tue, Jul 5, 2016 at 10:20 AM Roman Shaposhnik <
> > > ro...@shaposhnik.org
> > > > >
> > > > >>> wrote:
> > > > >>>
> > > >  On Tue, Jul 5, 2016 at 2:25 AM, Guo Gang 
> > wrote:
> > > >  > Goden, It seems that we need all of the features could be
> > compiled
> > > > in
> > > >  > source tarball (i.e. non-git workspace) before the release,
> but
> > > this
> > > >  seems
> > > >  > to be not the case in the tarball. See JIRA HAWQ-867 (Replace
> > the
> > > >  > git-submobule mechanism with git-clone). I think we need a new
> > > > source
> > > >  > version.
> > > > 
> > > >  Guo, Goden, it would be great if you guys could help all of us
> > > follow
> > > >  this
> > > >  release train by filing blocking JIRAs for the 2.0.0-incubating
> > > >  release. The
> > > >  way it currently stands, we seem to only have 1 blocking JIRA:
> > > > 
> > > > 
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/HAWQ/fixforversion/12334000/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
> > > > 
> > > >  Thanks,
> > > >  Roman.
> > > > 
> > > > >>>
> > > >
> > >
> >
>

[jira] [Created] (HAWQ-901) hawq init failed: hawqstandbywatch.py:test5:gpadmin-[WARNING]:-syncmaster not running

[Ming LI (JIRA)]

Ming LI created HAWQ-901:


 Summary: hawq init failed: 
hawqstandbywatch.py:test5:gpadmin-[WARNING]:-syncmaster not running
 Key: HAWQ-901
 URL: https://issues.apache.org/jira/browse/HAWQ-901
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Command Line Tools
Reporter: Ming LI
Assignee: Lei Chang


Error message in ~/hawqAdminLogs/hawq_init_.log

20160706:06:45:53:006218 hawq_start:test1:gpadmin-[INFO]:-Start hawq with args: 
['start', 'standby']
20160706:06:45:53:006218 hawq_start:test1:gpadmin-[INFO]:-Gathering information 
and validating the environment...
20160706:06:45:53:006218 hawq_start:test1:gpadmin-[INFO]:-Start standby master 
service
20160706:06:46:02:006218 hawq_start:test1:gpadmin-[INFO]:-Checking standby 
master status
20160706:06:45:55:004418 hawqstandbywatch.py:test5:gpadmin-[INFO]:-Monitoring 
logs
20160706:06:46:00:004418 hawqstandbywatch.py:test5:gpadmin-[INFO]:-checking if 
syncmaster is running
20160706:06:46:02:004418 
hawqstandbywatch.py:test5:gpadmin-[WARNING]:-syncmaster not running
20160706:06:46:02:006218 hawq_start:test1:gpadmin-[ERROR]:-Standby master start 
failed, exit
20160706:06:46:02:003999 hawqinit.sh:test5:gpadmin-[ERROR]:-Start HAWQ standby 
failed
--

(1) I suspect the root cause maybe: we only wait 5 seconds before we check 
standby running status, this interval is too small.  Could you please firstly 
change the standby running status check interval from 5 seconds to a loop like 
recovery running status check on master? 

(2) If the error 'syncmaster not running' will lead to init failure, we should 
change from [WARNING] to [ERROR]. 





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [Propose] More data skipping technology for IO intensive performance enhancement

[Lili Ma]

What about we work out a draft design describing how to implement data
skipping technology for HAWQ?


Thanks
Lili

On Wed, Jul 6, 2016 at 7:23 PM, Gmail  wrote:

> BTW, could you create some related issues in JIRA?
>
> Thanks
> xunzhang
>
> Send from my iPhone
>
> > 在 2016年7月2日，23:19，Ming Li  写道：
> >
> > Data skipping technology can extremely avoiding unnecessary IO,  so it
> can
> > extremely enhance performance for IO intensive query. Including
> eliminating
> > query on unnecessary table partition according to the partition key
> range ,
> > I think more options are available now:
> >
> > (1) Parquet / ORC format introduce a lightweight meta data info like
> > Min/Max/Bloom filter for each block, such meta data can be exploited when
> > predicate/filter info can be fetched before executing scan.
> >
> > However now in HAWQ, all data in parquet need to be scanned into memory
> > before processing predicate/filter. We don't generate the meta info when
> > INSERT into parquet table, the scan executor doesn't utilize the meta
> info
> > neither. Maybe some scan API need to be refactored so that we can get
> > predicate/filter
> > info before executing base relation scan.
> >
> > (2) Base on (1) technology,  especially with Bloom filter, more optimizer
> > technology can be explored furthur. E.g. Impala implemented Runtime
> > filtering(*
> https://www.cloudera.com/documentation/enterprise/latest/topics/impala_runtime_filtering.html
> > <
> https://www.cloudera.com/documentation/enterprise/latest/topics/impala_runtime_filtering.html
> >*
> > ),  which can be used at
> > - dynamic partition pruning
> > - converting join predicate to base relation predicate
> >
> > It tell the executor to wait for one moment(the interval time can be set
> in
> > guc) before executing base relation scan, if the interested values(e.g.
> the
> > column in join predicate only have very small set) arrived in time, it
> can
> > use these value to filter this scan, if doesn't arrived in time, it scan
> > without this filter, which doesn't impact result correctness.
> >
> > Unlike (1) technology, this technology cannot be used in any case, it
> only
> > outperform in some cases. So it just add some more query plan
> > choices/paths, and the optimizer need based on statistics info to
> calculate
> > the cost, and apply it when cost down.
> >
> > All in one, maybe more similar technology can be adoptable for HAWQ now,
> > let's start to think about performance related technology, moreover we
> need
> > to instigate how these technology can be implemented in HAWQ.
> >
> > Any ideas or suggestions are welcomed? Thanks.
>

Re: [VOTE] HAWQ 2.0.0-incubating Release

[Radar Da lei]

Hi Vineet,

Merge or re-create the branch is almost the same for our current status.

Re-create the branch can make all our good changes in, and it's more clear,
users can easy find out where it cut out, just following one version change
commit.

Anyway, I'm fine with each way. Thanks.

Regards,
Radar

On Thu, Jul 7, 2016 at 9:50 AM, Vineet Goel  wrote:

> Radar,
>
> The whole point of cutting a branch was to stabilize it for the first
> release. Why would we discard it and start a new one from master? Any
> changes since the 2.0.0-incubating branch needs to go in the next release
> scope (unless selective back-porting is necessary to meet the first release
> scope).
>
> Thanks
> -Vineet
>
>
> On Wed, Jul 6, 2016 at 6:39 PM, Radar Da lei  wrote:
>
> > Since we have a lot of commits for this release after creating the
> branch.
> > I will re-create the branch. So the first blocking issue is gone. Nothing
> > need to be merged.
> >
> > Any concerns, please let me know. Thanks.
> >
> > Regards,
> > Radar
> >
> > On Thu, Jul 7, 2016 at 2:04 AM, Ting(Goden) Yao  wrote:
> >
> > > So we have 2 blocking issues:
> > > https://issues.apache.org/jira/browse/HAWQ-867  (need to port this to
> > > incubating branch)
> > > https://issues.apache.org/jira/browse/HAWQ-892  (version naming issue)
> > > Anyone can help to resolve them, please reply ASAP.
> > > -Goden
> > >
> > > On Tue, Jul 5, 2016 at 5:59 PM Ting(Goden) Yao 
> wrote:
> > >
> > > > We found an issue during discussion:
> > > > https://issues.apache.org/jira/browse/HAWQ-892
> > > > We need to contain "incubating" in the version and hawq --version
> > > command.
> > > >
> > > > Thanks
> > > > -Goden
> > > >
> > > > On Tue, Jul 5, 2016 at 10:24 AM Ting(Goden) Yao 
> > wrote:
> > > >
> > > >> @Roman - for this JIRA, per Lei's suggestion, it's not blocking this
> > > >> release (as no IP issues) , so I'll update the release info in the
> > JIRA.
> > > >> HAWQ-783  Remove
> > > quicklz
> > > >> in medadata
> > > >>
> > > >> On Tue, Jul 5, 2016 at 10:22 AM Ting(Goden) Yao 
> > > wrote:
> > > >>
> > > >>> Thanks Guo, for your info. This JIRA was not marked in this
> release.
> > so
> > > >>> next time, if you think this is required, please mark it for the
> > > release
> > > >>> and contact release manager ASAP.
> > > >>> I'm checking this JIRA for details but I see it is closed already.
> > > >>>
> > > >>> On Tue, Jul 5, 2016 at 10:20 AM Roman Shaposhnik <
> > ro...@shaposhnik.org
> > > >
> > > >>> wrote:
> > > >>>
> > >  On Tue, Jul 5, 2016 at 2:25 AM, Guo Gang 
> wrote:
> > >  > Goden, It seems that we need all of the features could be
> compiled
> > > in
> > >  > source tarball (i.e. non-git workspace) before the release, but
> > this
> > >  seems
> > >  > to be not the case in the tarball. See JIRA HAWQ-867 (Replace
> the
> > >  > git-submobule mechanism with git-clone). I think we need a new
> > > source
> > >  > version.
> > > 
> > >  Guo, Goden, it would be great if you guys could help all of us
> > follow
> > >  this
> > >  release train by filing blocking JIRAs for the 2.0.0-incubating
> > >  release. The
> > >  way it currently stands, we seem to only have 1 blocking JIRA:
> > > 
> > > 
> > >
> >
> https://issues.apache.org/jira/browse/HAWQ/fixforversion/12334000/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
> > > 
> > >  Thanks,
> > >  Roman.
> > > 
> > > >>>
> > >
> >
>

Re: [VOTE] HAWQ 2.0.0-incubating Release

[Vineet Goel]

Radar,

The whole point of cutting a branch was to stabilize it for the first
release. Why would we discard it and start a new one from master? Any
changes since the 2.0.0-incubating branch needs to go in the next release
scope (unless selective back-porting is necessary to meet the first release
scope).

Thanks
-Vineet


On Wed, Jul 6, 2016 at 6:39 PM, Radar Da lei  wrote:

> Since we have a lot of commits for this release after creating the branch.
> I will re-create the branch. So the first blocking issue is gone. Nothing
> need to be merged.
>
> Any concerns, please let me know. Thanks.
>
> Regards,
> Radar
>
> On Thu, Jul 7, 2016 at 2:04 AM, Ting(Goden) Yao  wrote:
>
> > So we have 2 blocking issues:
> > https://issues.apache.org/jira/browse/HAWQ-867  (need to port this to
> > incubating branch)
> > https://issues.apache.org/jira/browse/HAWQ-892  (version naming issue)
> > Anyone can help to resolve them, please reply ASAP.
> > -Goden
> >
> > On Tue, Jul 5, 2016 at 5:59 PM Ting(Goden) Yao  wrote:
> >
> > > We found an issue during discussion:
> > > https://issues.apache.org/jira/browse/HAWQ-892
> > > We need to contain "incubating" in the version and hawq --version
> > command.
> > >
> > > Thanks
> > > -Goden
> > >
> > > On Tue, Jul 5, 2016 at 10:24 AM Ting(Goden) Yao 
> wrote:
> > >
> > >> @Roman - for this JIRA, per Lei's suggestion, it's not blocking this
> > >> release (as no IP issues) , so I'll update the release info in the
> JIRA.
> > >> HAWQ-783  Remove
> > quicklz
> > >> in medadata
> > >>
> > >> On Tue, Jul 5, 2016 at 10:22 AM Ting(Goden) Yao 
> > wrote:
> > >>
> > >>> Thanks Guo, for your info. This JIRA was not marked in this release.
> so
> > >>> next time, if you think this is required, please mark it for the
> > release
> > >>> and contact release manager ASAP.
> > >>> I'm checking this JIRA for details but I see it is closed already.
> > >>>
> > >>> On Tue, Jul 5, 2016 at 10:20 AM Roman Shaposhnik <
> ro...@shaposhnik.org
> > >
> > >>> wrote:
> > >>>
> >  On Tue, Jul 5, 2016 at 2:25 AM, Guo Gang  wrote:
> >  > Goden, It seems that we need all of the features could be compiled
> > in
> >  > source tarball (i.e. non-git workspace) before the release, but
> this
> >  seems
> >  > to be not the case in the tarball. See JIRA HAWQ-867 (Replace the
> >  > git-submobule mechanism with git-clone). I think we need a new
> > source
> >  > version.
> > 
> >  Guo, Goden, it would be great if you guys could help all of us
> follow
> >  this
> >  release train by filing blocking JIRAs for the 2.0.0-incubating
> >  release. The
> >  way it currently stands, we seem to only have 1 blocking JIRA:
> > 
> > 
> >
> https://issues.apache.org/jira/browse/HAWQ/fixforversion/12334000/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
> > 
> >  Thanks,
> >  Roman.
> > 
> > >>>
> >
>

Re: [VOTE] HAWQ 2.0.0-incubating Release

[Radar Da lei]

Since we have a lot of commits for this release after creating the branch.
I will re-create the branch. So the first blocking issue is gone. Nothing
need to be merged.

Any concerns, please let me know. Thanks.

Regards,
Radar

On Thu, Jul 7, 2016 at 2:04 AM, Ting(Goden) Yao  wrote:

> So we have 2 blocking issues:
> https://issues.apache.org/jira/browse/HAWQ-867  (need to port this to
> incubating branch)
> https://issues.apache.org/jira/browse/HAWQ-892  (version naming issue)
> Anyone can help to resolve them, please reply ASAP.
> -Goden
>
> On Tue, Jul 5, 2016 at 5:59 PM Ting(Goden) Yao  wrote:
>
> > We found an issue during discussion:
> > https://issues.apache.org/jira/browse/HAWQ-892
> > We need to contain "incubating" in the version and hawq --version
> command.
> >
> > Thanks
> > -Goden
> >
> > On Tue, Jul 5, 2016 at 10:24 AM Ting(Goden) Yao  wrote:
> >
> >> @Roman - for this JIRA, per Lei's suggestion, it's not blocking this
> >> release (as no IP issues) , so I'll update the release info in the JIRA.
> >> HAWQ-783  Remove
> quicklz
> >> in medadata
> >>
> >> On Tue, Jul 5, 2016 at 10:22 AM Ting(Goden) Yao 
> wrote:
> >>
> >>> Thanks Guo, for your info. This JIRA was not marked in this release. so
> >>> next time, if you think this is required, please mark it for the
> release
> >>> and contact release manager ASAP.
> >>> I'm checking this JIRA for details but I see it is closed already.
> >>>
> >>> On Tue, Jul 5, 2016 at 10:20 AM Roman Shaposhnik  >
> >>> wrote:
> >>>
>  On Tue, Jul 5, 2016 at 2:25 AM, Guo Gang  wrote:
>  > Goden, It seems that we need all of the features could be compiled
> in
>  > source tarball (i.e. non-git workspace) before the release, but this
>  seems
>  > to be not the case in the tarball. See JIRA HAWQ-867 (Replace the
>  > git-submobule mechanism with git-clone). I think we need a new
> source
>  > version.
> 
>  Guo, Goden, it would be great if you guys could help all of us follow
>  this
>  release train by filing blocking JIRAs for the 2.0.0-incubating
>  release. The
>  way it currently stands, we seem to only have 1 blocking JIRA:
> 
> 
> https://issues.apache.org/jira/browse/HAWQ/fixforversion/12334000/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
> 
>  Thanks,
>  Roman.
> 
> >>>
>

Re: Confusion around HAWQ versions in JIRA

[Lei Chang]

Agree. we should move quickly, do not let this block this release anymore.

Cheers
Lei


On Thu, Jul 7, 2016 at 8:40 AM, Vineet Goel  wrote:

> Hi all,
>
> If there are no major objections in the next day or so, then we should move
> to finalize the decision to continue using the 4-digit versioning. In that
> case, we'll simply need to update the JIRA versions to reflect
> "2.0.0.0-incubating"
> and so on going forward.
>
> While semantic versioning is nice to have, 4-digit versioning isn't too far
> off. User experience should not be that different between the two
> conventions.
>
> If you feel there is any downside with continuing the existing 4-digit
> versioning, please chime in as well.
>
> Thanks
> -Vineet
>
>
> On Wed, Jul 6, 2016 at 1:48 PM, Ting(Goden) Yao  wrote:
>
> > I still remember we had a huge discussion last year right after open
> > sourcing HAWQ and decided to switch from 4 digits to 3 digits semantic
> > versioning.
> > I can dig into email box to pull out the details. Can we check if all the
> > points from then are still valid or not?
> >
> >
> > On Wed, Jul 6, 2016 at 1:46 AM Xiang Sheng  wrote:
> >
> > > It is indeed better to keep the output of SQL 'select version();' with
> > the
> > > hawq version.
> > >
> > > Since the suffix '-incubating' indicates the project property, we
> should
> > > keep it before the project promoted to apache standard.
> > > So I think the output of the 'select version();' should add the suffix.
> > And
> > > obviously 4 digit number with suffix '-incubating' is more reasonable.
> > >
> > > On Wed, Jul 6, 2016 at 4:27 PM, Ming Li  wrote:
> > >
> > > > As for the version with postfix '-incubating', if after our project
> be
> > > > promoted to apache standard project, the version number still grows,
> so
> > > > from the perspective of service, it seems the postfix is useless.
> > > >
> > > > On the other hand, the output of SQL 'SELECT version();' doesn't
> > include
> > > > any
> > > >  '-incubating',  it is better to keep it same with the hawq version.
> > > >
> > > > So I prefer to use just 4 digit number without any postfix
> > '-incubating'.
> > > >
> > > >
> > > > On Wed, Jul 6, 2016 at 4:09 PM, Hong Wu 
> wrote:
> > > >
> > > > > For simplify, I prefer using the 4-digit mode.
> > > > >
> > > > > Best
> > > > > xunzhang
> > > > >
> > > > > 2016-07-06 16:07 GMT+08:00 Jiali Yao :
> > > > >
> > > > > > +1 for consolidating  the version.
> > > > > >
> > > > > > For 4-digit number, from the concept described above, I think 4
> > digit
> > > > > make
> > > > > > more sense. And from it, user can easily know whether specific
> > > upgrade
> > > > > > process needed or just binary switch if fine.
> > > > > > Based on that, for the "2.0.0", "2.0.0-incubating" or
> > > > > "2.0.0.0-incubating".
> > > > > > I prefer to 2.0.0.0-incubating since it would be consistent in
> JIRA
> > > and
> > > > > > code.
> > > > > >
> > > > > > Thanks
> > > > > > Jiali
> > > > > >
> > > > > >
> > > > > > On Wed, Jul 6, 2016 at 3:56 PM, Lei Chang 
> > > > wrote:
> > > > > >
> > > > > > > On Wed, Jul 6, 2016 at 3:17 PM, Vineet Goel <
> vvin...@apache.org>
> > > > > wrote:
> > > > > > >
> > > > > > > > Apologies for any confusion. Let me expand further:
> > > > > > > >
> > > > > > > > 1) My proposal was to update the JIRA versions. I didn't
> think
> > > > > > > > 2.0.0-incubating and 2.0.0 are the same, we should either
> > > > consolidate
> > > > > > > them
> > > > > > > > as one, or change the JIRA version numbers to be numerically
> > > > > different.
> > > > > > > > Version 2.0.0 shows 5 open JIRAs that may or may not belong
> to
> > > > > > > > "2.0.0-incubating" release. See link:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/HAWQ/fixforversion/12334195/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
> > > > > > > > vs
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/HAWQ/fixforversion/12334000/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
> > > > > > > >
> > > > > > > > We should update the 5 JIRAs listed in 2.0.0 with the correct
> > > > status
> > > > > > and
> > > > > > > > fix versions. This will make it easy to track the upcoming
> > > release.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > Agree. What I meant is also to consolidate the two into
> > > > > > "2.0.0-incubating"
> > > > > > > or "2.0.0.0-incubating" depending on which version schema we
> will
> > > > > choose.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > 2) Regarding the 4-digit versioning in the code, that's a
> good
> > > > > > discussion
> > > > > > > > to have.
> > > > > > > > What is the proposed convention for managing the 4 

Re: Confusion around HAWQ versions in JIRA

[Lei Chang]

I remember the discussion happened in pivotal internally long long time
ago, cannot remember it is before or after oss.

I think I can summarize the pros and cons from all the discussions:

1. current hawq 4-digit versioning

Make upgrade process quite clear and keep all the compatibility for
packages, libraries and external tools. Did not see issues from past
experience.

2. 3-digit semantic versioning

Show API changes from the number, this is good. But the APIs of a database
is quite complex, including SQL, udfs, jdbc, odbc, data formats, catalog,
tools It will make number changes somewhat unclear and very frequent.
And it is quite difficult to trace upgrade efforts. And it also has
compatibility issues.

And from discussion from this thread, looks most guys like option 1.

Cheers
Lei


On Thu, Jul 7, 2016 at 4:48 AM, Ting(Goden) Yao  wrote:

> I still remember we had a huge discussion last year right after open
> sourcing HAWQ and decided to switch from 4 digits to 3 digits semantic
> versioning.
> I can dig into email box to pull out the details. Can we check if all the
> points from then are still valid or not?
>
>
> On Wed, Jul 6, 2016 at 1:46 AM Xiang Sheng  wrote:
>
> > It is indeed better to keep the output of SQL 'select version();' with
> the
> > hawq version.
> >
> > Since the suffix '-incubating' indicates the project property, we should
> > keep it before the project promoted to apache standard.
> > So I think the output of the 'select version();' should add the suffix.
> And
> > obviously 4 digit number with suffix '-incubating' is more reasonable.
> >
> > On Wed, Jul 6, 2016 at 4:27 PM, Ming Li  wrote:
> >
> > > As for the version with postfix '-incubating', if after our project be
> > > promoted to apache standard project, the version number still grows, so
> > > from the perspective of service, it seems the postfix is useless.
> > >
> > > On the other hand, the output of SQL 'SELECT version();' doesn't
> include
> > > any
> > >  '-incubating',  it is better to keep it same with the hawq version.
> > >
> > > So I prefer to use just 4 digit number without any postfix
> '-incubating'.
> > >
> > >
> > > On Wed, Jul 6, 2016 at 4:09 PM, Hong Wu  wrote:
> > >
> > > > For simplify, I prefer using the 4-digit mode.
> > > >
> > > > Best
> > > > xunzhang
> > > >
> > > > 2016-07-06 16:07 GMT+08:00 Jiali Yao :
> > > >
> > > > > +1 for consolidating  the version.
> > > > >
> > > > > For 4-digit number, from the concept described above, I think 4
> digit
> > > > make
> > > > > more sense. And from it, user can easily know whether specific
> > upgrade
> > > > > process needed or just binary switch if fine.
> > > > > Based on that, for the "2.0.0", "2.0.0-incubating" or
> > > > "2.0.0.0-incubating".
> > > > > I prefer to 2.0.0.0-incubating since it would be consistent in JIRA
> > and
> > > > > code.
> > > > >
> > > > > Thanks
> > > > > Jiali
> > > > >
> > > > >
> > > > > On Wed, Jul 6, 2016 at 3:56 PM, Lei Chang 
> > > wrote:
> > > > >
> > > > > > On Wed, Jul 6, 2016 at 3:17 PM, Vineet Goel 
> > > > wrote:
> > > > > >
> > > > > > > Apologies for any confusion. Let me expand further:
> > > > > > >
> > > > > > > 1) My proposal was to update the JIRA versions. I didn't think
> > > > > > > 2.0.0-incubating and 2.0.0 are the same, we should either
> > > consolidate
> > > > > > them
> > > > > > > as one, or change the JIRA version numbers to be numerically
> > > > different.
> > > > > > > Version 2.0.0 shows 5 open JIRAs that may or may not belong to
> > > > > > > "2.0.0-incubating" release. See link:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/HAWQ/fixforversion/12334195/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
> > > > > > > vs
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/HAWQ/fixforversion/12334000/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
> > > > > > >
> > > > > > > We should update the 5 JIRAs listed in 2.0.0 with the correct
> > > status
> > > > > and
> > > > > > > fix versions. This will make it easy to track the upcoming
> > release.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > Agree. What I meant is also to consolidate the two into
> > > > > "2.0.0-incubating"
> > > > > > or "2.0.0.0-incubating" depending on which version schema we will
> > > > choose.
> > > > > >
> > > > > >
> > > > > >
> > > > > > > 2) Regarding the 4-digit versioning in the code, that's a good
> > > > > discussion
> > > > > > > to have.
> > > > > > > What is the proposed convention for managing the 4 digits and
> > what
> > > > sort
> > > > > > of
> > > > > > > code/API changes trigger a change in specific digits ? It would
> > be
> > > > good
> > > > > > to
> > > > > > > discuss the details.
> > 

Re: Confusion around HAWQ versions in JIRA

[Vineet Goel]

Hi all,

If there are no major objections in the next day or so, then we should move
to finalize the decision to continue using the 4-digit versioning. In that
case, we'll simply need to update the JIRA versions to reflect
"2.0.0.0-incubating"
and so on going forward.

While semantic versioning is nice to have, 4-digit versioning isn't too far
off. User experience should not be that different between the two
conventions.

If you feel there is any downside with continuing the existing 4-digit
versioning, please chime in as well.

Thanks
-Vineet


On Wed, Jul 6, 2016 at 1:48 PM, Ting(Goden) Yao  wrote:

> I still remember we had a huge discussion last year right after open
> sourcing HAWQ and decided to switch from 4 digits to 3 digits semantic
> versioning.
> I can dig into email box to pull out the details. Can we check if all the
> points from then are still valid or not?
>
>
> On Wed, Jul 6, 2016 at 1:46 AM Xiang Sheng  wrote:
>
> > It is indeed better to keep the output of SQL 'select version();' with
> the
> > hawq version.
> >
> > Since the suffix '-incubating' indicates the project property, we should
> > keep it before the project promoted to apache standard.
> > So I think the output of the 'select version();' should add the suffix.
> And
> > obviously 4 digit number with suffix '-incubating' is more reasonable.
> >
> > On Wed, Jul 6, 2016 at 4:27 PM, Ming Li  wrote:
> >
> > > As for the version with postfix '-incubating', if after our project be
> > > promoted to apache standard project, the version number still grows, so
> > > from the perspective of service, it seems the postfix is useless.
> > >
> > > On the other hand, the output of SQL 'SELECT version();' doesn't
> include
> > > any
> > >  '-incubating',  it is better to keep it same with the hawq version.
> > >
> > > So I prefer to use just 4 digit number without any postfix
> '-incubating'.
> > >
> > >
> > > On Wed, Jul 6, 2016 at 4:09 PM, Hong Wu  wrote:
> > >
> > > > For simplify, I prefer using the 4-digit mode.
> > > >
> > > > Best
> > > > xunzhang
> > > >
> > > > 2016-07-06 16:07 GMT+08:00 Jiali Yao :
> > > >
> > > > > +1 for consolidating  the version.
> > > > >
> > > > > For 4-digit number, from the concept described above, I think 4
> digit
> > > > make
> > > > > more sense. And from it, user can easily know whether specific
> > upgrade
> > > > > process needed or just binary switch if fine.
> > > > > Based on that, for the "2.0.0", "2.0.0-incubating" or
> > > > "2.0.0.0-incubating".
> > > > > I prefer to 2.0.0.0-incubating since it would be consistent in JIRA
> > and
> > > > > code.
> > > > >
> > > > > Thanks
> > > > > Jiali
> > > > >
> > > > >
> > > > > On Wed, Jul 6, 2016 at 3:56 PM, Lei Chang 
> > > wrote:
> > > > >
> > > > > > On Wed, Jul 6, 2016 at 3:17 PM, Vineet Goel 
> > > > wrote:
> > > > > >
> > > > > > > Apologies for any confusion. Let me expand further:
> > > > > > >
> > > > > > > 1) My proposal was to update the JIRA versions. I didn't think
> > > > > > > 2.0.0-incubating and 2.0.0 are the same, we should either
> > > consolidate
> > > > > > them
> > > > > > > as one, or change the JIRA version numbers to be numerically
> > > > different.
> > > > > > > Version 2.0.0 shows 5 open JIRAs that may or may not belong to
> > > > > > > "2.0.0-incubating" release. See link:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/HAWQ/fixforversion/12334195/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
> > > > > > > vs
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/HAWQ/fixforversion/12334000/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
> > > > > > >
> > > > > > > We should update the 5 JIRAs listed in 2.0.0 with the correct
> > > status
> > > > > and
> > > > > > > fix versions. This will make it easy to track the upcoming
> > release.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > Agree. What I meant is also to consolidate the two into
> > > > > "2.0.0-incubating"
> > > > > > or "2.0.0.0-incubating" depending on which version schema we will
> > > > choose.
> > > > > >
> > > > > >
> > > > > >
> > > > > > > 2) Regarding the 4-digit versioning in the code, that's a good
> > > > > discussion
> > > > > > > to have.
> > > > > > > What is the proposed convention for managing the 4 digits and
> > what
> > > > sort
> > > > > > of
> > > > > > > code/API changes trigger a change in specific digits ? It would
> > be
> > > > good
> > > > > > to
> > > > > > > discuss the details.
> > > > > > >
> > > > > >
> > > > > >
> > > > > > The 4-digit x.y.z.w versioning is:
> > > > > >
> > > > > > x: means major release
> > > > > > y. means minor release
> > > > > > z. means bug fix release
> > > > > > w. used for hot fix release
> > > > > >

Re: Replace git submodule with git clone + file with commit number?

[Ting(Goden) Yao]

I might have missed this - what's the rationale behind changing binary
dependency to git submodule (or git clone in this proposal) (e.g. gporca)
in the first place?

On Wed, Jul 6, 2016 at 9:03 AM Roman Shaposhnik 
wrote:

> On Wed, Jul 6, 2016 at 4:35 AM, Gmail  wrote:
> > I think the method using git clone is fine.
> >
> > But I suggest we'd better keep the makefile readable. I mean we should
> not add too trivial shell logic inside makefile itself.
> > For example, write a shell or Python script for users. Users should run
> the script before building HAWQ if they need to install this libraries.
>
> Huge +1 to the above. That's what I was suggesting in my previous
> emails saying that
> you should allow users to make dependencies available and then just
> point at them.
>
> > I also wonder what's the typical solution for similar problems of other
> Apache projects.
>
> Typically solution is to make sure that your build logic is capable of
> picking up
> binary dependencies.
>
> > Do they use git sub module?
>
> Not really. Most ASF projects have a single repo for the project (docs
> and site source
> code being an exception) and their dependency management is done through
> binary
> dependencies.
>
> > How do they solve this in their release?
>
> When you say 'release' this gets me slightly worried in the context of
> Git submodules.
> See, releases of ASF projects must be self-contained to a degree. If
> you require source
> in the Git submodule for the release to be functional it should give
> you a pretty strong
> indication that the source probably belongs to your project. If what
> submodules does
> is more of a plugin -- just declare a binary dependency.
>
> Thanks,
> Roman.
>

Re: [VOTE] HAWQ 2.0.0-incubating Release

[Ting(Goden) Yao]

So we have 2 blocking issues:
https://issues.apache.org/jira/browse/HAWQ-867  (need to port this to
incubating branch)
https://issues.apache.org/jira/browse/HAWQ-892  (version naming issue)
Anyone can help to resolve them, please reply ASAP.
-Goden

On Tue, Jul 5, 2016 at 5:59 PM Ting(Goden) Yao  wrote:

> We found an issue during discussion:
> https://issues.apache.org/jira/browse/HAWQ-892
> We need to contain "incubating" in the version and hawq --version command.
>
> Thanks
> -Goden
>
> On Tue, Jul 5, 2016 at 10:24 AM Ting(Goden) Yao  wrote:
>
>> @Roman - for this JIRA, per Lei's suggestion, it's not blocking this
>> release (as no IP issues) , so I'll update the release info in the JIRA.
>> HAWQ-783  Remove quicklz
>> in medadata
>>
>> On Tue, Jul 5, 2016 at 10:22 AM Ting(Goden) Yao  wrote:
>>
>>> Thanks Guo, for your info. This JIRA was not marked in this release. so
>>> next time, if you think this is required, please mark it for the release
>>> and contact release manager ASAP.
>>> I'm checking this JIRA for details but I see it is closed already.
>>>
>>> On Tue, Jul 5, 2016 at 10:20 AM Roman Shaposhnik 
>>> wrote:
>>>
 On Tue, Jul 5, 2016 at 2:25 AM, Guo Gang  wrote:
 > Goden, It seems that we need all of the features could be compiled in
 > source tarball (i.e. non-git workspace) before the release, but this
 seems
 > to be not the case in the tarball. See JIRA HAWQ-867 (Replace the
 > git-submobule mechanism with git-clone). I think we need a new source
 > version.

 Guo, Goden, it would be great if you guys could help all of us follow
 this
 release train by filing blocking JIRAs for the 2.0.0-incubating
 release. The
 way it currently stands, we seem to only have 1 blocking JIRA:

 https://issues.apache.org/jira/browse/HAWQ/fixforversion/12334000/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel

 Thanks,
 Roman.

>>>

Re: Replace git submodule with git clone + file with commit number?

[Roman Shaposhnik]

On Wed, Jul 6, 2016 at 4:35 AM, Gmail  wrote:
> I think the method using git clone is fine.
>
> But I suggest we'd better keep the makefile readable. I mean we should not 
> add too trivial shell logic inside makefile itself.
> For example, write a shell or Python script for users. Users should run the 
> script before building HAWQ if they need to install this libraries.

Huge +1 to the above. That's what I was suggesting in my previous
emails saying that
you should allow users to make dependencies available and then just
point at them.

> I also wonder what's the typical solution for similar problems of other 
> Apache projects.

Typically solution is to make sure that your build logic is capable of
picking up
binary dependencies.

> Do they use git sub module?

Not really. Most ASF projects have a single repo for the project (docs
and site source
code being an exception) and their dependency management is done through binary
dependencies.

> How do they solve this in their release?

When you say 'release' this gets me slightly worried in the context of
Git submodules.
See, releases of ASF projects must be self-contained to a degree. If
you require source
in the Git submodule for the release to be functional it should give
you a pretty strong
indication that the source probably belongs to your project. If what
submodules does
is more of a plugin -- just declare a binary dependency.

Thanks,
Roman.

Podling Report Reminder - July 2016

[johndament]

Dear podling,

This email was sent by an automated system on behalf of the Apache
Incubator PMC. It is an initial reminder to give you plenty of time to
prepare your quarterly board report.

The board meeting is scheduled for Wed, 20 July 2016, 10:30 am PDT.
The report for your podling will form a part of the Incubator PMC
report. The Incubator PMC requires your report to be submitted 2 weeks
before the board meeting, to allow sufficient time for review and
submission (Wed, July 06).

Please submit your report with sufficient time to allow the Incubator
PMC, and subsequently board members to review and digest. Again, the
very latest you should submit your report is 2 weeks prior to the board
meeting.

Thanks,

The Apache Incubator PMC

Submitting your Report

--

Your report should contain the following:

*   Your project name
*   A brief description of your project, which assumes no knowledge of
the project or necessarily of its field
*   A list of the three most important issues to address in the move
towards graduation.
*   Any issues that the Incubator PMC or ASF Board might wish/need to be
aware of
*   How has the community developed since the last report
*   How has the project developed since the last report.

This should be appended to the Incubator Wiki page at:

http://wiki.apache.org/incubator/July2016

Note: This is manually populated. You may need to wait a little before
this page is created from a template.

Mentors
---

Mentors should review reports for their project(s) and sign them off on
the Incubator wiki page. Signing off reports shows that you are
following the project - projects that are not signed may raise alarms
for the Incubator PMC.

Incubator PMC

Re: [Propose] More data skipping technology for IO intensive performance enhancement

[Gmail]

BTW, could you create some related issues in JIRA? 

Thanks
xunzhang

Send from my iPhone

> 在 2016年7月2日，23:19，Ming Li  写道：
> 
> Data skipping technology can extremely avoiding unnecessary IO,  so it can
> extremely enhance performance for IO intensive query. Including eliminating
> query on unnecessary table partition according to the partition key range ,
> I think more options are available now:
> 
> (1) Parquet / ORC format introduce a lightweight meta data info like
> Min/Max/Bloom filter for each block, such meta data can be exploited when
> predicate/filter info can be fetched before executing scan.
> 
> However now in HAWQ, all data in parquet need to be scanned into memory
> before processing predicate/filter. We don't generate the meta info when
> INSERT into parquet table, the scan executor doesn't utilize the meta info
> neither. Maybe some scan API need to be refactored so that we can get
> predicate/filter
> info before executing base relation scan.
> 
> (2) Base on (1) technology,  especially with Bloom filter, more optimizer
> technology can be explored furthur. E.g. Impala implemented Runtime
> filtering(*https://www.cloudera.com/documentation/enterprise/latest/topics/impala_runtime_filtering.html
> *
> ),  which can be used at
> - dynamic partition pruning
> - converting join predicate to base relation predicate
> 
> It tell the executor to wait for one moment(the interval time can be set in
> guc) before executing base relation scan, if the interested values(e.g. the
> column in join predicate only have very small set) arrived in time, it can
> use these value to filter this scan, if doesn't arrived in time, it scan
> without this filter, which doesn't impact result correctness.
> 
> Unlike (1) technology, this technology cannot be used in any case, it only
> outperform in some cases. So it just add some more query plan
> choices/paths, and the optimizer need based on statistics info to calculate
> the cost, and apply it when cost down.
> 
> All in one, maybe more similar technology can be adoptable for HAWQ now,
> let's start to think about performance related technology, moreover we need
> to instigate how these technology can be implemented in HAWQ.
> 
> Any ideas or suggestions are welcomed? Thanks.

Re: [Propose] More data skipping technology for IO intensive performance enhancement

[Gmail]

Ming, thanks for the proposal. I really like it!!

Performance is the important reason why users choose HAWQ. I think we need to 
improve the implementation of parquet format 'insert' and 'scan' according to 
your idea. Meanwhile, for further formats such as ORC supporting, we need to 
use their index to speed up reading.

Looking forward to see the enhancement in new release of HAWQ soon.

Send from my iPhone



发自我的 iPhone
> 在 2016年7月2日，23:19，Ming Li  写道：
> 
> Data skipping technology can extremely avoiding unnecessary IO,  so it can
> extremely enhance performance for IO intensive query. Including eliminating
> query on unnecessary table partition according to the partition key range ,
> I think more options are available now:
> 
> (1) Parquet / ORC format introduce a lightweight meta data info like
> Min/Max/Bloom filter for each block, such meta data can be exploited when
> predicate/filter info can be fetched before executing scan.
> 
> However now in HAWQ, all data in parquet need to be scanned into memory
> before processing predicate/filter. We don't generate the meta info when
> INSERT into parquet table, the scan executor doesn't utilize the meta info
> neither. Maybe some scan API need to be refactored so that we can get
> predicate/filter
> info before executing base relation scan.
> 
> (2) Base on (1) technology,  especially with Bloom filter, more optimizer
> technology can be explored furthur. E.g. Impala implemented Runtime
> filtering(*https://www.cloudera.com/documentation/enterprise/latest/topics/impala_runtime_filtering.html
> *
> ),  which can be used at
> - dynamic partition pruning
> - converting join predicate to base relation predicate
> 
> It tell the executor to wait for one moment(the interval time can be set in
> guc) before executing base relation scan, if the interested values(e.g. the
> column in join predicate only have very small set) arrived in time, it can
> use these value to filter this scan, if doesn't arrived in time, it scan
> without this filter, which doesn't impact result correctness.
> 
> Unlike (1) technology, this technology cannot be used in any case, it only
> outperform in some cases. So it just add some more query plan
> choices/paths, and the optimizer need based on statistics info to calculate
> the cost, and apply it when cost down.
> 
> All in one, maybe more similar technology can be adoptable for HAWQ now,
> let's start to think about performance related technology, moreover we need
> to instigate how these technology can be implemented in HAWQ.
> 
> Any ideas or suggestions are welcomed? Thanks.

[jira] [Created] (HAWQ-900) Add dependency in PL/R rpm build spec file plr.spec

[Paul Guo (JIRA)]

Paul Guo created HAWQ-900:
-

 Summary: Add dependency in PL/R rpm build spec file plr.spec
 Key: HAWQ-900
 URL: https://issues.apache.org/jira/browse/HAWQ-900
 Project: Apache HAWQ
  Issue Type: Bug
Reporter: Paul Guo
Assignee: Lei Chang


Building of plr depends on R-devel, while using of plr depends on R. In theory 
they depends on hawq also but we do not seem to be mandatory to have a hawq rpm 
for hawq installation, so the dependencies could be R stuffs only.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Confusion around HAWQ versions in JIRA

[Xiang Sheng]

It is indeed better to keep the output of SQL 'select version();' with the
hawq version.

Since the suffix '-incubating' indicates the project property, we should
keep it before the project promoted to apache standard.
So I think the output of the 'select version();' should add the suffix. And
obviously 4 digit number with suffix '-incubating' is more reasonable.

On Wed, Jul 6, 2016 at 4:27 PM, Ming Li  wrote:

> As for the version with postfix '-incubating', if after our project be
> promoted to apache standard project, the version number still grows, so
> from the perspective of service, it seems the postfix is useless.
>
> On the other hand, the output of SQL 'SELECT version();' doesn't include
> any
>  '-incubating',  it is better to keep it same with the hawq version.
>
> So I prefer to use just 4 digit number without any postfix '-incubating'.
>
>
> On Wed, Jul 6, 2016 at 4:09 PM, Hong Wu  wrote:
>
> > For simplify, I prefer using the 4-digit mode.
> >
> > Best
> > xunzhang
> >
> > 2016-07-06 16:07 GMT+08:00 Jiali Yao :
> >
> > > +1 for consolidating  the version.
> > >
> > > For 4-digit number, from the concept described above, I think 4 digit
> > make
> > > more sense. And from it, user can easily know whether specific upgrade
> > > process needed or just binary switch if fine.
> > > Based on that, for the "2.0.0", "2.0.0-incubating" or
> > "2.0.0.0-incubating".
> > > I prefer to 2.0.0.0-incubating since it would be consistent in JIRA and
> > > code.
> > >
> > > Thanks
> > > Jiali
> > >
> > >
> > > On Wed, Jul 6, 2016 at 3:56 PM, Lei Chang 
> wrote:
> > >
> > > > On Wed, Jul 6, 2016 at 3:17 PM, Vineet Goel 
> > wrote:
> > > >
> > > > > Apologies for any confusion. Let me expand further:
> > > > >
> > > > > 1) My proposal was to update the JIRA versions. I didn't think
> > > > > 2.0.0-incubating and 2.0.0 are the same, we should either
> consolidate
> > > > them
> > > > > as one, or change the JIRA version numbers to be numerically
> > different.
> > > > > Version 2.0.0 shows 5 open JIRAs that may or may not belong to
> > > > > "2.0.0-incubating" release. See link:
> > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/HAWQ/fixforversion/12334195/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
> > > > > vs
> > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/HAWQ/fixforversion/12334000/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
> > > > >
> > > > > We should update the 5 JIRAs listed in 2.0.0 with the correct
> status
> > > and
> > > > > fix versions. This will make it easy to track the upcoming release.
> > > > >
> > > > >
> > > > >
> > > > Agree. What I meant is also to consolidate the two into
> > > "2.0.0-incubating"
> > > > or "2.0.0.0-incubating" depending on which version schema we will
> > choose.
> > > >
> > > >
> > > >
> > > > > 2) Regarding the 4-digit versioning in the code, that's a good
> > > discussion
> > > > > to have.
> > > > > What is the proposed convention for managing the 4 digits and what
> > sort
> > > > of
> > > > > code/API changes trigger a change in specific digits ? It would be
> > good
> > > > to
> > > > > discuss the details.
> > > > >
> > > >
> > > >
> > > > The 4-digit x.y.z.w versioning is:
> > > >
> > > > x: means major release
> > > > y. means minor release
> > > > z. means bug fix release
> > > > w. used for hot fix release
> > > >
> > > > Catalog and data format changes need x or y change. From the number
> > > > changes, end users know whether it needs a hawq upgrade. for this
> > scheme,
> > > > API changes are not reflected in the number. For 3-digit semantic
> > > > versioning, the rules to increase the number is quite different, the
> > > number
> > > > change does not reflect catalog changes or data format changes but it
> > > > reflects API changes.
> > > >
> > > >
> > > > >
> > > > > Thanks
> > > > > -Vineet
> > > > >
> > > > >
> > > > > On Tue, Jul 5, 2016 at 11:35 PM, Ruilong Huo 
> > wrote:
> > > > >
> > > > > > I would prefer the option 1 to keep the 4-digit versions. This
> > > > mechanism
> > > > > > address the compatible issues of library in a more proper manner.
> > > > > >
> > > > > > PS, here are some background of the hawq versioning policy which
> > > might
> > > > > > help:
> > > > > > Postgres based systems, including GPDB and HAWQ, have
> > > > > > the notion of "MODULE_MAGIC" which is intended for the
> > > > > > purpose of guaranteeing version compatibility.  In addition
> > > > > > to the "MAGIC NUMBER", defined as the Major.Minor version
> > > > > > , GPDB and HAWQ also have the notion of a "MAGIC
> > > > > > PRODUCT" which GPDB uses to differentiate itself from
> > > > > > Postgres and provide clear messages regarding "this
> > > > > > library was built against Postgres" this mechanism
> > > > > > could be easily 

Re: Confusion around HAWQ versions in JIRA

[Ming Li]

As for the version with postfix '-incubating', if after our project be
promoted to apache standard project, the version number still grows, so
from the perspective of service, it seems the postfix is useless.

On the other hand, the output of SQL 'SELECT version();' doesn't include any
 '-incubating',  it is better to keep it same with the hawq version.

So I prefer to use just 4 digit number without any postfix '-incubating'.


On Wed, Jul 6, 2016 at 4:09 PM, Hong Wu  wrote:

> For simplify, I prefer using the 4-digit mode.
>
> Best
> xunzhang
>
> 2016-07-06 16:07 GMT+08:00 Jiali Yao :
>
> > +1 for consolidating  the version.
> >
> > For 4-digit number, from the concept described above, I think 4 digit
> make
> > more sense. And from it, user can easily know whether specific upgrade
> > process needed or just binary switch if fine.
> > Based on that, for the "2.0.0", "2.0.0-incubating" or
> "2.0.0.0-incubating".
> > I prefer to 2.0.0.0-incubating since it would be consistent in JIRA and
> > code.
> >
> > Thanks
> > Jiali
> >
> >
> > On Wed, Jul 6, 2016 at 3:56 PM, Lei Chang  wrote:
> >
> > > On Wed, Jul 6, 2016 at 3:17 PM, Vineet Goel 
> wrote:
> > >
> > > > Apologies for any confusion. Let me expand further:
> > > >
> > > > 1) My proposal was to update the JIRA versions. I didn't think
> > > > 2.0.0-incubating and 2.0.0 are the same, we should either consolidate
> > > them
> > > > as one, or change the JIRA version numbers to be numerically
> different.
> > > > Version 2.0.0 shows 5 open JIRAs that may or may not belong to
> > > > "2.0.0-incubating" release. See link:
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/HAWQ/fixforversion/12334195/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
> > > > vs
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/HAWQ/fixforversion/12334000/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
> > > >
> > > > We should update the 5 JIRAs listed in 2.0.0 with the correct status
> > and
> > > > fix versions. This will make it easy to track the upcoming release.
> > > >
> > > >
> > > >
> > > Agree. What I meant is also to consolidate the two into
> > "2.0.0-incubating"
> > > or "2.0.0.0-incubating" depending on which version schema we will
> choose.
> > >
> > >
> > >
> > > > 2) Regarding the 4-digit versioning in the code, that's a good
> > discussion
> > > > to have.
> > > > What is the proposed convention for managing the 4 digits and what
> sort
> > > of
> > > > code/API changes trigger a change in specific digits ? It would be
> good
> > > to
> > > > discuss the details.
> > > >
> > >
> > >
> > > The 4-digit x.y.z.w versioning is:
> > >
> > > x: means major release
> > > y. means minor release
> > > z. means bug fix release
> > > w. used for hot fix release
> > >
> > > Catalog and data format changes need x or y change. From the number
> > > changes, end users know whether it needs a hawq upgrade. for this
> scheme,
> > > API changes are not reflected in the number. For 3-digit semantic
> > > versioning, the rules to increase the number is quite different, the
> > number
> > > change does not reflect catalog changes or data format changes but it
> > > reflects API changes.
> > >
> > >
> > > >
> > > > Thanks
> > > > -Vineet
> > > >
> > > >
> > > > On Tue, Jul 5, 2016 at 11:35 PM, Ruilong Huo 
> wrote:
> > > >
> > > > > I would prefer the option 1 to keep the 4-digit versions. This
> > > mechanism
> > > > > address the compatible issues of library in a more proper manner.
> > > > >
> > > > > PS, here are some background of the hawq versioning policy which
> > might
> > > > > help:
> > > > > Postgres based systems, including GPDB and HAWQ, have
> > > > > the notion of "MODULE_MAGIC" which is intended for the
> > > > > purpose of guaranteeing version compatibility.  In addition
> > > > > to the "MAGIC NUMBER", defined as the Major.Minor version
> > > > > , GPDB and HAWQ also have the notion of a "MAGIC
> > > > > PRODUCT" which GPDB uses to differentiate itself from
> > > > > Postgres and provide clear messages regarding "this
> > > > > library was built against Postgres" this mechanism
> > > > > could be easily employed to differentiate HAWQ and GPDB
> > > > > and allow basing the "MAGIC NUMBER" off of the HAWQ version
> > > > >  instead of the GPDB version as it does today.
> > > > >
> > > > > Best regards,
> > > > > Ruilong Huo
> > > > >
> > > > > On Wed, Jul 6, 2016 at 2:26 PM, Radar Da lei 
> > wrote:
> > > > >
> > > > > > For Lei's proposal, I would prefer option 1 for below reasons:
> > > > > >
> > > > > > 1. Save time we may spend to solve incompatible issues.
> > > > > > 2. It will be hard to maintain semantic version if we increase
> > major
> > > > > > version every time when we are changing catalog and interface. If
> > so,
> > > > > HAWQ
> > > > > > version will 

Re: Confusion around HAWQ versions in JIRA

[Jiali Yao]

+1 for consolidating  the version.

For 4-digit number, from the concept described above, I think 4 digit make
more sense. And from it, user can easily know whether specific upgrade
process needed or just binary switch if fine.
Based on that, for the "2.0.0", "2.0.0-incubating" or "2.0.0.0-incubating".
I prefer to 2.0.0.0-incubating since it would be consistent in JIRA and
code.

Thanks
Jiali


On Wed, Jul 6, 2016 at 3:56 PM, Lei Chang  wrote:

> On Wed, Jul 6, 2016 at 3:17 PM, Vineet Goel  wrote:
>
> > Apologies for any confusion. Let me expand further:
> >
> > 1) My proposal was to update the JIRA versions. I didn't think
> > 2.0.0-incubating and 2.0.0 are the same, we should either consolidate
> them
> > as one, or change the JIRA version numbers to be numerically different.
> > Version 2.0.0 shows 5 open JIRAs that may or may not belong to
> > "2.0.0-incubating" release. See link:
> >
> >
> https://issues.apache.org/jira/browse/HAWQ/fixforversion/12334195/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
> > vs
> >
> >
> https://issues.apache.org/jira/browse/HAWQ/fixforversion/12334000/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
> >
> > We should update the 5 JIRAs listed in 2.0.0 with the correct status and
> > fix versions. This will make it easy to track the upcoming release.
> >
> >
> >
> Agree. What I meant is also to consolidate the two into "2.0.0-incubating"
> or "2.0.0.0-incubating" depending on which version schema we will choose.
>
>
>
> > 2) Regarding the 4-digit versioning in the code, that's a good discussion
> > to have.
> > What is the proposed convention for managing the 4 digits and what sort
> of
> > code/API changes trigger a change in specific digits ? It would be good
> to
> > discuss the details.
> >
>
>
> The 4-digit x.y.z.w versioning is:
>
> x: means major release
> y. means minor release
> z. means bug fix release
> w. used for hot fix release
>
> Catalog and data format changes need x or y change. From the number
> changes, end users know whether it needs a hawq upgrade. for this scheme,
> API changes are not reflected in the number. For 3-digit semantic
> versioning, the rules to increase the number is quite different, the number
> change does not reflect catalog changes or data format changes but it
> reflects API changes.
>
>
> >
> > Thanks
> > -Vineet
> >
> >
> > On Tue, Jul 5, 2016 at 11:35 PM, Ruilong Huo  wrote:
> >
> > > I would prefer the option 1 to keep the 4-digit versions. This
> mechanism
> > > address the compatible issues of library in a more proper manner.
> > >
> > > PS, here are some background of the hawq versioning policy which might
> > > help:
> > > Postgres based systems, including GPDB and HAWQ, have
> > > the notion of "MODULE_MAGIC" which is intended for the
> > > purpose of guaranteeing version compatibility.  In addition
> > > to the "MAGIC NUMBER", defined as the Major.Minor version
> > > , GPDB and HAWQ also have the notion of a "MAGIC
> > > PRODUCT" which GPDB uses to differentiate itself from
> > > Postgres and provide clear messages regarding "this
> > > library was built against Postgres" this mechanism
> > > could be easily employed to differentiate HAWQ and GPDB
> > > and allow basing the "MAGIC NUMBER" off of the HAWQ version
> > >  instead of the GPDB version as it does today.
> > >
> > > Best regards,
> > > Ruilong Huo
> > >
> > > On Wed, Jul 6, 2016 at 2:26 PM, Radar Da lei  wrote:
> > >
> > > > For Lei's proposal, I would prefer option 1 for below reasons:
> > > >
> > > > 1. Save time we may spend to solve incompatible issues.
> > > > 2. It will be hard to maintain semantic version if we increase major
> > > > version every time when we are changing catalog and interface. If so,
> > > HAWQ
> > > > version will reach 10.0.0 very soon.
> > > >
> > > > Thanks.
> > > >
> > > > Regards,
> > > > Radar
> > > >
> > > > On Wed, Jul 6, 2016 at 1:58 PM, Lei Chang 
> > wrote:
> > > >
> > > > > This is indeed a confusing issue. I am even confused by what Vineet
> > > > > proposed.
> > > > >
> > > > > There are several versions currently used across the systems:
> > > > >
> > > > > 1) the 3-digit JIRA versions: currently it has 2.0.0-incubating and
> > > > 2.0.0.
> > > > > and i think they are the same, "2.0.0-incubating" is more formal
> for
> > > > > incubating project.
> > > > >
> > > > > 2) the 4-digit versions in the code which is inherited from
> postgres
> > > and
> > > > > will be shown in "select version()" command;  it is somewhat
> related
> > to
> > > > > library compatibility and it is also related to third party tools.
> > Some
> > > > > tools may read and parse versions, and changing from 4 digit to 3
> > digit
> > > > > might introduce some unknown incompatibility issues.
> > > > >
> > > > >
> > > > > So currently there are 2 options:
> > > > >
> > > > > 1. Keep 4-digit version 

Re: Confusion around HAWQ versions in JIRA

[Lei Chang]

On Wed, Jul 6, 2016 at 3:17 PM, Vineet Goel  wrote:

> Apologies for any confusion. Let me expand further:
>
> 1) My proposal was to update the JIRA versions. I didn't think
> 2.0.0-incubating and 2.0.0 are the same, we should either consolidate them
> as one, or change the JIRA version numbers to be numerically different.
> Version 2.0.0 shows 5 open JIRAs that may or may not belong to
> "2.0.0-incubating" release. See link:
>
> https://issues.apache.org/jira/browse/HAWQ/fixforversion/12334195/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
> vs
>
> https://issues.apache.org/jira/browse/HAWQ/fixforversion/12334000/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
>
> We should update the 5 JIRAs listed in 2.0.0 with the correct status and
> fix versions. This will make it easy to track the upcoming release.
>
>
>
Agree. What I meant is also to consolidate the two into "2.0.0-incubating"
or "2.0.0.0-incubating" depending on which version schema we will choose.



> 2) Regarding the 4-digit versioning in the code, that's a good discussion
> to have.
> What is the proposed convention for managing the 4 digits and what sort of
> code/API changes trigger a change in specific digits ? It would be good to
> discuss the details.
>


The 4-digit x.y.z.w versioning is:

x: means major release
y. means minor release
z. means bug fix release
w. used for hot fix release

Catalog and data format changes need x or y change. From the number
changes, end users know whether it needs a hawq upgrade. for this scheme,
API changes are not reflected in the number. For 3-digit semantic
versioning, the rules to increase the number is quite different, the number
change does not reflect catalog changes or data format changes but it
reflects API changes.


>
> Thanks
> -Vineet
>
>
> On Tue, Jul 5, 2016 at 11:35 PM, Ruilong Huo  wrote:
>
> > I would prefer the option 1 to keep the 4-digit versions. This mechanism
> > address the compatible issues of library in a more proper manner.
> >
> > PS, here are some background of the hawq versioning policy which might
> > help:
> > Postgres based systems, including GPDB and HAWQ, have
> > the notion of "MODULE_MAGIC" which is intended for the
> > purpose of guaranteeing version compatibility.  In addition
> > to the "MAGIC NUMBER", defined as the Major.Minor version
> > , GPDB and HAWQ also have the notion of a "MAGIC
> > PRODUCT" which GPDB uses to differentiate itself from
> > Postgres and provide clear messages regarding "this
> > library was built against Postgres" this mechanism
> > could be easily employed to differentiate HAWQ and GPDB
> > and allow basing the "MAGIC NUMBER" off of the HAWQ version
> >  instead of the GPDB version as it does today.
> >
> > Best regards,
> > Ruilong Huo
> >
> > On Wed, Jul 6, 2016 at 2:26 PM, Radar Da lei  wrote:
> >
> > > For Lei's proposal, I would prefer option 1 for below reasons:
> > >
> > > 1. Save time we may spend to solve incompatible issues.
> > > 2. It will be hard to maintain semantic version if we increase major
> > > version every time when we are changing catalog and interface. If so,
> > HAWQ
> > > version will reach 10.0.0 very soon.
> > >
> > > Thanks.
> > >
> > > Regards,
> > > Radar
> > >
> > > On Wed, Jul 6, 2016 at 1:58 PM, Lei Chang 
> wrote:
> > >
> > > > This is indeed a confusing issue. I am even confused by what Vineet
> > > > proposed.
> > > >
> > > > There are several versions currently used across the systems:
> > > >
> > > > 1) the 3-digit JIRA versions: currently it has 2.0.0-incubating and
> > > 2.0.0.
> > > > and i think they are the same, "2.0.0-incubating" is more formal for
> > > > incubating project.
> > > >
> > > > 2) the 4-digit versions in the code which is inherited from postgres
> > and
> > > > will be shown in "select version()" command;  it is somewhat related
> to
> > > > library compatibility and it is also related to third party tools.
> Some
> > > > tools may read and parse versions, and changing from 4 digit to 3
> digit
> > > > might introduce some unknown incompatibility issues.
> > > >
> > > >
> > > > So currently there are 2 options:
> > > >
> > > > 1. Keep 4-digit version scheme, changing everything to 4 digit
> > versions,
> > > > and release it.
> > > >
> > > > 2. Change everything to 3 digits and this might introduce some
> unknown
> > > > incompatibility issues.
> > > >
> > > > Thoughts?
> > > >
> > > > Cheers
> > > > Lei
> > > >
> > > >
> > > > On Wed, Jul 6, 2016 at 1:21 PM, Vineet Goel 
> > wrote:
> > > >
> > > > > 1) Proposal - we can rename the 2.0.0 version to 2.0.1-incubating
> as
> > > the
> > > > > next planned maintenance release (for now). All JIRAs targeted for
> > > 2.0.0
> > > > > should be evaluated to see if any belong to the scope for the
> > upcoming
> > > > > 2.0.0-incubating
> > > > > release or not.
> > > >
> > > >
> > > > > 2) 

[jira] [Created] (HAWQ-899) Add feature test for nested null case with new test framework

[Lin Wen (JIRA)]

Lin Wen created HAWQ-899:


 Summary: Add feature test for nested null case with new test 
framework
 Key: HAWQ-899
 URL: https://issues.apache.org/jira/browse/HAWQ-899
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: Tests
Reporter: Lin Wen
Assignee: Jiali Yao






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-898) Add feature test for COPY with new test framework

[Lin Wen (JIRA)]

Lin Wen created HAWQ-898:


 Summary: Add feature test for COPY with new test framework 
 Key: HAWQ-898
 URL: https://issues.apache.org/jira/browse/HAWQ-898
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: Tests
Reporter: Lin Wen
Assignee: Jiali Yao






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Confusion around HAWQ versions in JIRA

[Radar Da lei]

For Lei's proposal, I would prefer option 1 for below reasons:

1. Save time we may spend to solve incompatible issues.
2. It will be hard to maintain semantic version if we increase major
version every time when we are changing catalog and interface. If so, HAWQ
version will reach 10.0.0 very soon.

Thanks.

Regards,
Radar

On Wed, Jul 6, 2016 at 1:58 PM, Lei Chang  wrote:

> This is indeed a confusing issue. I am even confused by what Vineet
> proposed.
>
> There are several versions currently used across the systems:
>
> 1) the 3-digit JIRA versions: currently it has 2.0.0-incubating and 2.0.0.
> and i think they are the same, "2.0.0-incubating" is more formal for
> incubating project.
>
> 2) the 4-digit versions in the code which is inherited from postgres and
> will be shown in "select version()" command;  it is somewhat related to
> library compatibility and it is also related to third party tools. Some
> tools may read and parse versions, and changing from 4 digit to 3 digit
> might introduce some unknown incompatibility issues.
>
>
> So currently there are 2 options:
>
> 1. Keep 4-digit version scheme, changing everything to 4 digit versions,
> and release it.
>
> 2. Change everything to 3 digits and this might introduce some unknown
> incompatibility issues.
>
> Thoughts?
>
> Cheers
> Lei
>
>
> On Wed, Jul 6, 2016 at 1:21 PM, Vineet Goel  wrote:
>
> > 1) Proposal - we can rename the 2.0.0 version to 2.0.1-incubating as the
> > next planned maintenance release (for now). All JIRAs targeted for 2.0.0
> > should be evaluated to see if any belong to the scope for the upcoming
> > 2.0.0-incubating
> > release or not.
>
>
> > 2) Regarding comments on JIRA-875, I have created a new JIRA (HAWQ-895)
> for
> > the investigation on migrating to semantic versioning. That raises the
> > question, should version 2.0.0-incubating really be 2.0.0.0-incubating ?
> > https://issues.apache.org/jira/browse/HAWQ-895
> >
> > Thanks
> > -Vineet
> >
> >
> > On Tue, Jul 5, 2016 at 5:09 PM, Goden Yao  wrote:
> >
> > > Hi all,
> > >
> > > I want to raise some concerns around HAWQ versions we used in Apache
> > JIRA.
> > > We right now have:
> > >
> > >- 2.0.0-incubating (this is the upcoming release we're working on)
> > >- 2.0.0 (this was used for JIRAs after originally planned
> > >2.0.0-incubating) , now I see a little bit issue if we releae
> > >2.0.0-incubating , what leaves with items associated with this
> > version?
> > >- 2.1.0 - supposedly , this is the next minor release
> > >- 3.0.0 - supposedly, this is the next major release
> > >- Backlog
> > >
> > >
> > > Then I see this JIRA: https://issues.apache.org/jira/browse/HAWQ-875
> > > (*Upgrade
> > > HAWQ version to 2.0.1.0*), which is not a version listed on the release
> > > page.
> > > Can we:
> > >
> > >- Clarify which version is for which release (goals, purpose, etc.)
> > see
> > >example I did for 2.0.0-incubating:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/HAWQ/HAWQ+Release+2.0.0-incubating
> > >
> > >- When you file JIRA, make sure you have a targeted version for it
> so
> > >it's easy to track from release perspective.
> > >
> > >
> > > Thanks
> > > -Goden
> > >
> >
>

[jira] [Created] (HAWQ-897) Add feature test for create table distribution with new test framework

[Lin Wen (JIRA)]

Lin Wen created HAWQ-897:


 Summary: Add feature test for create table distribution with new 
test framework
 Key: HAWQ-897
 URL: https://issues.apache.org/jira/browse/HAWQ-897
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: Tests
Reporter: Lin Wen
Assignee: Jiali Yao






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-896) Add feature test for create table with new test framework

[Lin Wen (JIRA)]

Lin Wen created HAWQ-896:


 Summary: Add feature test for create table with new test framework
 Key: HAWQ-896
 URL: https://issues.apache.org/jira/browse/HAWQ-896
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: Tests
Reporter: Lin Wen
Assignee: Jiali Yao






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)