Re: [DISCUSS] Graduate Apache HAWQ (incubating) as a TLP

[Hubert Zhang]

t; Committee (PMC), to be known as the "Apache HAWQ Project",
> > be and hereby is established pursuant to Bylaws of the
> > Foundation; and be it further
> >
> > RESOLVED, that the Apache HAWQ Project be and hereby is
> > responsible for the creation and maintenance of software
> > related to Hadoop native SQL query engine that
> > combines the key technological advantages of MPP database
> > with the scalability and convenience of Hadoop;
> > and be it further
> >
> > RESOLVED, that the office of "Vice President, Apache HAWQ" be
> > and hereby is created, the person holding such office to
> > serve at the direction of the Board of Directors as the chair
> > of the Apache HAWQ Project, and to have primary responsibility
> > for management of the projects within the scope of
> > responsibility of the Apache HAWQ Project; and be it further
> >
> > RESOLVED, that the persons listed immediately below be and
> > hereby are appointed to serve as the initial members of the
> > Apache HAWQ Project:
> >
> >  * Alan Gates   
> >  * Alexander Denissov   
> >  * Amy Bai  
> >  * Atri Sharma  
> >  * Bhuvnesh Chaudhary   
> >  * Bosco
> >  * Chunling Wang
> >  * David Yozie  
> >  * Ed Espino
> >  * Entong Shen  
> >  * Foyzur Rahman
> >  * Goden Yao
> >  * Gregory Chase
> >  * Hong Wu  
> >  * Hongxu Ma
> >  * Hubert Zhang 
> >  * Ivan Weng
> >  * Jesse Zhang  
> >  * Jiali Yao
> >  * Jun Aoki 
> >  * Kavinder Dhaliwal
> >  * Lav Jain 
> >  * Lei Chang
> >  * Lili Ma  
> >  * Lirong Jian  
> >  * Lisa Owen
> >  * Ming Li  
> >  * Mohamed Soliman  
> >  * Newton Alex  
> >  * Noa Horn 
> >  * Oleksandr Diachenko  
> >  * Paul Guo 
> >  * Radar Da Lei 
> >  * Roman Shaposhnik 
> >  * Ruilong Huo  
> >  * Shivram Mani 
> >  * Shubham Sharma   
> >  * Tushar Pednekar  
> >  * Venkatesh Raghavan   
> >  * Vineet Goel  
> >  * Wen Lin  
> >  * Xiang Sheng  
> >  * Yi Jin   
> >  * Zhanwei Wang 
> >      * Zhenglin Tao 
> >
> > NOW, THEREFORE, BE IT FURTHER RESOLVED, that Lei Chang
> > be appointed to the office of Vice President, Apache HAWQ, to
> > serve in accordance with and subject to the direction of the
> > Board of Directors and the Bylaws of the Foundation until
> > death, resignation, retirement, removal or disqualification,
> > or until a successor is appointed; and be it further
> >
> > RESOLVED, that the initial Apache HAWQ PMC be and hereby is
> > tasked with the creation of a set of bylaws intended to
> > encourage open development and increased participation in the
> > Apache HAWQ Project; and be it further
> >
> > RESOLVED, that the Apache HAWQ Project be and hereby
> > is tasked with the migration and rationalization of the Apache
> > Incubator HAWQ podling; and be it further
> >
> > RESOLVED, that all responsibilities pertaining to the Apache
> > Incubator HAWQ podling encumbered upon the Apache Incubator
> > Project are hereafter discharged.
> >
>
> --
> Regards,
> Hongxu.
>
>


-- 
Thanks

Hubert Zhang

Re: Remain with HAWQ project or not?

[Hubert Zhang]

Yes.

On Tue, May 8, 2018 at 9:30 AM, Lili Ma <lil...@apache.org> wrote:

> Yes, of course I want to remain as PMC member!
>
> Thanks Radar for the effort on HAWQ graduation:)
>
> Best Regards,
> Lili
>
> 2018-05-07 20:07 GMT-04:00 Lisa Owen <lo...@pivotal.io>:
>
> > yes, i would like to remain a committer.
> >
> >
> > -lisa owen
> >
> > On Mon, May 7, 2018 at 10:02 AM, Shubham Sharma <ssha...@pivotal.io>
> > wrote:
> >
> > > Yes. I am looking forward to contributing to Hawq.
> > >
> > > On Mon, May 7, 2018 at 12:53 PM, Lav Jain <lj...@pivotal.io> wrote:
> > >
> > > > Yes. I am very excited about HAWQ.
> > > >
> > > > Regards,
> > > >
> > > >
> > > > *Lav Jain*
> > > > *Pivotal Data*
> > > >
> > > > lj...@pivotal.io
> > > >
> > > > On Mon, May 7, 2018 at 6:51 AM, Alexander Denissov <
> > adenis...@pivotal.io
> > > >
> > > > wrote:
> > > >
> > > > > Yes.
> > > > >
> > > > > > On May 7, 2018, at 6:03 AM, Wen Lin <w...@pivotal.io> wrote:
> > > > > >
> > > > > > Yes. I'd like to keep on contributing to HAWQ.
> > > > > >
> > > > > >> On Mon, May 7, 2018 at 5:21 PM, Ivan Weng <iw...@pivotal.io>
> > wrote:
> > > > > >>
> > > > > >> Yes, I definitely would like to be with HAWQ.
> > > > > >>
> > > > > >> Regards,
> > > > > >> Ivan
> > > > > >>
> > > > > >>> On Mon, May 7, 2018 at 5:12 PM, Hongxu Ma <inte...@outlook.com
> >
> > > > wrote:
> > > > > >>>
> > > > > >>> Yes, let's make HAWQ better.
> > > > > >>>
> > > > > >>> Thanks.
> > > > > >>>
> > > > > >>>> 在 07/05/2018 16:11, Radar Lei 写道:
> > > > > >>>> HAWQ committers,
> > > > > >>>>
> > > > > >>>> Per the discussion in "Apache HAWQ graduation from incubator?"
> > > [1],
> > > > we
> > > > > >>> want
> > > > > >>>> to setup the PMC as part of HAWQ graduation resolution.
> > > > > >>>>
> > > > > >>>> So we'd like to confirm whether you want to remain as a
> > > > committer/PMC
> > > > > >>>> member of Apache HAWQ project?
> > > > > >>>>
> > > > > >>>> If you'd like to remain with HAWQ project, it's welcome and
> > please
> > > > > >>> *respond**
> > > > > >>>> 'Yes'* in this thread, or *respond 'No'* if you are not
> > interested
> > > > in
> > > > > >> any
> > > > > >>>> more. Thanks.
> > > > > >>>>
> > > > > >>>> This thread will be available for at least 72 hours, after
> that,
> > > we
> > > > > >> will
> > > > > >>>> send individual confirm emails.
> > > > > >>>>
> > > > > >>>> [1]
> > > > > >>>> https://lists.apache.org/thread.html/
> > > b4a0b5671ce377b3d51c9b7ab00496
> > > > > >>> a1eebfcbf1696ce8b67e078c64@%3Cdev.hawq.apache.org%3E
> > > > > >>>>
> > > > > >>>> Regards,
> > > > > >>>> Radar
> > > > > >>>>
> > > > > >>>
> > > > > >>> --
> > > > > >>> Regards,
> > > > > >>> Hongxu.
> > > > > >>>
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Shubham Sharma
> > > Staff Customer Engineer
> > > Pivotal Global Support Services
> > > ssha...@pivotal.io
> > > Direct Tel: +1(510)-304-8201
> > > Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT
> > > Out of Office Hours Contact +1 877-477-2269
> > >
> >
>



-- 
Thanks

Hubert Zhang

Re: Re: [VOTE]: Apache HAWQ 2.3.0.0-incubating Release (RC2)

[Hubert Zhang]

+1
Build and Installed. Tests passed.

On Thu, Mar 1, 2018 at 2:01 PM, Ruilong Huo <h...@apache.org> wrote:

> +1 for the HAWQ 2.3.0.0-incubating RC2
>
>
> Here are the checks have been done:
>
>
> 1. Reviewed LICENSE, NOTICE, DISCLAIMER, and pom.xml.
>
>
> 2. Passed RAT configuration check successfully.
>
>
> 3. Passed source and rpm package signature, md5 and sha256 checksum.
>
>
> 4. Compiled from source tarball and installed RC2 with feature check
> successful.
>
>
> 5. Downloaded rpm tarball, installed hawq on CentOS 7.2 VM following
> https://cwiki.apache.org/confluence/display/HAWQ/Build+
> Package+and+Install+with+RPM. The initialization and basic query passed.
>
>
> Best regards,
> Ruilong Huo
> At 2018-03-01 13:50:58, "Bai Jie" <amybai6...@gmail.com> wrote:
> >Build from branch 2.3.0.0 source code, installed , init , run feature test
> >and simple queries. Looks good to me. +1
> >
> >On Wed, Feb 28, 2018 at 11:24 AM, Hongxu Ma <inte...@outlook.com> wrote:
> >
> >> +1
> >>
> >> Both source and rpm package are verified in my environment: Red Hat
> >> Enterprise Linux Server release 7.2
> >> Include installation and execute some simple queries.
> >>
> >> 在 27/02/2018 12:19, Yi JIN 写道:
> >> > Hi All,
> >> >
> >> > This is the vote for Apache HAWQ (incubating) 2.3.0.0-incubating
> Release
> >> > Candidate 2 (RC2). It is a source release for HAWQ core, PXF, and
> Ranger;
> >> > and binary release for HAWQ core,  PXF and Ranger. We have rpm package
> >> > involved for the binary release.
> >> >
> >> > The vote will run for at least 72 hours and will close on Saturday,
> March
> >> > 3rd, 2017. Thanks.
> >> >
> >> > 1. Wiki page of the release:
> >> > *https://cwiki.apache.org/confluence/display/HAWQ/
> Apache+HAWQ+2.3.0.0-
> >> incubating+Release
> >> > <https://cwiki.apache.org/confluence/display/HAWQ/
> Apache+HAWQ+2.3.0.0-
> >> incubating+Release>*
> >> >
> >> >
> >> > 2. Release Notes (Apache Jira generated):
> >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> >> > version=12340262=Html=12318826
> >> >
> >> >
> >> > 3. Release verification steps can be found at:
> >> > For source tarball: https://cwiki.apache.org/confluence/display/HAWQ/
> >> > Release+Process%3A+Step+by+step+guide#ReleaseProcess:
> >> > Stepbystepguide-ValidatetheReleaseCandidate
> >> > For rpm package: https://cwiki.apache.org/confluence/display/HAWQ/
> >> > Build+Package+and+Install+with+RPM
> >> >
> >> >
> >> > 4. Git release branch:
> >> > https://git-wip-us.apache.org/repos/asf?p=incubator-hawq.
> >> > git;a=shortlog;h=refs/heads/2.3.0.0-incubating
> >> >
> >> > 5. Source and Binary release balls with signare:
> >> > https://dist.apache.org/repos/dist/dev/incubator/hawq/2.3.0.
> >> > 0-incubating.RC2/
> >> >
> >> >
> >> > 6. Keys to verify the signature of the release artifact are available
> at:
> >> > https://dist.apache.org/repos/dist/dev/incubator/hawq/KEYS
> >> >
> >> >
> >> > 7. The artifact(s) has been signed with Key ID: CE60F90D1333092A
> >> >
> >> > 8. Fixed issues in RC2.
> >> > https://issues.apache.org/jira/browse/HAWQ-1589
> >> > https://issues.apache.org/jira/browse/HAWQ-1590
> >> >
> >> > REMINDER: Please provide details of what you have tried and verified
> >> before
> >> > your vote conclusion. Thanks!
> >> >
> >> >
> >> > Please vote accordingly:
> >> > [ ] +1 approve
> >> > [ ] +0 no opinion
> >> > [ ] -1 disapprove (and reason why)
> >> >
> >> >
> >> > Best regards,
> >> > Yi (yjin)
> >> >
> >>
> >> --
> >> Regards,
> >> Hongxu.
> >>
> >>
>



-- 
Thanks

Hubert Zhang

Re: [VOTE]: Apache HAWQ 2.3.0.0-incubating Release (RC1)

[Hubert Zhang]

+1

On Wed, Feb 21, 2018 at 6:29 PM, Hongxu Ma <inte...@outlook.com> wrote:

> +1
>
> Both source and rpm package are verified in my environment: Red Hat
> Enterprise Linux Server release 7.2
> Include installation and execute some simple queries.
>
> 在 20/02/2018 10:42, Yi JIN 写道:
> > Hi All,
> >
> > This is the vote for Apache HAWQ (incubating) 2.3.0.0-incubating Release
> > Candidate 1 (RC1). It is a source release for HAWQ core, PXF, and Ranger;
> > and binary release for HAWQ core,  PXF and Ranger. We have rpm package
> > involved for the binary release.
> >
> > The vote will run for at least 72 hours and will close on Saturday, Feb
> 24,
> > 2017. Thanks.
> >
> > 1. Wiki page of the release:
> > *https://cwiki.apache.org/confluence/display/HAWQ/Apache+HAWQ+2.3.0.0-
> incubating+Release
> > <https://cwiki.apache.org/confluence/display/HAWQ/Apache+HAWQ+2.3.0.0-
> incubating+Release>*
> >
> >
> > 2. Release Notes (Apache Jira generated):
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> version=12340262=Html=12318826
> >
> >
> > 3. Release verification steps can be found at:
> > For source tarball: https://cwiki.apache.org/confluence/display/HAWQ
> > /Release+Process%3A+Step+by+step+guide#ReleaseProcess:Stepbystepguide-
> > ValidatetheReleaseCandidate
> > For rpm package: https://cwiki.apache.org/confluence/display/HAWQ
> > /Build+Package+and+Install+with+RPM
> >
> >
> > 4. Git release branch:
> > https://git-wip-us.apache.org/repos/asf?p=incubator-hawq.
> git;a=shortlog;h=refs/heads/2.3.0.0-incubating
> >
> > 5. Source and Binary release balls with signare:
> > https://dist.apache.org/repos/dist/dev/incubator/hawq/2.3.0.
> 0-incubating.RC1/
> >
> >
> > 6. Keys to verify the signature of the release artifact are available at:
> > https://dist.apache.org/repos/dist/dev/incubator/hawq/KEYS
> >
> >
> > 7. The artifact(s) has been signed with Key ID: CE60F90D1333092A
> >
> >
> > Please vote accordingly:
> > [ ] +1 approve
> > [ ] +0 no opinion
> > [ ] -1 disapprove (and reason why)
> >
> >
> > Best regards,
> > Yi (yjin)
> >
>
> --
> Regards,
> Hongxu.
>
>


-- 
Thanks

Hubert Zhang

Re: New Committer: Amy Bai

[Hubert Zhang]

Congrats to Amy.
Your contribution on TDE is impressive!

On Wed, Nov 1, 2017 at 2:02 PM, Wen Lin <w...@pivotal.io> wrote:

> Hi,
>
> The Project Management Committee (PMC) for Apache HAWQ (incubating) has
> invited Amy Bai to become a committer and we are pleased to announce that
> she has accepted.
> Being a committer enables easier contribution to the project since there is
> no need to go via the patch submission process. This should enable better
> productivity. Please join us in congratulating her and we are looking
> forward to collaborating with her in the open source community. Her
> contribution includes (but not limited to):
> List contributions to code base, documentation, code review, discussion in
> mailing list, JIRA, etc.
>
> Regards!
>



-- 
Thanks

Hubert Zhang

Re: New committer: Chunling Wang

[Hubert Zhang]

. Add
>   getting configuration from HDFS and YARN
>   - HAWQ-969 <https://issues.apache.org/jira/browse/HAWQ-969>. fix
> bugs
>   in last commit
>   - HAWQ-969 <https://issues.apache.org/jira/browse/HAWQ-969>. Modify
>   some functions in hdfs_config.cpp and yarn_config.cpp
>   - HAWQ-991 <https://issues.apache.org/jira/browse/HAWQ-991>. Refator
>   HAWQ Register code for partition table.
>   - HAWQ-1034 <https://issues.apache.org/jira/browse/HAWQ-1034>.
> Delete
>   code for hawq register repair mode
>   - HAWQ-1044 <https://issues.apache.org/jira/browse/HAWQ-1044>. Add
>   normal path testcase for hawq register when file exists, file does
>   not exist and --force
>   - HAWQ-1044 <https://issues.apache.org/jira/browse/HAWQ-1044>. Add
>   normal path testcase for hawq register --repair
>   - HAWQ-1044 <https://issues.apache.org/jira/browse/HAWQ-1044>.
> Remove
>   duplicated testcases for hawq resgister usage2
>   - HAWQ-1044 <https://issues.apache.org/jira/browse/HAWQ-1044>. Add
>   some error path test cases for hawq register (TestUsage2Case1Error
>   Encoding, TestUsage2Case1Bucket0, TestUsage2Case1IncludeDirectory)
>   - HAWQ-1044 <https://issues.apache.org/jira/browse/HAWQ-1044>. Add
>   some error path test case for hawq register(TestUsage2Case1ErrorH
>   ashTableRegistry, TestUsage2Case1LargerEof, TestUsage2Case1W
>   rongDistributionPolicy)
>   - HAWQ-1044 <https://issues.apache.org/jira/browse/HAWQ-1044>.
>   Enhance hawq register test cases for normal mode and force mode
>   - HAWQ-1044 <https://issues.apache.org/jira/browse/HAWQ-1044>. Fix
>   bugs in testcase TestHawqRegister.TestUsage2Case1IncludeDirectory
>   - HAWQ-1140 <https://issues.apache.org/jira/browse/HAWQ-1140>.
>   Parallelize test cases for hawqregister usage2.
>   - HAWQ-1140 <https://issues.apache.org/jira/browse/HAWQ-1140>.
> Rename
>   yml file name and table name in TestUsage2Case1ErrorHashTableR
> egistry
>   - HAWQ-1249 <https://issues.apache.org/jira/browse/HAWQ-1249>. Don't
>   do ACL checks on segments
>   - HAWQ-1359 <https://issues.apache.org/jira/browse/HAWQ-1359>. Fix
>   bugs in policy tests for HAWQ with Ranger enabled and add test
> for HAWQ-1367
>   - HAWQ-1418 <https://issues.apache.org/jira/browse/HAWQ-1418>. Print
>   executing command for hawq register
>   - HAWQ-1418 <https://issues.apache.org/jira/browse/HAWQ-1418>. Move
>   print executing command after setup logging
>
> *Indirect contributions to code base: *
> Provides a lot of valuable comments for PRs and help improve the quality of
> the codes.
> Reviewed for 34 closed RPs: https://github.com/apache
> /incubator-hawq/pulls?utf8=%E2%9C%93=is%3Apr%20commenter%3Awcl14
>
> *JIRA:*
> *35 JIRAs* are created and assigned with most of them mapping to PRs above.
> 33 JIRAs are closed and 2 JIRAs are in progress (opened pull requests).
> https://issues.apache.org/jira/browse/HAWQ-1416?jql=project%
> 20%3D%20HAWQ%20AND%20(reporter%20in%20(wcl14)%20OR%20assigne
> e%20in%20(wcl14))
>
> Regards,
> Ivan
>



-- 
Thanks

Hubert Zhang

Re: [VOTE]: Apache HAWQ 2.2.0.0-incubating Release (RC3)

[Hubert Zhang]

+1
download the src from https://dist.apache.org/repos/
dist/dev/incubator/hawq/2.2.0.0-incubating.RC3/apache-hawq-
src-2.2.0.0-incubating.tar.gz
make
make install
hawq init cluster and run query successfully

On Wed, Jun 28, 2017 at 9:50 PM, Radar Da lei <r...@pivotal.io> wrote:

> +1
>
> I downloaded the source code tarball, RAT check passed. Compiled, installed
> and initialized HAWQ successfully. InstallCheck-good tests passed.
>
> I also downloaded the binary release tarball, installed all the packages by
> following the wiki page on a clean redhat 7 vm:
> https://cwiki.apache.org/confluence/display/HAWQ/Build+
> Package+and+Install+with+RPM
> After installed the binary packages, I initialized hdfs and hawq
> successfully, basic queries passed. Also verified the
> License/Notice/DISCLAIMER files are installed with the binary packages.
>
>
>
>
>
>
> Regards,
> Radar
>
> On Wed, Jun 28, 2017 at 11:44 AM, Ruilong Huo <h...@apache.org> wrote:
>
> > Hi All,
> >
> >
> > This is the vote for Apache HAWQ (incubating) 2.2.0.0-incubating Release
> > Candidate 3 (RC3). It is a source release for HAWQ core, PXF, and Ranger;
> > and binary release for HAWQ core, and PXF. We have rpm package for the
> > binary release. Though it is not perfect to have binary tarball absent,
> we
> > can add that in the future binary release if there is demand in the
> > community.
> >
> >
> > The vote will run for at least 72 hours and will close on Saturday, July
> > 1, 2017.
> >
> >
> > 1. Issue fixed in RC3:
> > https://issues.apache.org/jira/browse/HAWQ-1475: Add LICENSE, NOTICE,
> and
> > DISCLAIMER files for c/c++ components for Apache HAWQ binary release
> > https://issues.apache.org/jira/browse/HAWQ-1489: Fix PXF jar files, add
> > LICENSE, NOTICE, and DISCLAIMER files for Apache HAWQ binary release
> >
> >
> > 2. Wiki page of the release:
> > https://cwiki.apache.org/confluence/display/HAWQ/Apache+HAWQ+2.2.0.0-
> > incubating+Release
> >
> >
> > 3. Release Notes (Apache Jira generated):
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > projectId=12318826=12339641
> >
> >
> > 4. Release verification steps can be found at:
> > For source tarball: https://cwiki.apache.org/confluence/display/HAWQ/
> > Release+Process%3A+Step+by+step+guide#ReleaseProcess:Stepbystepguide-
> > ValidatetheReleaseCandidate
> > For rpm package: https://cwiki.apache.org/confluence/display/HAWQ/Build+
> > Package+and+Install+with+RPM
> >
> >
> > 5. The tag to be voted on: 2.2.0.0-incubating-rc3 (
> > c08aa560a65af25f1357fc49d736407aff124d05), located here:
> > https://git-wip-us.apache.org/repos/asf?p=incubator-hawq.git;a=commit;h=
> > c08aa560a65af25f1357fc49d736407aff124d05
> >
> >
> > 6. Git release branch:
> > https://git-wip-us.apache.org/repos/asf?p=incubator-hawq.
> > git;a=shortlog;h=refs/heads/2.2.0.0-incubating
> >
> >
> > 7. Source release:
> > Package: https://dist.apache.org/repos/dist/dev/incubator/hawq/2.2.0.
> > 0-incubating.RC3/apache-hawq-src-2.2.0.0-incubating.tar.gz
> > PGP Signature: https://dist.apache.org/repos/
> > dist/dev/incubator/hawq/2.2.0.0-incubating.RC3/apache-hawq-
> > src-2.2.0.0-incubating.tar.gz.asc
> > SHA256 Hash: https://dist.apache.org/repos/dist/dev/incubator/hawq/2.2.0
> .
> > 0-incubating.RC3/apache-hawq-src-2.2.0.0-incubating.tar.gz.sha256
> > MD5 Hash: https://dist.apache.org/repos/dist/dev/incubator/hawq/2.2.0.
> > 0-incubating.RC3/apache-hawq-src-2.2.0.0-incubating.tar.gz.md5
> >
> >
> > 8. Binary release:
> > Package: https://dist.apache.org/repos/dist/dev/incubator/hawq/2.2.0.
> > 0-incubating.RC3/apache-hawq-rpm-2.2.0.0-incubating.tar.gz
> > PGP Signature: https://dist.apache.org/repos/
> > dist/dev/incubator/hawq/2.2.0.0-incubating.RC3/apache-hawq-
> > rpm-2.2.0.0-incubating.tar.gz.asc
> > SHA256 Hash: https://dist.apache.org/repos/dist/dev/incubator/hawq/2.2.0
> .
> > 0-incubating.RC3/apache-hawq-rpm-2.2.0.0-incubating.tar.gz.sha256
> > MD5 Hash: https://dist.apache.org/repos/dist/dev/incubator/hawq/2.2.0.
> > 0-incubating.RC3/apache-hawq-rpm-2.2.0.0-incubating.tar.gz.md5
> >
> >
> > 9. Keys to verify the signature of the release artifact are available at:
> > https://dist.apache.org/repos/dist/dev/incubator/hawq/KEYS
> >
> >
> > 10. The artifact(s) has been signed with Key ID: 1B8B6872
> >
> >
> >
> >
> > Please vote accordingly:
> > [ ] +1 approve
> > [ ] +0 no opinion
> > [ ] -1 disapprove (and reason why)
> >
> >
> > Best regards,
> > Ruilong Huo
>



-- 
Thanks

Hubert Zhang

[jira] [Created] (HAWQ-1476) Augment enable-ranger-plugin.sh to support kerberos.

[Hubert Zhang (JIRA)]

Hubert Zhang created HAWQ-1476:
--

 Summary: Augment enable-ranger-plugin.sh to support kerberos.
 Key: HAWQ-1476
 URL: https://issues.apache.org/jira/browse/HAWQ-1476
 Project: Apache HAWQ
  Issue Type: Sub-task
Reporter: Hubert Zhang
Assignee: Ed Espino


Now ranger can lookup hawq resource in kerberized environment. So we also need 
to change enable-ranger-plugin.sh to automatically fill the authentication type 
and hawq kerberos service name fields



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: New committer: Xiang Sheng

[Hubert Zhang]

Cons!

On Wed, May 17, 2017 at 10:37 AM, Lili Ma <lil...@apache.org> wrote:

> Congratulate Xiang!
>
> Well deserved!
>
> Thanks
> Lili
>
> 2017-05-17 10:34 GMT+08:00 Ma Hongxu <inte...@outlook.com>:
>
> > Congrats!
> > 
> >
> > 在 17/05/2017 09:53, Wen Lin 写道:
> > > Hi,
> > >
> > > The Project Management Committee (PMC) for Apache HAWQ (incubating) has
> > > invited Xiang Sheng to become a committer and we are pleased to
> announce
> > > that he has accepted.
> > > Being a committer enables easier contribution to the project since
> there
> > is
> > > no need to go via the patch submission process. This should enable
> better
> > > productivity. Please join us in congratulating him and we are looking
> > > forward to collaborating with him in the open source community. His
> > > contribution includes (but not limited to):
> > > List contributions to code base, documentation, code review, discussion
> > in
> > > mailing list, JIRA, etc.
> > >
> > > Regards!
> > > Wen
> > >
> >
> > --
> > Regards,
> > Hongxu.
> >
> >
>



-- 
Thanks

Hubert Zhang

[jira] [Created] (HAWQ-1443) Implement Ranger lookup for HAWQ with Kerberos enabled.

[Hubert Zhang (JIRA)]

Hubert Zhang created HAWQ-1443:
--

 Summary: Implement Ranger lookup for HAWQ with Kerberos enabled.
 Key: HAWQ-1443
 URL: https://issues.apache.org/jira/browse/HAWQ-1443
 Project: Apache HAWQ
  Issue Type: Sub-task
Reporter: Hubert Zhang
Assignee: Ed Espino


When add a HAWQ service in Ranger, we also need to configure Ranger look up 
service for HAWQ. Lookup service can be done through JDBC with username and 
password. But It cannot support Kerberos authentication currently.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] Apache HAWQ Incubator "powered by" logo vote

[Hubert Zhang]

+1 for HAWQ logo, with "HAWQ" text, with drop shadow

On Sun, Apr 23, 2017 at 5:08 AM, Alex (Oleksandr) Diachenko <
odiache...@pivotal.io> wrote:

> +1 for HAWQ logo, with "HAWQ" text, with drop shadow
>
> On Sat, Apr 22, 2017 at 9:11 AM, Lili Ma <l...@pivotal.io> wrote:
>
> > +1 HAWQ logo, with "HAWQ" text, with drop shadow
> >
> > On Sat, Apr 22, 2017 at 2:43 AM, Ivan Weng <iw...@pivotal.io> wrote:
> >
> > > +1 HAWQ logo, with "HAWQ" text, with drop shadow
> > >
> > >
> > > Regards,
> > > Ivan
> > >
> > > On Sat, Apr 22, 2017 at 7:30 AM, Shivram Mani <shivram.m...@gmail.com>
> > > wrote:
> > >
> > > > +1 HAWQ logo, with "HAWQ" text, with drop shadow
> > > >
> > > > On Fri, Apr 21, 2017 at 10:48 AM, Ed Espino <esp...@apache.org>
> wrote:
> > > >
> > > > > I have uploaded a few variations of the Apache HAWQ "powered by"
> > logo.
> > > > > Please select your preference via vote. This vote will remain open
> > for
> > > at
> > > > > least 72 hours.
> > > > >
> > > > > [ ] HAWQ logo, no text, no drop shadow
> > > > > [ ] HAWQ logo, no text, with drop shadow
> > > > > [ ] HAWQ logo, with "HAWQ" text, no drop shadow
> > > > > [ ] HAWQ logo, with "HAWQ" text, with drop shadow
> > > > >
> > > > > The full variations and additional sizes can be viewed here:
> > > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > > action?pageId=69407067
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Ed Espino*
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > shivram mani
> > > >
> > >
> >
>



-- 
Thanks

Hubert Zhang

Re: [VOTE]: Apache HAWQ 2.2.0.0-incubating Release (RC2)

[Hubert Zhang]

+1
Download the rpm and install successfully on centos7.
Run SQL successfully with ranger enabled.

On Wed, Apr 12, 2017 at 6:18 AM, Kavinder Dhaliwal <kdhali...@pivotal.io>
wrote:

> +1
>
> Successfully Installed bigtop hadoop rpms on centos7
> Successfully installed hawq & pxf rpms from the binary release
> Successfully verified PGP signatures and checksums
> Successfully created a native table, inserted tuples and selected tuples
> Successfully created an external table with PXF and query a hive table via
> the HiveORC profile
>
> On Mon, Apr 10, 2017 at 3:13 AM, Ruilong Huo <h...@apache.org> wrote:
>
> > Hi All,
> >
> > This is the vote for Apache HAWQ (incubating) 2.2.0.0-incubating
> > Release Candidate 2 (RC2). It is a source release and binary
> > release. We have rpm package for the binary release. Though it is
> > not perfect to have binary tarball absent, we can add that in the future
> > binary release if there is demand in the community.
> >
> > The vote will run for at least 72 hours and will close on Thursday, April
> > 13, 2017.
> >
> > 1. Issue fixed in RC2:
> > https://issues.apache.org/jira/browse/HAWQ-1425: Incorrect init cluster
> > error message while ssh connection failed
> >
> > 2. Wiki page of the release:
> > https://cwiki.apache.org/confluence/display/HAWQ/Apache+HAWQ+2.2.0.0-
> > incubating+Release
> >
> > 3. Release Notes (Apache Jira generated):
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
> > ctId=12318826=12339641
> >
> > 4. Release verification steps can be found at:
> > For source tarball: https://cwiki.apache.org/confluence/display/HAWQ/Re
> > lease+Process%3A+Step+by+step+guide#ReleaseProcess:Stepbyste
> > pguide-ValidatetheReleaseCandidate
> > For rpm package: https://cwiki.apache.org/confluence/display/HAWQ/Bu
> > ild+Package+and+Install+with+RPM
> >
> > 5. The tag to be voted on: 2.2.0.0-incubating-rc2 (
> > 32e01c76b54782d0367204737d412cdf5b6146ff), located here:
> > https://git-wip-us.apache.org/repos/asf?p=incubator-hawq.git;a=commit;h=
> > 32e01c76b54782d0367204737d412cdf5b6146ff
> >
> > 6. Git release branch:
> > https://git-wip-us.apache.org/repos/asf?p=incubator-hawq.git
> > ;a=shortlog;h=refs/heads/2.2.0.0-incubating
> >
> > 7. Source release:
> > Package: https://dist.apache.org/repos/dist/dev/incubator/
> > hawq/2.2.0.0-incubating.RC2/apache-hawq-src-2.2.0.0-incubating.tar.gz
> > PGP Signature: https://dist.apache.org/repos/dist/dev/
> > incubator/hawq/2.2.0.0-incubating.RC2/apache-hawq-
> > src-2.2.0.0-incubating.tar.gz.asc
> > SHA256 Hash: https://dist.apache.org/repos/dist/dev/incubator/hawq/
> > 2.2.0.0-incubating.RC2/apache-hawq-src-2.2.0.0-incubating.tar.gz.sha256
> > MD5 Hash: https://dist.apache.org/repos/dist/dev/incubator/hawq/
> > 2.2.0.0-incubating.RC2/apache-hawq-src-2.2.0.0-incubating.tar.gz.md5
> >
> > 8. Binary release:
> > Package: https://dist.apache.org/repos/dist/dev/incubator/
> > hawq/2.2.0.0-incubating.RC2/apache-hawq-rpm-2.2.0.0-incubating.tar.gz
> > PGP Signature: https://dist.apache.org/repos/dist/dev/
> > incubator/hawq/2.2.0.0-incubating.RC2/apache-hawq-
> > rpm-2.2.0.0-incubating.tar.gz.asc
> > SHA256 Hash: https://dist.apache.org/repos/dist/dev/incubator/hawq/
> > 2.2.0.0-incubating.RC2/apache-hawq-rpm-2.2.0.0-incubating.tar.gz.sha256
> > MD5 Hash: https://dist.apache.org/repos/dist/dev/incubator/hawq/
> > 2.2.0.0-incubating.RC2/apache-hawq-rpm-2.2.0.0-incubating.tar.gz.md5
> >
> > 9. Keys to verify the signature of the release artifact are available at:
> > https://dist.apache.org/repos/dist/dev/incubator/hawq/KEYS
> >
> > 10. The artifact(s) has been signed with Key ID: 1B8B6872
> >
> > To help in tallying the vote, Apache HAWQ Incubator PMC members
> > (http://home.apache.org/phonebook.html?pmc=incubator) please be sure
> > to indicate "(binding)" with your vote.
> >
> > Please vote accordingly:
> > [ ] +1 approve
> > [ ] +0 no opinion
> > [ ] -1 disapprove (and reason why)
> >
>



-- 
Thanks

Hubert Zhang

Re: HAWQ: Web external table on segments.

[Hubert Zhang]

Virtual segments are required from Resource Manager, It cannot guarantee
that resource is available on a certain node at a certain time.
@yijin, Do you have some comments on it?

On Thu, Apr 6, 2017 at 10:13 AM, Hubert Zhang <hzh...@pivotal.io> wrote:

> Why not use gpssh to excute shell on each node?
>
> On Wed, Apr 5, 2017 at 3:11 PM, Cyrille Lintz <cli...@pivotal.io> wrote:
>
>> Hello,
>>
>> From the HDB guide (
>> http://hdb.docs.pivotal.io/212/hawq/reference/sql/CREATE-EXT
>> ERNAL-TABLE.html#topic1__section4),
>> I read about Web external tables
>>
>> *Note: ON ALL/HOST is deprecated when creating a readable external table,
>> as HAWQ cannot guarantee scheduling executors on a specific host. Instead,
>> use ON MASTER, ON , or SEGMENT  to specify which
>> segment instances will execute the command.*
>>
>>
>> In my opinion, if possible, we should re-introduce the ON ALL option for
>> the external WEB tables,
>> I am concerned with the option ON  in the external web table
>> definition. We have to use the number of current hosts. So if we expand
>> the
>> cluster, we will have to change this external web table.
>>
>> - If we have a value smaller than the actual number of hosts, some rows
>> will miss.
>> - If we have a value greater than the actual number of hosts, some rows
>> will be duplicated.
>>
>>
>> If we add the option ON ALL:
>>
>> - it will help to monitor the spill files
>> - it will help to read the segment log files (see the commented DDL
>> hawq_toolkit._hawq_log_segment_ext in the file $GPHOME/share/postgresql)
>>
>>
>> I know that the option ON HOST and ON ALL were deprecated due to elastic
>> runtime in HAWQ 2.x. It is related to the Hadoop architecture.
>>
>> However, how could we execute once a shell on each host of the cluster via
>> an external web table?
>> In this case, we are not using Hadoop FS, but local FS.
>>
>> Thanks,
>>
>>
>> *Cyrille LINTZ*Advisory Solution Architect  |  Pivotal Europe South
>> Mobile: + 33 (0)6 11 48 71 10 | cli...@pivotal.io
>>
>
>
>
> --
> Thanks
>
> Hubert Zhang
>



-- 
Thanks

Hubert Zhang

Re: HAWQ: Web external table on segments.

[Hubert Zhang]

Why not use gpssh to excute shell on each node?

On Wed, Apr 5, 2017 at 3:11 PM, Cyrille Lintz <cli...@pivotal.io> wrote:

> Hello,
>
> From the HDB guide (
> http://hdb.docs.pivotal.io/212/hawq/reference/sql/CREATE-
> EXTERNAL-TABLE.html#topic1__section4),
> I read about Web external tables
>
> *Note: ON ALL/HOST is deprecated when creating a readable external table,
> as HAWQ cannot guarantee scheduling executors on a specific host. Instead,
> use ON MASTER, ON , or SEGMENT  to specify which
> segment instances will execute the command.*
>
>
> In my opinion, if possible, we should re-introduce the ON ALL option for
> the external WEB tables,
> I am concerned with the option ON  in the external web table
> definition. We have to use the number of current hosts. So if we expand the
> cluster, we will have to change this external web table.
>
> - If we have a value smaller than the actual number of hosts, some rows
> will miss.
> - If we have a value greater than the actual number of hosts, some rows
> will be duplicated.
>
>
> If we add the option ON ALL:
>
> - it will help to monitor the spill files
> - it will help to read the segment log files (see the commented DDL
> hawq_toolkit._hawq_log_segment_ext in the file $GPHOME/share/postgresql)
>
>
> I know that the option ON HOST and ON ALL were deprecated due to elastic
> runtime in HAWQ 2.x. It is related to the Hadoop architecture.
>
> However, how could we execute once a shell on each host of the cluster via
> an external web table?
> In this case, we are not using Hadoop FS, but local FS.
>
> Thanks,
>
>
> *Cyrille LINTZ*Advisory Solution Architect  |  Pivotal Europe South
> Mobile: + 33 (0)6 11 48 71 10 | cli...@pivotal.io
>



-- 
Thanks

Hubert Zhang

Re: [VOTE]: Apache HAWQ 2.2.0.0-incubating Release

[Hubert Zhang]

+1

On Wed, Apr 5, 2017 at 10:16 AM, Hubert Zhang <hzh...@pivotal.io> wrote:

> Download the src, build and install successfully.
> Run SQL successfully with ranger enabled.
>
> On Wed, Apr 5, 2017 at 6:42 AM, Shivram Mani <shivram.m...@gmail.com>
> wrote:
>
>> +1
>>
>> On Centos7
>> Downloaded/installed the bigtop hadoop rpms
>> Downloaded/installed hawq and pxf rpms from the above binary release
>> tarbal
>> Verified PGP signatures and SHA256/MD4 checksums
>> Started the components
>> Validated a simple HAWQ query
>> Validated a simple HAWQ External table query via PXF using HdfsTextProfile
>>
>> On Mon, Apr 3, 2017 at 5:30 PM, Yi Jin <y...@pivotal.io> wrote:
>>
>> > +1.
>> >
>> > Checked wiki page, release notes.
>> > Using MacOS downloaded and verified package of both rpm and source code
>> > including (asc, md5, sha256)
>> > Using MacOS downloaded source code tar ball, configured, compiled,
>> > initialized hawq cluster and started/stoped cluster, ran basic feature
>> > tests.
>> > Using Centos7 downloaded rpm, installed, initialized hawq cluster,
>> > started/stopped cluster, ran basic queries.
>> >
>> > Best,
>> > Yi
>> >
>> >
>> > On Sun, Apr 2, 2017 at 1:29 AM, Ruilong Huo <h...@apache.org> wrote:
>> >
>> > > Hi All,
>> > >
>> > > This is the vote for Apache HAWQ (incubating) 2.2.0.0-incubating
>> > > Release Candidate 1 (RC1). It is a source release and binary
>> > > release. We have rpm package for the binary release. Though it is
>> > > not perfect to have binary tarball absent, we can add that in the
>> future
>> > > binary release if there is demand in the community.
>> > >
>> > > The vote will run for at least 120 hours and will close on Thursday,
>> > > April 6, 2017 as there is weekend and holiday in next few days.
>> > >
>> > > 1. Wiki page of the release:
>> > > https://cwiki.apache.org/confluence/display/HAWQ/Apache+HAWQ
>> > > +2.2.0.0-incubating+Release
>> > >
>> > > 2. Release Notes (Apache Jira generated):
>> > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
>> > > ctId=12318826=12339641
>> > >
>> > > 3. Release verification steps can be found at:
>> > > For source tarball:
>> > > https://cwiki.apache.org/confluence/display/HAWQ/Release+Pro
>> > > cess%3A+Step+by+step+guide#ReleaseProcess:Stepbystepguide-
>> > > ValidatetheReleaseCandidate
>> > > For rpm package:
>> > > https://cwiki.apache.org/confluence/display/HAWQ/Build+Packa
>> > > ge+and+Install+with+RPM
>> > >
>> > > 4. The tag to be voted on: 2.2.0.0-incubating-rc1
>> > > (18e928c91e36ec0bf6af7f9760fc8a7445537816), located here:
>> > > https://git-wip-us.apache.org/repos/asf?p=incubator-hawq.git
>> > > ;a=commit;h=18e928c91e36ec0bf6af7f9760fc8a7445537816
>> > >
>> > > 5. Git release branch:
>> > > https://git-wip-us.apache.org/repos/asf?p=incubator-hawq.git
>> > > ;a=shortlog;h=refs/heads/2.2.0.0-incubating
>> > >
>> > > 6. Source release:
>> > > Package:
>> > > https://dist.apache.org/repos/dist/dev/incubator/hawq/2.2.0.
>> > > 0-incubating.RC1/apache-hawq-src-2.2.0.0-incubating.tar.gz
>> > > PGP Signature:
>> > > https://dist.apache.org/repos/dist/dev/incubator/hawq/2.2.0.
>> > > 0-incubating.RC1/apache-hawq-src-2.2.0.0-incubating.tar.gz.asc
>> > > SHA256
>> > > <https://dist.apache.org/repos/dist/dev/incubator/hawq/
>> > 2.2.0.0-incubating.RC1/apache-hawq-src-2.2.0.0-incubating.ta
>> r.gz.ascSHA256
>> > >
>> > > Hash:
>> > > https://dist.apache.org/repos/dist/dev/incubator/hawq/2.2.0.
>> > > 0-incubating.RC1/apache-hawq-src-2.2.0.0-incubating.tar.gz.sha256
>> > > MD5
>> > > <https://dist.apache.org/repos/dist/dev/incubator/hawq/
>> > 2.2.0.0-incubating.RC1/apache-hawq-src-2.2.0.0-incubating.ta
>> r.gz.sha256MD5
>> > >
>> > > Hash:
>> > > https://dist.apache.org/repos/dist/dev/incubator/hawq/2.2.0.
>> > > 0-incubating.RC1/apache-hawq-src-2.2.0.0-incubating.tar.gz.md5
>> > >
>> > > 7. Binary release:
>> > > Package:
>> > > https://dist.apache.org/repos/dist/dev/incubator/hawq/2.2.0.
&

Re: [VOTE]: Apache HAWQ 2.2.0.0-incubating Release

[Hubert Zhang]

t; https://dist.apache.org/repos/dist/dev/incubator/hawq/2.2.0.
> > > 0-incubating.RC1/apache-hawq-rpm-2.2.0.0-incubating.tar.gz.sha256
> > > MD5
> > > <https://dist.apache.org/repos/dist/dev/incubator/hawq/
> > 2.2.0.0-incubating.RC1/apache-hawq-rpm-2.2.0.0-incubating.
> tar.gz.sha256MD5
> > >
> > > Hash:
> > > https://dist.apache.org/repos/dist/dev/incubator/hawq/2.2.0.
> > > 0-incubating.RC1/apache-hawq-rpm-2.2.0.0-incubating.tar.gz.md5
> > >
> > > 8. Keys to verify the signature of the release artifact are available
> at:
> > > https://dist.apache.org/repos/dist/dev/incubator/hawq/KEYS
> > >
> > > 9. The artifact(s) has been signed with Key ID: 1B8B6872
> > >
> > > To help in tallying the vote, Apache HAWQ Incubator PMC members
> > > (http://home.apache.org/phonebook.html?pmc=incubator) please be sure
> > > to indicate "(binding)" with your vote.
> > >
> > > Please vote accordingly:
> > > [ ] +1 approve
> > > [ ] +0 no opinion
> > > [ ] -1 disapprove (and reason why)
> > >
> >
>
>
>
> --
> shivram mani
>



-- 
Thanks

Hubert Zhang

[jira] [Created] (HAWQ-1375) Ranger should always using the current user to do privilege check.

[Hubert Zhang (JIRA)]

Hubert Zhang created HAWQ-1375:
--

 Summary: Ranger should always using the current user to do 
privilege check.
 Key: HAWQ-1375
 URL: https://issues.apache.org/jira/browse/HAWQ-1375
 Project: Apache HAWQ
  Issue Type: Bug
Reporter: Hubert Zhang
Assignee: Ed Espino


Failure Case:
{code}
user u1 create table a(i int);
user u2 create view av as select * from a;
user u3 select * from av.
{code}
When ORCA is on, u3 will first ask select privilege as user u3 to Ranger, and 
then ask select privilege as user u2 to Ranger.
The second check should be removed, since there may be no privilege for u2 to 
select av even if av is created by u2 in Ranger mode.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HAWQ-1370) Misuse of regular expressions in init_file of feature test.

[Hubert Zhang (JIRA)]

Hubert Zhang created HAWQ-1370:
--

 Summary: Misuse of regular expressions in init_file of feature 
test.
 Key: HAWQ-1370
 URL: https://issues.apache.org/jira/browse/HAWQ-1370
 Project: Apache HAWQ
  Issue Type: Bug
Reporter: Hubert Zhang
Assignee: Ed Espino


in global_init_file of feature test, we want to skip expressions which include 
file and line number, e.g.(aclchk.c:123), or (aclchk.cpp:134).
But currently, the regular expressions is \(.*c[p]+:\d+\) which need to be 
replaced by (.*c[p]*:\d+\)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: PR build of Jenkins failed

[Hubert Zhang]

Thanks Ed. I will follow it too.

On Mon, Feb 27, 2017 at 3:31 PM, Ed Espino <esp...@apache.org> wrote:

> FYI: In order to debug this and as it is impacting all PR builds in the
> Apache Jenkins environment, I will be manually triggering builds to try and
> understand and resolve this issue. I may also be aborting current jobs so
> please do not be alarmed if your PR's Apache Jenkins check aborts.
>
> -=e
>
> On Sun, Feb 26, 2017 at 10:56 PM, Ed Espino <esp...@apache.org> wrote:
>
>> Hubert,
>>
>> The Pull Request (PR) jobs take place in the Apache Jenkins service
>> HAWQ-build-pullrequest
>> <https://builds.apache.org/job/HAWQ-build-pullrequest/> job. This was in
>> place prior to me actively joining the Apache HAWQ dev community. I have
>> just noticed the jobs for multiple PRs have been failing for the same
>> reason. I will take a look to see if I can identify what the core issue is.
>>
>> BTW: I have CC'd the dev list for wider visibility. Maybe someone on the
>> list can also provide some insight on this issue.
>>
>> -=e
>>
>>
>> On Sun, Feb 26, 2017 at 9:50 PM, Hubert Zhang <hzh...@pivotal.io> wrote:
>>
>>> Hi Ed,
>>>
>>> Jenkins (build of PR1146) cannot clean up environment with the following
>>> error messages:
>>>
>>> Cloning the remote Git repository
>>>
>>> Cloning repository git://github.com/apache/incubator-hawq.git
>>>
>>> ERROR: Failed to clean the workspace
>>> java.io.IOException: Unable to delete 
>>> '/home/jenkins/jenkins-slave/workspace/HAWQ-build-pullrequest'. Tried 3 
>>> times (of a maximum of 3) waiting 0.1 sec between attempts.
>>>
>>>
>>>
>>> I think it's an environment issue and PR1146 is the blocker of HDB
>>> pipeline, so we will +1 firstly to push the code and make HDB pipeline
>>> green.
>>>
>>> Could you also show some hints on how to fix Jenkins env problem, and we
>>> can fix it by ourselves next time.
>>>
>>> --
>>> Thanks
>>>
>>> Hubert Zhang
>>>
>>
>>
>>
>> --
>> *Ed Espino*
>> *esp...@apache.org <esp...@apache.org>*
>>
>
>
>
> --
> *Ed Espino*
> *esp...@apache.org <esp...@apache.org>*
>



-- 
Thanks

Hubert Zhang

[jira] [Created] (HAWQ-1360) Check privilege of sequence pass the wrong type to RPS.

[Hubert Zhang (JIRA)]

Hubert Zhang created HAWQ-1360:
--

 Summary: Check privilege of sequence pass the wrong type to RPS.
 Key: HAWQ-1360
 URL: https://issues.apache.org/jira/browse/HAWQ-1360
 Project: Apache HAWQ
  Issue Type: Bug
Reporter: Hubert Zhang
Assignee: Ed Espino


When check privilege of query with ranger enabled, a json request will be send 
to RPS, including information about: privilege type, resource type and role 
name. 
Currently, checking sequence privilege will send resource type "table" instead 
of  "sequence".



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HAWQ-1359) Add policy test for HAWQ with Ranger enabled.

[Hubert Zhang (JIRA)]

Hubert Zhang created HAWQ-1359:
--

 Summary: Add policy test for HAWQ with Ranger enabled.
 Key: HAWQ-1359
 URL: https://issues.apache.org/jira/browse/HAWQ-1359
 Project: Apache HAWQ
  Issue Type: Sub-task
Reporter: Hubert Zhang
Assignee: Ed Espino


Policy test includes different json requests(used by ranger) for different 
quries, For example, for  query "select * from a;", it needs usage privilege of 
schema public, and select privilege of table a.
There are also queries can only be executed by superuser, we also test them in 
policy test.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HAWQ-1358) Refactor gpfdist library in featuretest.

[Hubert Zhang (JIRA)]

Hubert Zhang created HAWQ-1358:
--

 Summary: Refactor gpfdist library in featuretest.
 Key: HAWQ-1358
 URL: https://issues.apache.org/jira/browse/HAWQ-1358
 Project: Apache HAWQ
  Issue Type: Improvement
Reporter: Hubert Zhang
Assignee: Ed Espino


extract gpfdist as a common library, to be used by both exttable and ranger 
feature test.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HAWQ-1357) Super user also need to check create privilege of public schema from Ranger.

[Hubert Zhang (JIRA)]

Hubert Zhang created HAWQ-1357:
--

 Summary: Super user also need to check create privilege of public 
schema from Ranger.
 Key: HAWQ-1357
 URL: https://issues.apache.org/jira/browse/HAWQ-1357
 Project: Apache HAWQ
  Issue Type: Bug
Reporter: Hubert Zhang
Assignee: Ed Espino


[HAWQ-1318|https://issues.apache.org/jira/browse/HAWQ-1318] add create|usage 
privilege of public schema to superuser to fix hawq stop BUG caused by Resource 
Manager. 
But RM only need the usage privilege of public schema to query HAWQ, So create 
privilege need to be removed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HAWQ-1356) Add waring when user does not have usage privilege of namespace.

[Hubert Zhang (JIRA)]

Hubert Zhang created HAWQ-1356:
--

 Summary: Add waring when user does not have usage privilege of 
namespace.
 Key: HAWQ-1356
 URL: https://issues.apache.org/jira/browse/HAWQ-1356
 Project: Apache HAWQ
  Issue Type: Bug
Reporter: Hubert Zhang
Assignee: Ed Espino


Currently, in Ranger mode, when a user doesn't have usage privilege on public 
schema, she cannot see any tables, functions, schemas on public schema. When 
she login into database and run a query "select * from a"; The error message 
would be "table a doesn't exist", which makes user confuse.
This fix add a warning to tell user that she does not have the usage privilege 
of  a schema.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HAWQ-1355) Namespace check may occur multiple times in first query.

[Hubert Zhang (JIRA)]

Hubert Zhang created HAWQ-1355:
--

 Summary: Namespace check may occur multiple times in first query.
 Key: HAWQ-1355
 URL: https://issues.apache.org/jira/browse/HAWQ-1355
 Project: Apache HAWQ
  Issue Type: Bug
Reporter: Hubert Zhang
Assignee: Ed Espino


When running a query, HAWQ need to check namespace usage privilege in function 
recomputeNamespacePath. This function will be called repeatedly but check will 
be skipped when last_query_sign is equal to current_query_sign.
There is a bug that running the first query doesn't set the last_query_sign.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] Release Apache HAWQ 2.1.0.0-incubating (RC4)

[Hubert Zhang]

/dist.apache.org/repos/dist/dev/incubator/hawq/2.1.0.
> 0-incubating.RC4/apache-hawq-src-2.1.0.0-incubating.tar.gz.asc
> * SHA256/MD5 Hash:
>
> https://dist.apache.org/repos/dist/dev/incubator/hawq/2.1.0.
> 0-incubating.RC4/apache-hawq-src-2.1.0.0-incubating.tar.gz.sha256
>
> https://dist.apache.org/repos/dist/dev/incubator/hawq/2.1.0.
> 0-incubating.RC4/apache-hawq-src-2.1.0.0-incubating.tar.gz.md5
>
> Keys to verify the signature of the release artifact are available at:
> * https://dist.apache.org/repos/dist/dev/incubator/hawq/KEYS
>
> The artifact(s) has been signed with Key ID: 57325522
>
> Previous Vote Thread(s):
> * RC1 [VOTE] Thread
>
> https://lists.apache.org/thread.html/882dc11880f8794fc5603aee470c0e
> 5f912f579a7c247c270dbeb9a4@%3Cdev.hawq.apache.org%3E
> * RC1 [RESULT][VOTE] Thread
>
> https://lists.apache.org/thread.html/ada53fa0dc9547bee47d8e73cc09ec
> 3e99e6335c291a78155b5af139@%3Cdev.hawq.apache.org%3E
> * RC2 [VOTE] Thread
>
> https://lists.apache.org/thread.html/02cb5e2218c516b3c17c3819afdcc4
> fdbff670a0c01918168650e981@%3Cdev.hawq.apache.org%3E
> * RC2 [RESULT][VOTE] Thread
>
> https://lists.apache.org/thread.html/de507d211356e00e1c07b857811a89
> 258755feaacf1533e2d49cbfb9@%3Cdev.hawq.apache.org%3E
> * RC3 [VOTE] Thread
>
> https://lists.apache.org/thread.html/ac664618426411fa2021b2b36b4ce4
> 2f9d5bc505a9aa8d901c14fbbd@%3Cdev.hawq.apache.org%3E
> * RC3 [RESULT][VOTE] Thread
>
> https://lists.apache.org/thread.html/4463b3f7bee2d7e3e10e71fe01ce70
> 348bd23a493e00d08730343eea@%3Cdev.hawq.apache.org%3E
>
> To help in tallying the vote, Apache HAWQ Incubator PMC members
> (http://home.apache.org/phonebook.html?pmc=incubator) please be sure
> to indicate "(binding)" with your vote.
>
> Please vote accordingly:
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove (and reason why)
>
> Regards,
> -=e
>
>
> --
> *Ed Espino*
> *esp...@apache.org <esp...@apache.org>*
>



-- 
Thanks

Hubert Zhang

[jira] [Created] (HAWQ-1328) Add deny and exclude policy template for hawq service in ranger.

[Hubert Zhang (JIRA)]

Hubert Zhang created HAWQ-1328:
--

 Summary: Add deny and exclude policy template for hawq service in 
ranger.
 Key: HAWQ-1328
 URL: https://issues.apache.org/jira/browse/HAWQ-1328
 Project: Apache HAWQ
  Issue Type: Sub-task
Reporter: Hubert Zhang
Assignee: Ed Espino


Currently, there is no template of deny and exclude policy for HAWQ service in 
Ranger, we need to open this option in ranger-servicedef-hawq.json by default.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HAWQ-1311) Optimize the performance of hawq with ranger enabled.

[Hubert Zhang (JIRA)]

Hubert Zhang created HAWQ-1311:
--

 Summary: Optimize the performance of hawq with ranger enabled.
 Key: HAWQ-1311
 URL: https://issues.apache.org/jira/browse/HAWQ-1311
 Project: Apache HAWQ
  Issue Type: Sub-task
Reporter: Hubert Zhang
Assignee: Ed Espino


There are still more than one ranger plugin server(rps) call for a single 
query. We need to first analyse the cost of ranger aclcheck and then using a 
query level aclcache to minimise the times of rps call for each query. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Ranger Discussion: Scope for HAWQ Ranger Integration

[Hubert Zhang]

tch_xlog
> >  -
> >
> >  pg_stat_reset
> >  3.
> >
> >Catalog table/system embedded function/owner check kept in HAWQ side
> >4.
> >
> >Forbid grant/revoke command for non-heap table in HAWQ once Ranger is
> >configured
> >5.
> >
> >Documents for telling people if they firstly enable ranger and then
> >don't want to use it, they need manually recreate all the policies of
> >Ranger in HAWQ own side
> >6.
> >
> >Documents for how to enable Ranger in Ambari:  Firstly set ranger off,
> >init HAWQ master, then set ranger on, and then restart HAWQ cluster.
> >
> >
> > Out of Scope for the first Release
> >
> >1.
> >
> >Ambari modification: We just use Ambari's customized GUC for the first
> >release instead of adding a new GUC, so no change from Ambari side
> >2.
> >
> >Kerberos/SSL connection from HAWQ to RPS and RPS to Ranger
> >3.
> >
> >HA: There are two levels of HA: Ranger Server HA and RPS HA. For
> >Ranger Server HA, RPS needs to be designed to be tolerant for this;
> For RPS
> >HA, HAWQ master and standby master should be able to connect to
> another RPS
> >if one is down.
> >4.
> >
> >Catalog table/System embedded function/owner check in Ranger
> >5.
> >
> >Allow both Grant/Revoke command and Ranger side ACL control for the
> >same objects such as non-heap table
> >6.
> >
> >Tool for converting all privileges defined in Ranger to HAWQ
> >grant/revoke command
> >7.
> >
> >Tool for syncing HAWQ user information from LDAP
> >8.
> >
> >Ranger check for drop table/create database
> >
> >
> > Best Regards,
> > Lili
> >
>



-- 
Thanks

Hubert Zhang

[jira] [Created] (HAWQ-1282) Shared Input Scan may result in endless loop

[Hubert Zhang (JIRA)]

Hubert Zhang created HAWQ-1282:
--

 Summary: Shared Input Scan may result in endless loop
 Key: HAWQ-1282
 URL: https://issues.apache.org/jira/browse/HAWQ-1282
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Core
Reporter: Hubert Zhang
Assignee: Ed Espino


There are residual process after running some queries.  Through the call stack, 
we find that there is an endless loop in function writer_wait_for_acks() in 
shared input scan.
We plan to add max retry times to avoid this problem.
Also, we fix file handler leak in retry_read and retry_write.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-1275) Check build-in catalogs, tables and functions in native aclcheck.

[Hubert Zhang (JIRA)]

Hubert Zhang created HAWQ-1275:
--

 Summary: Check build-in catalogs, tables and functions in native 
aclcheck.
 Key: HAWQ-1275
 URL: https://issues.apache.org/jira/browse/HAWQ-1275
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: Security
Reporter: Hubert Zhang
Assignee: Ed Espino


We plan to do privilege check in hawq side for build-in catalogs, tables and 
functions. The reasons are two folds;
1 Ranger mainly manage the user data, but build-in catalogs and tables are not 
related to user data(note that some of them contain statistics information of 
user data such as catalog table pg_aoseg_*).
2 We haven't finish the code of merge of all the privilege check requests into 
one big request. Without it query such as "\d" and "analyze" will lead to 
hundreds of RPS request.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE]: Apache HAWQ 2.1.0.0-incubating Release

[Hubert Zhang]

Built and installed successfully!
+1

On Thu, Jan 12, 2017 at 3:13 AM, Hong <xunzh...@apache.org> wrote:

> Build succeeded on new MBP, RAT check passed.
> +1 on this great release! Good job!
>
> Regards,
> Hong
>
> 2017-01-10 22:16 GMT-05:00 Ed Espino <esp...@apache.org>:
>
> > This is the vote for 2.1.0.0-incubating of Apache HAWQ (incubating).
> This
> > is a Source only release.
> >
> > The vote will run for at least 72 hours and will close on Jan 14, 2017.
> >
> > Release Notes (Jira generated):
> >
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > projectId=12318826=12338900
> >
> > Release verification steps can be found at:
> >
> > https://cwiki.apache.org/confluence/display/HAWQ/
> > Release+Process%3A+Step+by+step+guide#ReleaseProcess:Stepbystepguide-
> > ValidatetheReleaseCandidate
> >
> > Git branch for the release:
> > https://github.com/apache/incubator-hawq/tree/2.1.0.0-incubating
> > Sources for the release:
> >
> > https://dist.apache.org/repos/dist/dev/incubator/hawq/2.1.0.
> > 0-incubating.RC1/apache-hawq-src-2.1.0.0-incubating.tar.gz
> > Source release verification:
> > PGP Signature:
> >
> > https://dist.apache.org/repos/dist/dev/incubator/hawq/2.1.0.
> > 0-incubating.RC1/apache-hawq-src-2.1.0.0-incubating.tar.gz.asc
> > MD5/SHA256 Hash:
> >
> > https://dist.apache.org/repos/dist/dev/incubator/hawq/2.1.0.
> > 0-incubating.RC1/apache-hawq-src-2.1.0.0-incubating.tar.gz.md5
> >
> > https://dist.apache.org/repos/dist/dev/incubator/hawq/2.1.0.
> > 0-incubating.RC1/apache-hawq-src-2.1.0.0-incubating.tar.gz.sha256
> > Keys to verify the signature of the release artifact are available at:
> > https://dist.apache.org/repos/dist/dev/incubator/hawq/KEYS
> > The artifact(s) have been signed with Key : 57325522
> >
> > Please vote accordingly:
> > [ ] +1  approve
> > [ ] +0  no opinion
> > [ ] -1  disapprove (and reason why)
> >
> >
> > --
> > *Ed Espino*
> > *esp...@apache.org <esp...@apache.org>*
> >
>



-- 
Thanks

Hubert Zhang

[jira] [Created] (HAWQ-1243) Add suffix name for ranger restful service.

[Hubert Zhang (JIRA)]

Hubert Zhang created HAWQ-1243:
--

 Summary: Add suffix name for ranger restful service.
 Key: HAWQ-1243
 URL: https://issues.apache.org/jira/browse/HAWQ-1243
 Project: Apache HAWQ
  Issue Type: Improvement
Reporter: Hubert Zhang
Assignee: Ed Espino


Except rps_addr_host and rps_addr_port, we also need rps_addr_suffix to
We will use this GUC to construct rest service address:
http://rps_addr_host:rps_addr_port/rps_addr_suffix



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-1220) Support ranger plugin server HA in hawq side.

[Hubert Zhang (JIRA)]

Hubert Zhang created HAWQ-1220:
--

 Summary: Support ranger plugin server HA in hawq side.
 Key: HAWQ-1220
 URL: https://issues.apache.org/jira/browse/HAWQ-1220
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: Security
Reporter: Hubert Zhang
Assignee: Ed Espino


RPS will run both at master and at standby master, If connection to master RPS 
failed, we should try to connect to standby master instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Random tables loaded from files

[Hubert Zhang]

:
>executors used(total/cached/new connection): (60/1/59); dispatcher
> time(total/connection/dispatch data): (3254.181 ms/1480539781639.241
> ms/0.483 ms).
>dispatch data time(max/min/avg): (0.021 ms/0.005 ms/0.007 ms); consume
> executor data time(max/min/avg): (0.044 ms/0.003 ms/0.009 ms); free
> executor time(max/min/avg
> ): (0.000 ms/0.000 ms/0.000 ms).
>  Data locality statistics:
>data locality ratio: 1.000; virtual segment number: 60; different host
> number: 10; virtual segment number per host(avg/min/max): (6/6/6); segment
> size(avg/min/max):
>  (186145.950 B/182662 B/189757 B); segment size with penalty(avg/min/max):
> (0.000 B/0 B/0 B); continuity(avg/min/max): (1.000/1.000/1.000); DFS
> metadatacache: 0.263 ms
> ; resource allocation: 0.521 ms; datalocality calculation: 0.114 ms.
>  Total runtime: 19549.798 ms
> (21 rows)
>
> Time: 19651.159 ms
>
> This is just a small dimension table too.
>
> I then loaded that data from the local table to a new random table.
>
> gpadmin=# explain analyze select count(*) from tpcds_opt.date_dim;
>
>
>
>   QUERY PLAN
>
>
>
>
> 
> 
> ---
> 
> 
> ---
> 
> --
>  Aggregate  (cost=0.00..436.36 rows=1 width=8)
>Rows out:  Avg 1.0 rows x 1 workers.
> Max/Last(seg-1:ip-172-21-4-229.ec2.internal/seg-1:ip-172-
> 21-4-229.ec2.internal)
> 1/1 rows with 2624/2624 ms to end, start offse
> t by 1.081/1.081 ms.
>->  Gather Motion 1:1  (slice1; segments: 1)  (cost=0.00..436.36 rows=1
> width=8)
>  Rows out:  Avg 1.0 rows x 1 workers at destination.
> Max/Last(seg-1:ip-172-21-4-229.ec2.internal/seg-1:ip-172-
> 21-4-229.ec2.internal)
> 1/1 rows with 2624/2624 m
> s to end, start offset by 1.082/1.082 ms.
>  ->  Aggregate  (cost=0.00..436.36 rows=1 width=8)
>Rows out:  Avg 1.0 rows x 1 workers.
> Max/Last(seg0:ip-172-21-4-225.ec2.internal/seg0:ip-172-21-4-
> 225.ec2.internal)
> 1/1 rows with 2621/2621 ms to end, s
> tart offset by 0.846/0.846 ms.
>->  Table Scan on date_dim  (cost=0.00..436.22 rows=73049
> width=1)
>  Rows out:  Avg 73049.0 rows x 1 workers.
> Max/Last(seg0:ip-172-21-4-225.ec2.internal/seg0:ip-172-21-4-
> 225.ec2.internal)
> 73049/73049 rows with 2.96
> 6/2.966 ms to first row, 2595/2595 ms to end, start offset by 0.847/0.847
> ms.
>  Slice statistics:
>(slice0)Executor memory: 170K bytes.
>(slice1)Executor memory: 343K bytes
> (seg0:ip-172-21-4-225.ec2.internal).
>  Statement statistics:
>Memory used: 262144K bytes
>  Settings:  default_hash_table_bucket_number=60; optimizer=on
>  Optimizer status: PQO version 1.638
>  Dispatcher statistics:
>executors used(total/cached/new connection): (1/1/0); dispatcher
> time(total/connection/dispatch data): (0.122 ms/0.000 ms/0.015 ms).
>dispatch data time(max/min/avg): (0.015 ms/0.015 ms/0.015 ms); consume
> executor data time(max/min/avg): (0.011 ms/0.011 ms/0.011 ms); free
> executor time(max/min/avg
> ): (0.000 ms/0.000 ms/0.000 ms).
>  Data locality statistics:
>data locality ratio: 0.296; virtual segment number: 1; different host
> number: 1; virtual segment number per host(avg/min/max): (1/1/1); segment
> size(avg/min/max): (
> 11168757.000 B/11168757 B/11168757 B); segment size with
> penalty(avg/min/max): (11247341.000 B/11247341 B/11247341 B);
> continuity(avg/min/max): (1.000/1.000/1.000); DF
> S metadatacache: 0.254 ms; resource allocation: 0.387 ms; datalocality
> calculation: 0.130 ms.
>  Total runtime: 2627.711 ms
> (21 rows)
>
> Time: 2728.409 ms
>
> I'm able to decrease the query execution time of many queries by
> increasing hawq_rm_nvseg_perquery_perseg_limit but only for the tables
> loaded from other local tables and not tables loaded from external tables.
> Based on the documentation, this appears to be the expected behavior.
>
> - Are there plans to correct this?
> - What happens if the cluster expands?  Will these random need to be
> redistributed?
> - Are there workarounds to this issue?
>
> Jon Roberts
>



-- 
Thanks

Hubert Zhang

Re: new committer: Paul Guo

[Hubert Zhang]

   ssh_utils.py:addHostNameAlternatives() should not use short
> >  hostname
> >   - *2 documentation improvement*
> >  - HAWQ-868 <https://issues.apache.org/jira/browse/HAWQ-868>
> README.md
> >  needs to update
> >  - HAWQ-718 <https://issues.apache.org/jira/browse/HAWQ-718>
> wiki
> >  of "Build and Install" for CentOS 7.x has some little issues
> >   - *Indirect** contribution(Code Reviews) to code base:*
> >- Provided a lot of insightful comments and collaborated interactively
> >   with contributor and committer to improve the quality of the
> product
> >   - Reviewed for 69 PR
> >   <https://github.com/apache/incubator-hawq/pulls?utf8=%E2%
> 9C%93=is%3Apr%20commenter%3Apaul-guo->
> > including
> >   66 closed and 3 open
> >   - https://cwiki.apache.org/confluence/display/HAWQ/Build+
> and+Install
> >- *Mailing List:*
> >- Raised a lot of topics to improve the usability as well as advocated
> >   product overall in the community. Email count shows that: 28 in
> > July, 6 in
> >   Aug., 9 in Sep.
> >   - http://mail-archives.apache.org/mod_mbox/incubator-hawq-dev
> >   - *Jira Created/Assigned*:
> >- 58 JIRAs are created and assigned with most of them mapping to PR
> >   above: use project = HAWQ AND (reporter in ("Paul Guo") OR assignee
> >   in ("Paul Guo"))
> >   <https://issues.apache.org/jira/issues/?jql=project%20%
> 3D%20HAWQ%20AND%20(reporter%20in%20(%22Paul%20Guo%22)%
> 20OR%20assignee%20in%20(%22Paul%20Guo%22))>
> > to
> >   filter
> >
> > Best regards,
> > Ruilong Huo
>



-- 
Thanks

Hubert Zhang

[jira] [Created] (HAWQ-1143) Libhdfs create semantic is not consistent with posix standard.

[Hubert Zhang (JIRA)]

Hubert Zhang created HAWQ-1143:
--

 Summary: Libhdfs create semantic is not consistent with posix 
standard.
 Key: HAWQ-1143
 URL: https://issues.apache.org/jira/browse/HAWQ-1143
 Project: Apache HAWQ
  Issue Type: Bug
  Components: libhdfs
Reporter: Hubert Zhang
Assignee: Lei Chang


Open a file under posix standard, if o_create flag is set to true and the file 
exists, no side effect except O_EXCL is also be set true.
Open a file in HDFS with  hdfs::create flag will report errors if file exists.

In libhdfs, the o_create flag is interpreted to hdfs::create, which leads to 
errors if file exists no matter O_EXCL is set or not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Apache HAWQ 2.0.0.0-incubating RC4

[Hubert Zhang]

t; > > > > > > 
> > > > > > > [INFO] Total time: 2.921 s
> > > > > > > [INFO] Finished at: 2016-09-24T10:58:19-07:00
> > > > > > > [INFO] Final Memory: 12M/364M
> > > > > > > [INFO] --
> > > --
> > > > > > > 
> > > > > > > ✔ ~/workspace/HAWQ-projects/apache-hawq-src-2.0.0.0-incubating
> > > > > > > 10:58 $
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > #
> > > > > > > APACHE RAT OUTOUT
> > > > > > > #
> > > > > > >
> > > > > > > *
> > > > > > > Summary
> > > > > > > ---
> > > > > > > Generated at: 2016-09-24T10:36:58-07:00
> > > > > > > Notes: 4
> > > > > > > Binaries: 0
> > > > > > > Archives: 1
> > > > > > > Standards: 2893
> > > > > > >
> > > > > > > Apache Licensed: 2061
> > > > > > > Generated Documents: 0
> > > > > > >
> > > > > > > JavaDocs are generated and so license header is optional
> > > > > > > Generated files do not required license headers
> > > > > > >
> > > > > > > 0 Unknown Licenses
> > > > > > >
> > > > > > > ***
> > > > > > >
> > > > > > > Unapproved licenses:
> > > > > > >
> > > > > > >
> > > > > > > ***
> > > > > > >
> > > > > > > Archives:
> > > > > > >
> > > > > > >  + pxf/pxf-json/src/test/resources/tweets.tar.gz
> > > > > > >
> > > > > > > *
> > > > > > >   Files with Apache License headers will be marked AL
> > > > > > >   Binary files (which do not require AL headers) will be
> marked B
> > > > > > >   Compressed archives will be marked A
> > > > > > >   Notices, licenses etc will be marked N
> > > > > > >   PGSQL contrib/contrib-global.mk
> > > > > > >   ALcontrib/extprotocol/gpextprotocol.c
> > > > > > >   ALcontrib/extprotocol/Makefile
> > > > > > >   ALcontrib/formatter_fixedwidth/fixedwidth.c
> > > > > > >   ALcontrib/formatter_fixedwidth/Makefile
> > > > > > > ...
> > > > > > > 
> > > > > > > ...
> > > > > > >
> > > > > > >   ALtools/sbin/gpoperation.py
> > > > > > >   ALtools/sbin/hawqstandbywatch.py
> > > > > > >   ALtools/sbin/Makefile
> > > > > > >
> > > > > > > *
> > > > > > >  Printing headers for files without AL header...
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Sat, Sep 24, 2016 at 9:53 AM, Goden Yao <
> goden...@apache.org>
> > > > > wrote:
> > > > > > >
> > > > > > >> This is the 1st release for Apache HAWQ (incubating), version:
> > > > > > >> 2.0.0.0-incubating
> > > > > > >> I want to take the opportunity to thank Ed, Roman and Justin
> for
> > > the
> > > > > > final
> > > > > > >> verification of all IP issues.
> > > > > > >>
> > > > > > >> *It fixes the following issues:*
> > > > > > >> Clear all IP related issues for HAWQ and this is a source code
> > > > tarball
> > > > > > >> only
> > > > > > >> release.
> > > > > > >> Full list of JIRAs fixed/related to the release: link
> > > > > > >> <https://cwiki.apache.org/confluence/display/HAWQ/HAWQ+Relea
> > > > > > >> se+2.0.0.0-incubating>
> > > > > > >>
> > > > > > >> *** Please download, review and vote by *Friday 12pm Sep 30,
> > 2016
> > > > PST*
> > > > > > ***
> > > > > > >> or When we have enough votes to bring this source tarball to
> > IPMC
> > > > > > >>
> > > > > > >> *We're voting upon the release branch:*
> > > > > > >> 2.0.0.0-incubating
> > > > > > >> HEAD: commit
> > > > > > >> <https://git-wip-us.apache.org/repos/asf?p=incubator-hawq.
> > > > > > >> git;a=commit;h=6b2de9d09f86fa08cf4fef0dd4e3dbe3b1b3dfd5>
> > > > > > >>
> > > > > > >> To run check RAT, please do:
> > > > > > >>
> > > > > > >> $mvn verify
> > > > > > >>
> > > > > > >> first to get the correct RAT output.  Look inside of pom.xml
> to
> > > see
> > > > > the
> > > > > > >> classes of exceptions we're managing there for RAT.
> > > > > > >>
> > > > > > >> *Source Files:*
> > > > > > >> https://dist.apache.org/repos/dist/dev/incubator/hawq/2.0.0.
> > > > > > >> 0-incubating.RC4
> > > > > > >>
> > > > > > >> *KEYS file containing PGP Keys we use to sign the release:*
> > > > > > >> https://dist.apache.org/repos/dist/dev/incubator/hawq/KEYS
> > > > > > >> (please note that I used new key to sign and my old key is
> going
> > > to
> > > > be
> > > > > > >> revoked)
> > > > > > >>
> > > > > > >> Thanks
> > > > > > >> -Goden
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *Ed Espino*
> > > > > > *esp...@apache.org <esp...@apache.org>*
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Ed Espino*
> > > > *esp...@apache.org <esp...@apache.org>*
> > > >
> > > --
> > > Goden
> > >
> >
>



-- 
Thanks

Hubert Zhang

Re: Re: External scan error: There are more external files (URLs) than primary segments that can read them (COptTasks.cpp:1756)

[Hubert Zhang]

ction Value : 128
>>> GUC : max_prepared_transactions Value : 250
>>> GUC : max_stack_depth Value : 2048
>>> GUC : max_work_mem Value : 1024000
>>> GUC : metadata_cache_testfile Value :
>>> GUC : optimizer Value : on
>>> GUC : optimizer_analyze_root_partition Value : on
>>> GUC : optimizer_minidump Value : onerror
>>> GUC : optimizer_parts_to_force_sort_on_insert Value : 160
>>> GUC : password_encryption Value : on
>>> GUC : password_hash_algorithm Value : MD5
>>> GUC : pljava_classpath Value :
>>> GUC : pljava_release_lingering_savepoints Value : off
>>> GUC : pljava_statement_cache_size Value : 0
>>> GUC : pljava_vmoptions Value :
>>> GUC : port Value : 6432
>>> GUC : pxf_enable_filter_pushdown Value : on
>>> GUC : pxf_enable_locality_optimizations Value : on
>>> GUC : pxf_enable_stat_collection Value : on
>>> GUC : pxf_remote_service_login Value :
>>> GUC : pxf_remote_service_secret Value :
>>> GUC : pxf_service_address Value : localhost:51200
>>> GUC : pxf_stat_max_fragments Value : 100
>>> GUC : random_page_cost Value : 100
>>> GUC : regex_flavor Value : advanced
>>> GUC : runaway_detector_activation_percent Value : 95
>>> GUC : search_path Value : "$user",public
>>> GUC : seg_max_connections Value : 3000
>>> GUC : segment_directory Value :
>>> GUC : seq_page_cost Value : 1
>>> GUC : server_encoding Value : UTF8
>>> GUC : server_ticket_renew_interval Value : 4320
>>> GUC : server_version Value : 8.2.15
>>> GUC : server_version_num Value : 80215
>>> GUC : shared_buffers Value : 4000
>>> GUC : shared_preload_libraries Value :
>>> GUC : ssl Value : off
>>> GUC : ssl_ciphers Value : ALL:!ADH:!LOW:!EXP:!MD5:@STRENGTH
>>> GUC : standard_conforming_strings Value : off
>>> GUC : standby_address_host Value : localhost
>>> GUC : statement_timeout Value : 0
>>> GUC : superuser_reserved_connections Value : 3
>>> GUC : tcp_keepalives_count Value : 9
>>> GUC : tcp_keepalives_idle Value : 7200
>>> GUC : tcp_keepalives_interval Value : 75
>>> GUC : temp_buffers Value : 1024
>>> GUC : TimeZone Value : PRC
>>> GUC : timezone_abbreviations Value : Default
>>> GUC : track_activities Value : on
>>> GUC : track_counts Value : off
>>> GUC : transaction_isolation Value : read committed
>>> GUC : transaction_read_only Value : off
>>> GUC : transform_null_equals Value : off
>>> GUC : unix_socket_directory Value :
>>> GUC : unix_socket_group Value :
>>> GUC : unix_socket_permissions Value : 511
>>> GUC : update_process_title Value : on
>>> GUC : vacuum_cost_delay Value : 0
>>> GUC : vacuum_cost_limit Value : 200
>>> GUC : vacuum_cost_page_dirty Value : 20
>>> GUC : vacuum_cost_page_miss Value : 10
>>> GUC : vacuum_freeze_min_age Value : 1
>>> GUC : work_mem Value : 51200
>>>
>>>
>>>
>>>
>>>
>>> At 2016-09-21 13:00:41, "Vineet Goel" <vvin...@apache.org> wrote:
>>>
>>> Could you please post your SQL DDL statement? How many URLs do you have
>>> in your external table? Also, your HASH dist table - how many buckets are
>>> defined, if any? Are the # of URLs more than the # of buckets or
>>> default_hash_table_bucket_number value? Perhaps you can attach your
>>> hawq-site.xml file as well.
>>>
>>> Also see:
>>> http://hdb.docs.pivotal.io/20/datamgmt/load/g-gpfdist-protocol.html
>>>
>>> Thanks
>>> Vineet
>>>
>>>
>>> On Tue, Sep 20, 2016 at 7:07 PM 来熊 <yin@163.com> wrote:
>>>
>>>> Hi,all:
>>>> I am testing hawq 2.0.0 , and I find a problem like this:
>>>>  I load data from an external table (created using "like target_table"
>>>> statement) ,
>>>> if the target table was distributed by some column(s), it raise this
>>>> error:
>>>>  External scan error: There are more external files (URLs) than primary
>>>> segments that can read them (COptTasks.cpp:1756)
>>>> if the target table was distributed randomly, it works well,
>>>> I don't set any parameter special,does anybody know how to resolve this
>>>> problem?
>>>> thanks a lot.
>>>>
>>>


-- 
Thanks

Hubert Zhang

Re: Co-located Joins & Data Locality in HAWQ

[Hubert Zhang]

Randomly distributed tables make Hawq2.x more elastic: big queries use more
resources, while small queries will use less resources.
But hash distributed table need to use the same number of resource as the
bucket number of table, no matter the query cost is large or small.  As a
result, a scan on a very small table will also need a large number(default
6 * #node) of resources.
 Moreover, HDFS blocks are distributed on different nodes. To achieve
better local disk read ratio, the data locality algorithm assign each block
to the appropriate virtual segment for randomly distributed table. But for
hash distributed table, all the blocks in one file must be processed by one
virtual segment(property of hash table), so we have to assign hash table at
file granularity. If blocks in a file are located on quite different nodes,
data locality ratio would become low, which often happens after cluster
expanding or shrinking.

So even if hash distributed table will reduce some data motion, we still
recommend to use randomly distributed table as default in Hawq2.x.

Thanks
Hubert

On Wed, Sep 21, 2016 at 2:14 PM, Vineet Goel <vvin...@apache.org> wrote:

> Hi all,
>
> I have received a fair number of questions on the topic of handling data
> locality and co-located joins in HAWQ 2. Most of the questions are coming
> from the background where HAWQ 1.x defaulted to HASH distributed tables
> distributed by a key and hence resulted in local joins in most cases for
> better performance.
>
> With the new architecture and RANDOM distribution policy as default, I
> thought it would be good to crowd-source some useful info here from the
> community on how performance is achieved with the new architecture and data
> distribution policy? Questions around how data movement is minimized,
> how/when dynamic redistribution is utilized, how joins are co-located etc.
> Can someone start by providing insights on this topic?
>
> Regards,
> Vineet
>



-- 
Thanks

Hubert Zhang

Re: [VOTE] Apache HAWQ 2.0.0.0-incubating RC3

[Hubert Zhang]

Downloaded and compiled.
 +1.

On Sun, Sep 18, 2016 at 2:10 PM, Chunling Wang <wangchunlin...@126.com>
wrote:

> I have downloaded and compiled the RC3 code successfully. +1.
>
> > 在 2016年9月18日，14:05，Ivan Weng <iw...@pivotal.io> 写道：
> >
> > I have compiled the RC3 code successfully. +1.
> >
> >
> > Regards,
> > Ivan
> >
> > On Sun, Sep 18, 2016 at 1:39 PM, Hong Wu <xunzhang...@gmail.com> wrote:
> >
> >> I have downloaded the release code, and successfully compiled it.
> >>
> >> +1.
> >>
> >> Best
> >> Hong
> >>
> >> 2016-09-16 2:13 GMT+08:00 Goden Yao <goden...@apache.org>:
> >>
> >>> As I haven't seen responses from the community, I'll extend the
> deadline
> >> to
> >>> Friday 12pm Sept 23 2016 PST 
> >>>
> >>> On Mon, Sep 12, 2016 at 5:25 PM Goden Yao <goden...@apache.org> wrote:
> >>>
> >>>> This is the 1st release for Apache HAWQ (incubating), version:
> >>>> 2.0.0.0-incubating
> >>>>
> >>>> *It fixes the following issues:*
> >>>> Clear all IP related issues for HAWQ and this is a source code tarball
> >>>> only release.
> >>>> Full list of JIRAs fixed/related to the release: link
> >>>> <https://cwiki.apache.org/confluence/display/HAWQ/HAWQ+
> >>> Release+2.0.0.0-incubating>
> >>>>
> >>>> *** Please download, review and vote by *Friday 12pm Sep 16, 2016 PST*
> >>> ***
> >>>> or When we have enough votes to bring this source tarball to IPMC
> >>>>
> >>>> *We're voting upon the release branch:*
> >>>> 2.0.0.0-incubating
> >>>> HEAD: commit
> >>>> <https://git-wip-us.apache.org/repos/asf?p=incubator-
> >>> hawq.git;a=commit;h=8eff60336df48c1b6830f21b443313b8a1887047>
> >>>>
> >>>> To run check RAT, please do:
> >>>>
> >>>> $mvn verify
> >>>>
> >>>> first to get the correct RAT output.  Look inside of pom.xml to see
> the
> >>>> classes of exceptions we're managing there for RAT.
> >>>>
> >>>> *Source Files:*
> >>>>
> >>>> https://dist.apache.org/repos/dist/dev/incubator/hawq/2.0.0.
> >>> 0-incubating.RC3
> >>>>
> >>>> *KEYS file containing PGP Keys we use to sign the release:*
> >>>> https://dist.apache.org/repos/dist/dev/incubator/hawq/KEYS
> >>>>
> >>>> Thanks
> >>>> -Goden
> >>>>
> >>>
> >>
>
>
>


-- 
Thanks

Hubert Zhang

[jira] [Created] (HAWQ-1039) Add test case of bucket number may not be consistent with parent table.

[Hubert Zhang (JIRA)]

Hubert Zhang created HAWQ-1039:
--

 Summary: Add test case of bucket number may not be consistent with 
parent table.
 Key: HAWQ-1039
 URL: https://issues.apache.org/jira/browse/HAWQ-1039
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: Core
Reporter: Hubert Zhang
Assignee: Lei Chang


add test case for HAWQ-1032



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Planner invoked twice on master

[Hubert Zhang]

Hi lirong,
Thanks for your sharing the history of the resource negotiator.
But your opinion to "embed the logic into planner" I think is not suitable
for HAWQ.
IMO, The logic in your context is hawq specific and it is the policy to
determine how many resources are
needed for the current query. The policy is application depended, for
example, the resources
are mainly determined by the HDFS block number and hash table bucket number
in HAWQ,
which is not used by other database.
HAWQ use both legacy planner and ORCA,  but ORCA is a general optimizer,
also used by
other MPP database, such as Greenplum.
Embedding hawq specific logic into ORCA is not appropriate.


On Wed, Aug 31, 2016 at 1:14 PM, Hubert Zhang <hzh...@pivotal.io> wrote:

> Hi Kavinder,
> If your problem "multiple REST requests" is really caused by planner
> invoked twice,
> my suggestions is to avoid the related function call(maybe a call to rest
> service) at the
> first plan stage.
>
> You can refers to src/backend/optimizer/util/clauses.c:2834
> using is_in_planning_phase() to determine whether the legacy planner is at
> the first
> or the second stage.
>
> Thanks
>
>
> On Wed, Aug 31, 2016 at 12:31 AM, Lirong Jian <jianlir...@gmail.com>
> wrote:
>
>> Hi Kavinder,
>>
>> If you are looking for some guy "to be blamed" for making the decision
>> of invoking the planner twice on master, unfortunately, that guy would be
>> me, :).
>>
>> First of all, I have left the company (Pivotal Inc.) about haft a year ago
>> and was no longer working on the Apache HAWQ project full time anymore.
>> And
>> I have little time to keep tracking the code changes these days. Thus,
>> what
>> I am going to say is probably inconsistent with the latest code.
>>
>> * Problem*
>>
>> To improve the scalability and throughput of the system, we want to
>> implement the dynamic resource allocation mechanism for HAWQ 2.0.  In
>> other
>> words, we want to allocate exactly needed resource (the number of virtual
>> segments), rather than the fixed amount of resource (like HAWQ 1.x), to
>> run
>> the underlying query. In order to achieve that, we need to calculate the
>> cost of the query before allocating the resource, which means we need to
>> figure out the execution plan first. However, in the framework of the
>> current planner (either the old planner or the new optimizer ORCA),
>> the optimal execution plan is generated given the to-be-run query and the
>> number of segments. Thus, this is a chicken and egg problem.
>>
>> *Solution*
>>
>> IMO, the ideal solution for the above problem is to use an iterative
>> algorithm: given a default number of segments, calculate the optimal
>> execution plan; based on the output optimal execution plan, figure out the
>> appropriate number of segments needed to run this query; and calculate the
>> optimal execution plan again, and again, until the result is stable.
>>
>> *Implementation*
>>
>> In the actual implementation, we set the number of iterations to 2 for two
>> major reasons: (1) two iterations are enough to give out a good result;
>> (2)
>> there is some cost associated with invoking the planner, especially the
>> new
>> optimizer ORCA.
>>
>> After implementing the first version, we later found that determining the
>> number of virtual segments based on the cost of the query sometimes gave
>> out very bad results (although this is the issue of the planner, because
>> the cost of the planner provided doesn't imply the actual running cost of
>> the query correctly). So, borrowing the idea from Hadoop MapReduce, we
>> calculate the cost based on the total size of all tables needed to be
>> scanned for the underlying query. It seemed we don't need to invoke the
>> planner before allocating resource anymore. However, in our current
>> resource manager, the allocated resource is segment-based, not
>> process-based. For example, if an execution plan consists of three slices,
>> meaning we need to setup three processes on each segment to run this
>> query.
>> One allocated resource unit (virtual segment) is for all three processes.
>> In order to avoid the case where too many processes are started on one
>> physical host, we need to know how many processes (the number of slices of
>> the execution plan) are going to start on one virtual segment when we
>> require resource from the resource manager. Thus, the execution plan is
>> still needed. We could write a new function to calculate number of slices
>> of the plan rather than invoking the planner, but after

[jira] [Created] (HAWQ-999) Treat hash table as random when file count is not in proportion to bucket number of table.

[Hubert Zhang (JIRA)]

Hubert Zhang created HAWQ-999:
-

 Summary: Treat hash table as random when file count is not in 
proportion to bucket number of table.
 Key: HAWQ-999
 URL: https://issues.apache.org/jira/browse/HAWQ-999
 Project: Apache HAWQ
  Issue Type: Improvement
  Components: Core
Reporter: Hubert Zhang
Assignee: Lei Chang


By definition, file count of a hash table should be equal to or a multiple of 
the bucket number of the table. So if mismatch happens, we should not treat it 
as hash table in data locality algorithm.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: HAWQ Ranger Integration Design Doc

[Hubert Zhang]

@ruilong
Q1:yes, you can tune the sync interval parameter in conf file of UserSync,
default is 5mins for Unix
Q2:  If Ranger is down, all the queries in HAWQ cannot get privilege and
will be refused. New connections to HAWQ should be refused too.

Re: Jira comments vs PR comments

[Hubert Zhang]

+1. Comments on PR should only be related to the code, not design.

On Thu, Jul 21, 2016 at 1:59 AM, Goden Yao <goden...@apache.org> wrote:

> I filed an infra ticket to disconnect this integration between github PR
> and JIRA comment:
> https://issues.apache.org/jira/browse/INFRA-12310
>
> On Tue, Jul 19, 2016 at 10:31 PM Lili Ma <lil...@apache.org> wrote:
>
> > Agree!  Just keeping the "issue links to" is enough to track the pull
> > request.  Leaving the comments on JIRA there is more neat.
> >
> > Thanks
> > Lili
> >
> > 2016-07-20 10:34 GMT+08:00 Paul Guo <paul...@gmail.com>:
> >
> > > +1.  Github has already have fine-grained notification mechanism. I
> think
> > > JIRA
> > > could just pull the initial git pull request info, not each comment in
> > > github.
> > >
> > > I guess this is either a JIRA configuration issue or an
> > asf/infrustructure
> > > issue.
> > >
> > > 2016-07-20 2:57 GMT+08:00 Goden Yao <goden...@apache.org>:
> > >
> > > > 2nd that. JIRA right now can already link to the PR under "issue
> links
> > > to"
> > > > section. (e.g.: https://issues.apache.org/jira/browse/HAWQ-936 )
> > > > Not sure if that was automatic or manual but I prefer that way
> instead
> > of
> > > > auto-pulling every PR comment into the JIRA (even closing PR is a
> > > comment).
> > > >
> > > > JIRA discussion should be focused on the problem to solve and design,
> > > etc.
> > > > While PR comments are mainly for coding details, logic errors, naming
> > > > issues, formats, coding style etc.
> > > >
> > > > -Goden
> > > >
> > > > On Tue, Jul 19, 2016 at 11:52 AM Shivram Mani <
> shivram.m...@gmail.com>
> > > > wrote:
> > > >
> > > > > Its great that Jira and github has been linked, but this
> integration
> > is
> > > > > also making it hard for us to skim through actual comments posted
> in
> > > > Jira.
> > > > > Jiras usually end up being innundated with every single comment
> > posted
> > > in
> > > > > the pull request.
> > > > > This makes it hard for us to actually read through/search for
> actual
> > > > > comments/discussions posted on the Jira amidst the pull requestion
> > > > > notifications for each comment.
> > > > >
> > > > > Can we disable pull request comment based notifications from being
> > > posted
> > > > > on the Jira ?
> > > > > Thoughts ?
> > > > >
> > > > > --
> > > > > shivram mani
> > > > >
> > > >
> > >
> >
>



-- 
Thanks

Hubert Zhang

How about handle stages with different strategies

[Hubert Zhang]

Hi, all
  In HAWQ, different stages in HAWQ will be treated as the same. no matter
from the scheduler view or consider the number of processes.
  But in some other systems like Presto, There are two schedulers, one is
sourcePartitionedScheduler used to dispatch scan stage, the other is
FixedCountScheduler, used to dispatch intermediate stages.
  I think that one is more flexible. Flexible means that we can write a new
scanScheduler, which dispatches at split level, for some nodes, which are
faster than others, will scan more splits than others. This strategy may
reduce the average IO time.
   Is there any suggestion?

-- 
Thanks

Hubert Zhang

Re: Rename "greenplum" to "hawq"

[Hubert Zhang]

+1 This is a legacy problem. Clear GUCs including "GP" may be the first
step.

On Wed, Jul 13, 2016 at 2:07 PM, Wen Lin <w...@pivotal.io> wrote:

> It's definitely a good thing, but needs very careful thought on effects to
> customers.
> We can try to list these effects as much as possile.
>
> On Wed, Jul 13, 2016 at 1:55 PM, Yi Jin <y...@pivotal.io> wrote:
>
> > I think it is a must-do, but some concerns of customer using convention
> and
> > legacy applications, scripts etc.
> >
> > On Wed, Jul 13, 2016 at 1:44 PM, 陶征霖 <ztao1...@apache.org> wrote:
> >
> > > Good idea, but need quite a lot of effort and may also affect custormer
> > > behavior. Should handle it carefully.
> > >
> > > 2016-07-13 9:54 GMT+08:00 Ivan Weng <iw...@pivotal.io>:
> > >
> > > > Agree with this good idea. But as Paul said, there are maybe already
> > many
> > > > users use greeenplum_path.sh or something else in their environment.
> So
> > > we
> > > > need to think about it.
> > > >
> > > >
> > > > Regards,
> > > > Ivan
> > > >
> > > > On Wed, Jul 13, 2016 at 9:31 AM, Paul Guo <paul...@gmail.com> wrote:
> > > >
> > > > > I've asked this before. Seems that affects some old users. I'm not
> > sure
> > > > > about the details.
> > > > > I agree that we should change it to a better name in a release.
> > > > >
> > > > > 2016-07-13 9:25 GMT+08:00 Roman Shaposhnik <ro...@shaposhnik.org>:
> > > > >
> > > > > > On Tue, Jul 12, 2016 at 6:21 PM, Xiang Sheng <xsh...@pivotal.io>
> > > > wrote:
> > > > > > > Agree . @xunzhang.
> > > > > > > However , some greenplum strings can be easily replaced , but
> > there
> > > > are
> > > > > > too
> > > > > > > many in the code or comments.  Changing all of them costs too
> > much
> > > > > > efforts.
> > > > > > >
> > > > > > > So changing the strings that users can see is enough.
> > > > > >
> > > > > > Huge +1 to this! Btw, is this something we may be able to tackle
> in
> > > our
> > > > > > next Apache release?
> > > > > >
> > > > > > Thanks,
> > > > > > Roman.
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Zhenglin
> > >
> >
>



-- 
Thanks

Hubert Zhang

Re: Question on hawq_rm_nvseg_perquery_limit

[Hubert Zhang]

+1 with Yi's answer.
Vseg numbers are controlled by Resource Negotiator(a module before
planner),  all the vseg related GUCs will affect the behaviour of RN, some
of them will also affect Resource Manager.
To be specific, hawq_rm_nvseg_perquery_limit and
hawq_rm_nvseg_perquery_perseg_limit
are both considered by Resource Negotiator(RN) and Resource Manager(RM),
while default_hash_table_bucket_number is only considered by RN.
As a result, suppose default_hash_table_bucket_number  = 60, query like
"select * from hash_table" will request #60 vsegs in RN and if
hawq_rm_nvseg_perquery_limit
is less than 60, RM will not able to allocate 60 vsegs.

So we need to ensure default_hash_table_bucket_number is less than the
capacity of RM.

On Wed, Jul 13, 2016 at 1:40 PM, Yi Jin <y...@pivotal.io> wrote:

> Hi Vineet,
>
> Some my comment.
>
> For question 1.
> Yes,
> perquery_limit is introduced mainly for restrict resource usage in large
> scale cluster; perquery_perseg_limit is to avoid allocating too many
> processes in one segment, which may cause serious performance issue. So,
> two gucs are for different performance aspects. Along with the variation of
> cluster scale, one of the two limits actually takes effect. We dont have to
> let both active for resource allocation.
>
> For question 2.
>
> In fact, perquery_perseg_limit is a general resource restriction for all
> queries not only hash table queries and external table queries, this is why
> this guc is not merged with another one. For example, when we run some
> queries upon random distributed tables, it does not make sense to let
> resource manager refer a guc for hash table.
>
> For the last topic item.
>
> In my opinion, it is not necessary to adjust hawq_rm_nvseg_perquery_limit,
> say, we just need to leave it unchanged and actually not active until we
> really want to run a large-scale HAWQ cluster, for example, 100+ nodes.
>
> Best,
> Yi
>
> On Wed, Jul 13, 2016 at 1:18 PM, Vineet Goel <vvin...@apache.org> wrote:
>
> > Hi all,
> >
> > I’m trying to document some GUC usage in detail and have questions on
> > hawq_rm_nvseg_perquery_limit and hawq_rm_nvseg_perquery_perseg_limit
> > tuning.
> >
> > *hawq_rm_nvseg_perquery_limit* = (default value = 512) . Let’s call it
> > *perquery_limit* in short.
> > *hawq_rm_nvseg_perquery_perseg_limit* (default value = 6) . Let’s call it
> > *perquery_perseg_limit* in short.
> >
> >
> > 1) Is there ever any benefit in having perquery_limit *greater than*
> > (perquery_perseg_limit * segment host count) ?
> > For example in a 10-node cluster, HAWQ will never allocate more than (GUC
> > default 6 * 10 =) 60 v-segs, so the perquery_limit default of 512 doesn’t
> > have any effect. It seems perquery_limit overrides (takes effect)
> > perquery_perseg_limit only when it’s value is less than
> > (perquery_perseg_limit * segment host count).
> >
> > Is that the correct assumption? That would make sense, as users may want
> to
> > keep a check on how much processing a single query can take up (that
> > implies that the limit must be lower than the total possible v-segs). Or,
> > it may make sense in large clusters (100-nodes or more) where we need to
> > limit the pressure on HDFS.
> >
> >
> > 2) Now, if the purpose of hawq_rm_nvseg_perquery_limit is to keep a check
> > on single query resource usage (by limiting the # of v-segs), doesn’t if
> > affect default_hash_table_bucket_number because queries will fail when
> > *default_hash_table_bucket_number* is greater than
> > hawq_rm_nvseg_perquery_limit ? In that case, the purpose of
> > hawq_rm_nvseg_perquery_limit conflicts with the ability to run queries on
> > HASH dist tables. This then means that tuning
> hawq_rm_nvseg_perquery_limit
> > down is not a good idea, which seems conflicting to the purpose of the
> GUC
> > (in relation to other GUC).
> >
> >
> > Perhaps someone can provide some examples on *how and when would you
> > tune hawq_rm_nvseg_perquery_limit* in this 10-node example:
> >
> > *Defaults on a 10-node cluster are:*
> > a) *hawq_rm_nvseg_perquery_perseg_limit* = 6 (hence ability to spin up 6
> *
> > 10 = 60 total v-segs for random tables)
> > b) *hawq_rm_nvseg_perquery_limit* = 512 (but HAWQ will never dispatch
> more
> > than 60 v-segs on random table, so value of 512 does not seem practical)
> > c) *default_hash_table_bucket_number* = 60 (6 * 10)
> >
> >
> >
> > Thanks
> > Vineet
> >
>



-- 
Thanks

Hubert Zhang

Re: [Propose] New PXF profile optimized for ORC (predicate pushdown)

[Hubert Zhang]

+1 for lazy reader, It can save a lot of decompression and
deserialization(CPU bound) time.

On Wed, Jul 6, 2016 at 7:00 AM, Roman Shaposhnik <ro...@shaposhnik.org>
wrote:

> On Tue, Jul 5, 2016 at 12:01 PM, Shivram Mani <shivram.m...@gmail.com>
> wrote:
> > I've created the following jira HAWQ-866
> > <https://issues.apache.org/jira/browse/HAWQ-886> which is focussed on
> > improving/enhancing the existing PXF profile to read ORC files. The goal
> is
> > to make use of the underlying ORC reader's capability of supporting
> > predicate push-down among others.
> >
> > Presto has also contributed an alternative ORC reader which provides both
> > predicate push down and Lazy reads
> >
> https://code.facebook.com/posts/370832626374903/even-faster-data-at-the-speed-of-presto-orc/
> > .
> >
> > Will be evaluating both the options as part of this effort.
>
> Great to see this effort! Do you plan to come up with any kind of
> benchmark to
> be able to compare the native ORC reader vs. PXF ORC reader performance
> and capabilities?
>
> Or does it really all just boil down to TPC?
>
> Thanks,
> Roman.
>



-- 
Thanks

Hubert Zhang

[jira] [Created] (HAWQ-887) Query should be continued when bucket number of result hash relation is bigger than the on number of command external table.

[Hubert Zhang (JIRA)]

Hubert Zhang created HAWQ-887:
-

 Summary: Query should be continued when bucket number of result 
hash relation is bigger than the on number of command external table.
 Key: HAWQ-887
 URL: https://issues.apache.org/jira/browse/HAWQ-887
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Core
Reporter: Hubert Zhang
Assignee: Lei Chang


When user create a hash table test with bucket number 6, and a command external 
table e_test with on number 1, 
query likes "insert into test select * from e_test" will be refused due to 
bucket number and on number are not equal.
This restriction can be relaxed, if bucket number is bigger than on number, the 
queries should be OK, and no wrong answer will be generated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: elastic execution engine question

[Hubert Zhang]

This GUC hawq_rm_stmt_nvseg is used by resource manager.
Yi, could you please give some detailed information?

On Fri, Jul 1, 2016 at 12:48 PM, Vineet Goel <vg...@pivotal.io> wrote:

> What about hawq_rm_stmt_nvseg ?
>
>
> http://hdb.docs.pivotal.io/20/reference/guc/parameter_definitions.html#hawq_rm_stmt_nvseg
>
> -Vineet
>
>
>
> On Jun 30, 2016, at 6:43 PM, Hubert Zhang <hzh...@pivotal.io> wrote:
>
> You can try to the GUC
>
> enforce_virtual_segment_number.
>
> It works for queries without UDF.
>
> On Fri, Jul 1, 2016 at 9:40 AM, yanqing weng <wengyanq...@gmail.com>
> wrote:
>
> > Thanks Hubert.
> >
> > Can it be set only use one virtual segment for the query ? Because this
> > can debug query more easily.
> >
> >
> > On Fri, Jul 1, 2016 at 9:35 AM, Hubert Zhang <hzh...@pivotal.io> wrote:
> >
> >> For hash tables, The number of virtual segments is equal to the bucket
> >> number of the table. Bucket number can be specified when creating the
> table.
> >>
> >> On Fri, Jul 1, 2016 at 9:31 AM, Ivan Weng <iw...@pivotal.io> wrote:
> >>
> >>> How about hash table?
> >>>
> >>> On Fri, Jul 1, 2016 at 9:30 AM, Hubert Zhang <hzh...@pivotal.io>
> wrote:
> >>>
> >>>> For random table, the virtual segment number is related to the data
> >>> size of
> >>>> the table. For large tables, we prefer use more virtual segments to
> >>> achieve
> >>>> better performance. While for small tables, less virtual segments are
> >>>> needed, which could save resources and enable more concurrent queries.
> >>>
> >>>>
> >>>> On Fri, Jul 1, 2016 at 9:22 AM, yanqing weng <wengyanq...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Hi HAWQ developer,
> >>>>>
> >>>>> Elastic execution engine is the important feature in HAWQ 2.0.
> >>>>>
> >>>>> I have a question for this feature in the HAWQ debugging. Is there
> >>>> possible
> >>>>> to control the virtual segment number when run query ? If so, which
> >>> GUC
> >>>> or
> >>>>> parameter I should set ? Thanks.
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Thanks
> >>>>
> >>>> Hubert Zhang
> >>>>
> >>>
> >>
> >>
> >>
> >> --
> >> Thanks
> >>
> >> Hubert Zhang
> >>
> >
> >
>
>
> --
> Thanks
>
> Hubert Zhang
>
>


-- 
Thanks

Hubert Zhang

Re: elastic execution engine question

[Hubert Zhang]

You can try to the GUC

enforce_virtual_segment_number.

It works for queries without UDF.

On Fri, Jul 1, 2016 at 9:40 AM, yanqing weng <wengyanq...@gmail.com> wrote:

> Thanks Hubert.
>
> Can it be set only use one virtual segment for the query ? Because this
> can debug query more easily.
>
>
> On Fri, Jul 1, 2016 at 9:35 AM, Hubert Zhang <hzh...@pivotal.io> wrote:
>
>> For hash tables, The number of virtual segments is equal to the bucket
>> number of the table. Bucket number can be specified when creating the table.
>>
>> On Fri, Jul 1, 2016 at 9:31 AM, Ivan Weng <iw...@pivotal.io> wrote:
>>
>>> How about hash table?
>>>
>>> On Fri, Jul 1, 2016 at 9:30 AM, Hubert Zhang <hzh...@pivotal.io> wrote:
>>>
>>> > For random table, the virtual segment number is related to the data
>>> size of
>>> > the table. For large tables, we prefer use more virtual segments to
>>> achieve
>>> > better performance. While for small tables, less virtual segments are
>>> > needed, which could save resources and enable more concurrent queries.
>>>
>>> >
>>> > On Fri, Jul 1, 2016 at 9:22 AM, yanqing weng <wengyanq...@gmail.com>
>>> > wrote:
>>> >
>>> > > Hi HAWQ developer,
>>> > >
>>> > > Elastic execution engine is the important feature in HAWQ 2.0.
>>> > >
>>> > > I have a question for this feature in the HAWQ debugging. Is there
>>> > possible
>>> > > to control the virtual segment number when run query ? If so, which
>>> GUC
>>> > or
>>> > > parameter I should set ? Thanks.
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > Thanks
>>> >
>>> > Hubert Zhang
>>> >
>>>
>>
>>
>>
>> --
>> Thanks
>>
>> Hubert Zhang
>>
>
>


-- 
Thanks

Hubert Zhang

Re: elastic execution engine question

[Hubert Zhang]

For hash tables, The number of virtual segments is equal to the bucket
number of the table. Bucket number can be specified when creating the table.

On Fri, Jul 1, 2016 at 9:31 AM, Ivan Weng <iw...@pivotal.io> wrote:

> How about hash table?
>
> On Fri, Jul 1, 2016 at 9:30 AM, Hubert Zhang <hzh...@pivotal.io> wrote:
>
> > For random table, the virtual segment number is related to the data size
> of
> > the table. For large tables, we prefer use more virtual segments to
> achieve
> > better performance. While for small tables, less virtual segments are
> > needed, which could save resources and enable more concurrent queries.
> >
> > On Fri, Jul 1, 2016 at 9:22 AM, yanqing weng <wengyanq...@gmail.com>
> > wrote:
> >
> > > Hi HAWQ developer,
> > >
> > > Elastic execution engine is the important feature in HAWQ 2.0.
> > >
> > > I have a question for this feature in the HAWQ debugging. Is there
> > possible
> > > to control the virtual segment number when run query ? If so, which GUC
> > or
> > > parameter I should set ? Thanks.
> > >
> >
> >
> >
> > --
> > Thanks
> >
> > Hubert Zhang
> >
>



-- 
Thanks

Hubert Zhang

Re: elastic execution engine question

[Hubert Zhang]

For random table, the virtual segment number is related to the data size of
the table. For large tables, we prefer use more virtual segments to achieve
better performance. While for small tables, less virtual segments are
needed, which could save resources and enable more concurrent queries.

On Fri, Jul 1, 2016 at 9:22 AM, yanqing weng <wengyanq...@gmail.com> wrote:

> Hi HAWQ developer,
>
> Elastic execution engine is the important feature in HAWQ 2.0.
>
> I have a question for this feature in the HAWQ debugging. Is there possible
> to control the virtual segment number when run query ? If so, which GUC or
> parameter I should set ? Thanks.
>



-- 
Thanks

Hubert Zhang

Re: Can we create one wiki page for FAQ?

[Hubert Zhang]

Good Jobs!  Ming
I suggest to divide the developer FAQ and user FAQ into two independent
pages or add an index at the begining of the FAQ page.

Thanks.

On Tue, Jun 28, 2016 at 12:35 PM, Ming Li <m...@pivotal.io> wrote:

> Good idea.
>
> (1) Added one FAQ page here:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144284
> (2) This link with other common used links also added to README.md (
>
> https://github.com/apache/incubator-hawq/commit/5a8622707749cfe98c61188ab212a61c83778998
> )
>
> So let's continue to add FAQ in that page or move previous.  ~..~
>
>
> On Fri, Jun 24, 2016 at 5:31 PM, Guo Gang <paul...@gmail.com> wrote:
>
> > I agree that we need a separate FAQ page on apache. Several other
> > suggestions:
> >
> > 1) Add the FAQ info and url to readme.md to educate users to search FAQ
> > before asking.
> > 2) Announce it on user mail list when it is done.
> > 3) Move some previous installation related FAQ to the new FAQ page.
> >
> > 2016-06-24 17:01 GMT+08:00 Lili Ma <l...@pivotal.io>:
> >
> > > Thumbs-up!
> > >
> > > We can put some common questions and answers on the wiki page, so that
> > > users can easily find solutions for their problem.
> > >
> > > https://cwiki.apache.org/confluence/display/HAWQ/Apache+HAWQ+Home
> > >
> > > This page already exist some useful informations, including HAWQ
> > > introduction, Build HAWQ, contributing to HAWQ, HAWQ RoadMap, etc. I
> > think
> > > we can add one sub-page named FAQ for it.
> > >
> > > What do you think?
> > >
> > >
> > > Thanks
> > > Lili
> > >
> > > On Fri, Jun 24, 2016 at 3:35 PM, Greg Chase <gch...@gmail.com> wrote:
> > >
> > > > The Wiki should serve that function. An FAQ might be a page or it
> might
> > > be
> > > > the structure.
> > > >
> > > > The Wiki is more accessible than the Github for this.
> > > >
> > > > This email encrypted by tiny buttons & fat thumbs, beta voice
> > > recognition,
> > > > and autocorrect on my iPhone.
> > > >
> > > > > On Jun 24, 2016, at 12:21 AM, hong wu <xunzhang...@gmail.com>
> wrote:
> > > > >
> > > > > Great point ! We really need this place to find everything out.
> > > > >
> > > > > BTW, should this kind of wiki or document belongs to the HAWQ repo
> > > which
> > > > > could record every change with git commits, not just a link. I
> think
> > > the
> > > > > benefit is to sync with code changes timely.
> > > > >
> > > > > Best
> > > > > xunzhang
> > > > >
> > > > > 2016-06-24 14:56 GMT+08:00 Ming Li <m...@pivotal.io>:
> > > > >
> > > > >> Hi all,
> > > > >>
> > > > >> We have many sources about hawq discussions ( jira /mail list/
> > > > >> stackoverflow), however for many questions, if we have definite
> > > > solution, I
> > > > >> would like we can offer an official answer, and it is better to
> > > > maintain in
> > > > >> one place, so that we can maintain and search them quickly.
> > > > >>
> > > > >> So I suggest to create one wiki page for it, so that everyone can
> > > > summary
> > > > >> any FAQ by hands, and put them on that.
> > > > >>
> > > > >> Any opinions/suggestions is welcomed. Thanks.
> > > > >>
> > > >
> > >
> >
>



-- 
Thanks

Hubert Zhang

[jira] [Created] (HAWQ-862) Make user defined function get_ao_distribution work.

[Hubert Zhang (JIRA)]

Hubert Zhang created HAWQ-862:
-

 Summary: Make user defined function get_ao_distribution work.
 Key: HAWQ-862
 URL: https://issues.apache.org/jira/browse/HAWQ-862
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Core
Reporter: Hubert Zhang
Assignee: Lei Chang


In Hawq we had a functon that could do a fast row-count on AO tables based on 
querying the catalog versus issuing 'select count(*)' SQL.  Was a lot faster 
for larger tables.  I found the script and tried using it with HAWQ but one of 
the underlying Postgres functions that gets called (get_ao_distribution) bombs 
since it doesn't seem to be able to find the table files when they are stored 
in HDFS.

gpadmin=# select sum(tupcount) from get_ao_distribution('public.foo');
ERROR:  could not open relation 1663/24731/24740: No such file or directory 
(seg0 localhost.localdomain:4 pid=82964)
DETAIL:   Database directory "base/24731" does not exist
CONTEXT:  SQL statement "select gp_segment_id, sum(tupcount) from 
gp_dist_random('pg_aoseg.pg_aoseg_24738') group by (gp_segment_id)"
gpadmin=#



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-861) Wrong logerror position in insert into hashtable select * from gpfdist external table.

[Hubert Zhang (JIRA)]

Hubert Zhang created HAWQ-861:
-

 Summary: Wrong logerror position in insert into hashtable select * 
from gpfdist external table.
 Key: HAWQ-861
 URL: https://issues.apache.org/jira/browse/HAWQ-861
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Core
Reporter: Hubert Zhang
Assignee: Lei Chang


For statement insert into hashtable select * from gpfdist external table,
when hashtable bucket number is less than location number of gpfdist external 
table, error should be reported in cdbdatalocality module, with error message 
like this:
ERROR:  Could not allocate enough memory! bucket number of result hash table 
and external table should match each other (cdbdatalocality.c:4222)




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-850) Planner supports refineCachedPlan interface.

[Hubert Zhang (JIRA)]

Hubert Zhang created HAWQ-850:
-

 Summary: Planner supports refineCachedPlan interface.
 Key: HAWQ-850
 URL: https://issues.apache.org/jira/browse/HAWQ-850
 Project: Apache HAWQ
  Issue Type: Improvement
  Components: Core
Reporter: Hubert Zhang
Assignee: Lei Chang


in PBE(prepare bind execution) cases, plan will be cached, and only be 
generated once in postgres.
But in HAWQ since resource is dynamic or elastic, the plan maybe changed during 
the different execution of PBE. Also, the file split allocation result which is 
stored in plan is dynamically too, so the split-vseg mapping may also be 
changed.

For example, We call a plpgsql function which do "insert into t select * from 
t".
in Prepare we cache the plan, with fixed split-vseg mapping(e.g. 10 blocks), 
but after insert, there would be 20 blocks so the old  split-vseg mapping is 
invalid. We need to update it. Also when we call this function again and again, 
the data size of t may request more resource to handle it than the first call, 
so we should recalculate the entire plan with the new vseg number.

This function is implemented in a interface called refineCachedPlan.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-827) Refactor aggregate_with_groupingsets checkinstall cases using new test framework.

[Hubert Zhang (JIRA)]

Hubert Zhang created HAWQ-827:
-

 Summary: Refactor aggregate_with_groupingsets checkinstall cases 
using new test framework.
 Key: HAWQ-827
 URL: https://issues.apache.org/jira/browse/HAWQ-827
 Project: Apache HAWQ
  Issue Type: Test
  Components: Tests
Reporter: Hubert Zhang
Assignee: Jiali Yao


refactor installcheck-good test



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-818) Add order by for aggregate feature test.

[Hubert Zhang (JIRA)]

Hubert Zhang created HAWQ-818:
-

 Summary: Add order by for aggregate feature test.
 Key: HAWQ-818
 URL: https://issues.apache.org/jira/browse/HAWQ-818
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Tests
Reporter: Hubert Zhang
Assignee: Jiali Yao


feature test should add order by to achieve exact match



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-803) Enable filter in code coverage.

[Hubert Zhang (JIRA)]

Hubert Zhang created HAWQ-803:
-

 Summary: Enable filter in code coverage.
 Key: HAWQ-803
 URL: https://issues.apache.org/jira/browse/HAWQ-803
 Project: Apache HAWQ
  Issue Type: Test
  Components: Tests
Reporter: Hubert Zhang
Assignee: Jiali Yao


For code coverage is time consuming,  we want to filter some sub directories 
which we care about.
Syntax:
make coverage-show filter=./src/backend/executor



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-801) Refactor create_aggregate checkinstall cases using new test framework

[Hubert Zhang (JIRA)]

Hubert Zhang created HAWQ-801:
-

 Summary: Refactor create_aggregate checkinstall cases using new 
test framework
 Key: HAWQ-801
 URL: https://issues.apache.org/jira/browse/HAWQ-801
 Project: Apache HAWQ
  Issue Type: Test
  Components: Tests
Reporter: Hubert Zhang
Assignee: Jiali Yao


refactor the create_aggregate test cases



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: how to debug hawq segment

[Hubert Zhang]

Hi Jin Tao,
   In hawq, QE(query executor) will be reused, so that after running the
first query, you can debug QE in the following queries.
   Your method works on debugging QE as well.

Thanks
Huan Zhang


On Fri, Feb 26, 2016 at 3:27 PM, jin tao  wrote:

> hi folks,
>
> I'm now reading the source code and tried to review and debug some issues.
>
> I chose issue HAWQ-442 ,and
> using gdb to debug the code.
> my procedure is :
> 1.open a psql  and using select pg_backend_pid() to get the pid,
> 2.open another terminal  and using gdb postgres #pid# to debug the program.
> I could use backtrace to know the process of function exected,like:
> (gdb) bt
> #0  0x7fa8623ab5bb in recv () from /lib64/libpthread.so.0
> #1  0x006a8473 in secure_read ()
> #2  0x006b06d6 in pq_recvbuf ()
> #3  0x006b12c5 in pq_getbyte ()
> #4  0x007d350c in SocketBackend ()
> #5  0x007d9425 in PostgresMain ()
> #6  0x0078bf65 in ServerLoop ()
> #7  0x0078ec59 in PostmasterMain ()
> #8  0x004a16bf in main ()
>
> It seems only waht happened in the master.How could I trace and debug each
> segement and konw the process of function called?
>
> Many thanks!
>
> Best regards!
> jin tao
>

Re: [VOTE] HAWQ 2.0.0-beta-incubating RC4

[Hubert Zhang]

Downloaded and reviewed
+1

On Tue, Jan 26, 2016 at 2:20 PM, 陶征霖  wrote:

> +1
> Downloaded, deployed and tested
>
> 2016-01-26 13:35 GMT+08:00 Yi Jin :
>
> > +1
> >
> > What I have done:
> > 1) Downloaded, deployed and tested the project.
> > 2) Reviewed LICENSE, COPYRIGHT, DISCLAIMER and NOTICE files.
> >
> > Best,
> > Yi Jin
> >
> > On Tue, Jan 26, 2016 at 4:14 PM, Ruilong Huo  wrote:
> >
> > > Downloaded, deployed, tested and reviewed it. +1
> > >
> > > Best regards,
> > > Ruilong Huo
> > >
> > > On Tue, Jan 26, 2016 at 12:41 PM, Lei Chang 
> > wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > Cheers
> > > > Lei
> > > >
> > > >
> > > > On Tue, Jan 26, 2016 at 6:33 AM, Roman Shaposhnik <
> > ro...@shaposhnik.org>
> > > > wrote:
> > > >
> > > > > Lei, can you please explicitly cast +1/-1 vote?
> > > > >
> > > > > Thanks,
> > > > > Roman.
> > > > >
> > > > > On Sun, Jan 24, 2016 at 7:50 PM, Lei Chang 
> > > wrote:
> > > > > > Look good!
> > > > > >
> > > > > > Run mvn verify, get:
> > > > > >
> > > > > > *
> > > > > >
> > > > > > Summary
> > > > > >
> > > > > > ---
> > > > > >
> > > > > > Generated at: 2016-01-24T19:50:43-08:00
> > > > > >
> > > > > > Notes: 5
> > > > > >
> > > > > > Binaries: 0
> > > > > >
> > > > > > Archives: 0
> > > > > >
> > > > > > Standards: 2627
> > > > > >
> > > > > > Apache Licensed: 1796
> > > > > >
> > > > > > Generated Documents: 0
> > > > > >
> > > > > > JavaDocs are generated and so license header is optional
> > > > > >
> > > > > > Generated files do not required license headers
> > > > > >
> > > > > > 0 Unknown Licenses
> > > > > >
> > > > > > ***
> > > > > >
> > > > > > Unapproved licenses:
> > > > > >
> > > > > > ***
> > > > > >
> > > > > > Cheers
> > > > > >
> > > > > > Lei Chang
> > > > > >
> > > > > > On Sat, Jan 23, 2016 at 9:14 AM, Roman Shaposhnik <
> > > > ro...@shaposhnik.org>
> > > > > > wrote:
> > > > > >
> > > > > >> On Fri, Jan 22, 2016 at 11:20 AM, Caleb Welton <
> > cwel...@pivotal.io>
> > > > > wrote:
> > > > > >> > First question for me is:
> > > > > >> > - We have a couple existing jiras on IP clearance, including
> > > > > >> >   - https://issues.apache.org/jira/browse/HAWQ-184
> > > > > >> >   - https://issues.apache.org/jira/browse/HAWQ-207
> > > > > >> >
> > > > > >> > In particular if HAWQ-184 has *not* been resolved then how are
> > we
> > > > > clear
> > > > > >> of
> > > > > >> > our IP issues?  I see that there was a commit associated with
> > the
> > > > > issue,
> > > > > >> so
> > > > > >> > perhaps this is just a lack of jira hygiene?
> > > > > >>
> > > > > >> I wasn't too eager to resolve 184 until the vote passes,
> > > > > >> as for 207 -- that ended up being fixed in a different way
> > > > > >> and I resolve it as not applicable anymore.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Roman.
> > > > > >>
> > > > >
> > > >
> > >
> >
>
>
>
> --
> 陶征霖
> Pivotal
>