Re: [DISCUSS] Graduate Apache HAWQ (incubating) as a TLP

[Wen Lin]

and hereby is created, the person holding such office to
>serve at the direction of the Board of Directors as the chair
>of the Apache HAWQ Project, and to have primary responsibility
>for management of the projects within the scope of
>responsibility of the Apache HAWQ Project; and be it further
>
>RESOLVED, that the persons listed immediately below be and
>hereby are appointed to serve as the initial members of the
>Apache HAWQ Project:
>
> * Alan Gates   
> * Alexander Denissov   
> * Amy Bai  
> * Atri Sharma  
> * Bhuvnesh Chaudhary   
> * Bosco
> * Chunling Wang
> * David Yozie  
> * Ed Espino
> * Entong Shen  
> * Foyzur Rahman
> * Goden Yao
> * Gregory Chase
> * Hong Wu  
> * Hongxu Ma
> * Hubert Zhang 
> * Ivan Weng
> * Jesse Zhang  
> * Jiali Yao
> * Jun Aoki 
> * Kavinder Dhaliwal
> * Lav Jain 
> * Lei Chang
> * Lili Ma  
> * Lirong Jian  
> * Lisa Owen
> * Ming Li  
> * Mohamed Soliman  
> * Newton Alex  
> * Noa Horn 
> * Oleksandr Diachenko  
> * Paul Guo 
> * Radar Da Lei 
> * Roman Shaposhnik 
> * Ruilong Huo  
> * Shivram Mani 
> * Shubham Sharma   
> * Tushar Pednekar  
> * Venkatesh Raghavan   
> * Vineet Goel  
> * Wen Lin  
> * Xiang Sheng  
> * Yi Jin   
> * Zhanwei Wang 
> * Zhenglin Tao 
>
>NOW, THEREFORE, BE IT FURTHER RESOLVED, that Lei Chang
>be appointed to the office of Vice President, Apache HAWQ, to
>serve in accordance with and subject to the direction of the
>Board of Directors and the Bylaws of the Foundation until
>death, resignation, retirement, removal or disqualification,
>or until a successor is appointed; and be it further
>
>RESOLVED, that the initial Apache HAWQ PMC be and hereby is
>tasked with the creation of a set of bylaws intended to
>encourage open development and increased participation in the
>Apache HAWQ Project; and be it further
>
>RESOLVED, that the Apache HAWQ Project be and hereby
>is tasked with the migration and rationalization of the Apache
>Incubator HAWQ podling; and be it further
>
>RESOLVED, that all responsibilities pertaining to the Apache
>Incubator HAWQ podling encumbered upon the Apache Incubator
>Project are hereafter discharged.
>

Re: Remain with HAWQ project or not?

[Wen Lin]

Yes. I'd like to keep on contributing to HAWQ.

On Mon, May 7, 2018 at 5:21 PM, Ivan Weng  wrote:

> Yes, I definitely would like to be with HAWQ.
>
> Regards,
> Ivan
>
> On Mon, May 7, 2018 at 5:12 PM, Hongxu Ma  wrote:
>
> > Yes, let's make HAWQ better.
> >
> > Thanks.
> >
> > 在 07/05/2018 16:11, Radar Lei 写道:
> > > HAWQ committers,
> > >
> > > Per the discussion in "Apache HAWQ graduation from incubator?" [1], we
> > want
> > > to setup the PMC as part of HAWQ graduation resolution.
> > >
> > > So we'd like to confirm whether you want to remain as a committer/PMC
> > > member of Apache HAWQ project?
> > >
> > > If you'd like to remain with HAWQ project, it's welcome and please
> > *respond**
> > > 'Yes'* in this thread, or *respond 'No'* if you are not interested in
> any
> > > more. Thanks.
> > >
> > > This thread will be available for at least 72 hours, after that, we
> will
> > > send individual confirm emails.
> > >
> > > [1]
> > > https://lists.apache.org/thread.html/b4a0b5671ce377b3d51c9b7ab00496
> > a1eebfcbf1696ce8b67e078c64@%3Cdev.hawq.apache.org%3E
> > >
> > > Regards,
> > > Radar
> > >
> >
> > --
> > Regards,
> > Hongxu.
> >
> >
>

Re: [DISCUSS] Apache HAWQ Graduation from Incubator

[Wen Lin]

I think HAWQ is in a very healthy status.
I am looking forward to HAWQ's graduation.

On Thu, Apr 26, 2018 at 10:16 AM, Radar Lei  wrote:

> Hi All,
>
> With the Apache HAWQ community keep healthy and growing, I think now would
> be a good time to discuss where we stand regarding to graduation from the
> Incubator, and what requirements remains.
>
> Apache HAWQ entered incubation in September of 2015, since then, the HAWQ
> community learned a lot about how to do things in Apache ways. Now we are a
> very helpful and engaged community, ready to help on all questions
> from the HAWQ
> community. We delivered four releases including two binary releases, now we
> can do self-driving releases in good cadence. The community is growing,
>  new committers and PPMC members keep joining. We addressed the maturity
> issues one by one followed by Apache Project Maturity Model [1], currently
> all
> the License and IP issues we can see got resolved.
>
> Base on those, I believes it's time for us to graduate to TLP. [2]
>
> Any thoughts? And welcome advice from HAWQ Mentors?
>
> Thanks,
>
> [1] https://cwiki.apache.org/confluence/display/HAWQ/ASF+
> Maturity+Evaluation
> [2] https://incubator.apache.org/guides/graduation.html#grad
> uating_to_a_top_level_project
>
> Regards,
> Radar
>

Design of Runtime Filter for HAWQ

[Wen Lin]

Hi, All,

A draft design of runtime filter for HAWQ has been uploaded to jira
HAWQ-1597.
https://issues.apache.org/jira/browse/HAWQ-1597
Please feel free to review it and give any suggestion.

Thanks!

Re: Re: [VOTE]: Apache HAWQ 2.3.0.0-incubating Release (RC2)

[Wen Lin]

Build from source, installed and run feature tests.
+1

On Sat, Mar 3, 2018 at 11:02 AM, Yi JIN  wrote:

> Guys, this is a reminder, please vote asap. Thanks
>
> Best,
> Yi (yjin)
>
> On Fri, Mar 2, 2018 at 1:07 PM, Hubert Zhang  wrote:
>
> > +1
> > Build and Installed. Tests passed.
> >
> > On Thu, Mar 1, 2018 at 2:01 PM, Ruilong Huo  wrote:
> >
> > > +1 for the HAWQ 2.3.0.0-incubating RC2
> > >
> > >
> > > Here are the checks have been done:
> > >
> > >
> > > 1. Reviewed LICENSE, NOTICE, DISCLAIMER, and pom.xml.
> > >
> > >
> > > 2. Passed RAT configuration check successfully.
> > >
> > >
> > > 3. Passed source and rpm package signature, md5 and sha256 checksum.
> > >
> > >
> > > 4. Compiled from source tarball and installed RC2 with feature check
> > > successful.
> > >
> > >
> > > 5. Downloaded rpm tarball, installed hawq on CentOS 7.2 VM following
> > > https://cwiki.apache.org/confluence/display/HAWQ/Build+
> > > Package+and+Install+with+RPM. The initialization and basic query
> passed.
> > >
> > >
> > > Best regards,
> > > Ruilong Huo
> > > At 2018-03-01 13:50:58, "Bai Jie"  wrote:
> > > >Build from branch 2.3.0.0 source code, installed , init , run feature
> > test
> > > >and simple queries. Looks good to me. +1
> > > >
> > > >On Wed, Feb 28, 2018 at 11:24 AM, Hongxu Ma 
> > wrote:
> > > >
> > > >> +1
> > > >>
> > > >> Both source and rpm package are verified in my environment: Red Hat
> > > >> Enterprise Linux Server release 7.2
> > > >> Include installation and execute some simple queries.
> > > >>
> > > >> 在 27/02/2018 12:19, Yi JIN 写道:
> > > >> > Hi All,
> > > >> >
> > > >> > This is the vote for Apache HAWQ (incubating) 2.3.0.0-incubating
> > > Release
> > > >> > Candidate 2 (RC2). It is a source release for HAWQ core, PXF, and
> > > Ranger;
> > > >> > and binary release for HAWQ core,  PXF and Ranger. We have rpm
> > package
> > > >> > involved for the binary release.
> > > >> >
> > > >> > The vote will run for at least 72 hours and will close on
> Saturday,
> > > March
> > > >> > 3rd, 2017. Thanks.
> > > >> >
> > > >> > 1. Wiki page of the release:
> > > >> > *https://cwiki.apache.org/confluence/display/HAWQ/
> > > Apache+HAWQ+2.3.0.0-
> > > >> incubating+Release
> > > >> >  > > Apache+HAWQ+2.3.0.0-
> > > >> incubating+Release>*
> > > >> >
> > > >> >
> > > >> > 2. Release Notes (Apache Jira generated):
> > > >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > > >> > version=12340262=Html=12318826
> > > >> >
> > > >> >
> > > >> > 3. Release verification steps can be found at:
> > > >> > For source tarball: https://cwiki.apache.org/
> > confluence/display/HAWQ/
> > > >> > Release+Process%3A+Step+by+step+guide#ReleaseProcess:
> > > >> > Stepbystepguide-ValidatetheReleaseCandidate
> > > >> > For rpm package: https://cwiki.apache.org/
> confluence/display/HAWQ/
> > > >> > Build+Package+and+Install+with+RPM
> > > >> >
> > > >> >
> > > >> > 4. Git release branch:
> > > >> > https://git-wip-us.apache.org/repos/asf?p=incubator-hawq.
> > > >> > git;a=shortlog;h=refs/heads/2.3.0.0-incubating
> > > >> >
> > > >> > 5. Source and Binary release balls with signare:
> > > >> > https://dist.apache.org/repos/dist/dev/incubator/hawq/2.3.0.
> > > >> > 0-incubating.RC2/
> > > >> >
> > > >> >
> > > >> > 6. Keys to verify the signature of the release artifact are
> > available
> > > at:
> > > >> > https://dist.apache.org/repos/dist/dev/incubator/hawq/KEYS
> > > >> >
> > > >> >
> > > >> > 7. The artifact(s) has been signed with Key ID: CE60F90D1333092A
> > > >> >
> > > >> > 8. Fixed issues in RC2.
> > > >> > https://issues.apache.org/jira/browse/HAWQ-1589
> > > >> > https://issues.apache.org/jira/browse/HAWQ-1590
> > > >> >
> > > >> > REMINDER: Please provide details of what you have tried and
> verified
> > > >> before
> > > >> > your vote conclusion. Thanks!
> > > >> >
> > > >> >
> > > >> > Please vote accordingly:
> > > >> > [ ] +1 approve
> > > >> > [ ] +0 no opinion
> > > >> > [ ] -1 disapprove (and reason why)
> > > >> >
> > > >> >
> > > >> > Best regards,
> > > >> > Yi (yjin)
> > > >> >
> > > >>
> > > >> --
> > > >> Regards,
> > > >> Hongxu.
> > > >>
> > > >>
> > >
> >
> >
> >
> > --
> > Thanks
> >
> > Hubert Zhang
> >
>

Re: [VOTE]: Apache HAWQ 2.3.0.0-incubating Release (RC1)

[Wen Lin]

Compile from source, install and run feature tests.
+1.

On Tue, Feb 20, 2018 at 11:33 AM, Hong  wrote:

> +1
>
> 2018-02-19 21:42 GMT-05:00 Yi JIN :
>
> > Hi All,
> >
> > This is the vote for Apache HAWQ (incubating) 2.3.0.0-incubating Release
> > Candidate 1 (RC1). It is a source release for HAWQ core, PXF, and Ranger;
> > and binary release for HAWQ core,  PXF and Ranger. We have rpm package
> > involved for the binary release.
> >
> > The vote will run for at least 72 hours and will close on Saturday, Feb
> 24,
> > 2017. Thanks.
> >
> > 1. Wiki page of the release:
> > *https://cwiki.apache.org/confluence/display/HAWQ/Apache+HAWQ+2.3.0.0-
> > incubating+Release
> >  > incubating+Release>*
> >
> >
> > 2. Release Notes (Apache Jira generated):
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > version=12340262=Html=12318826
> >
> >
> > 3. Release verification steps can be found at:
> > For source tarball: https://cwiki.apache.org/confluence/display/HAWQ
> > /Release+Process%3A+Step+by+step+guide#ReleaseProcess:Stepbystepguide-
> > ValidatetheReleaseCandidate
> > For rpm package: https://cwiki.apache.org/confluence/display/HAWQ
> > /Build+Package+and+Install+with+RPM
> >
> >
> > 4. Git release branch:
> > https://git-wip-us.apache.org/repos/asf?p=incubator-hawq.
> > git;a=shortlog;h=refs/heads/2.3.0.0-incubating
> >
> > 5. Source and Binary release balls with signare:
> > https://dist.apache.org/repos/dist/dev/incubator/hawq/2.3.0.
> > 0-incubating.RC1/
> >
> >
> > 6. Keys to verify the signature of the release artifact are available at:
> > https://dist.apache.org/repos/dist/dev/incubator/hawq/KEYS
> >
> >
> > 7. The artifact(s) has been signed with Key ID: CE60F90D1333092A
> >
> >
> > Please vote accordingly:
> > [ ] +1 approve
> > [ ] +0 no opinion
> > [ ] -1 disapprove (and reason why)
> >
> >
> > Best regards,
> > Yi (yjin)
> >
>

Re: Apache HAWQ 2.3.0.0-incubating release work starts

[Wen Lin]

That's great. Thanks for organizing this release!

On Tue, Jan 30, 2018 at 6:53 PM, Yi JIN  wrote:

> Hi all,
>
> As all planed features and bug fixes are done and delivered, I will start
> release process tomorrow. Thanks for your support.
>
> Best,
> Yi
>

Re: HAWQ-1548 - Ambiguous message while logging hawq utilization

[Wen Lin]

Hi, Shubham,

The second message means that, this YARN cluster's total resource is 1376256
MB, 168.00 CORE.
And HAWQ clusters have the resource: 98304MB, 12.00 CORE of the whole
YARN cluster now.
The other resources in YARN haven't been allocated for HAWQ.
Maybe there are no HAWQ segments running on these segments at the moment,
or other reasons.

Thanks!


On Tue, Nov 14, 2017 at 9:25 AM, Shubham Sharma  wrote:

> While YARN mode is enabled, resource broker logs two things -
>
>- YARN cluster total resource
>- HAWQ's total resource per node.
>
> Following messages are logged
>
> 2017-11-11 23:21:40.944904
> UTC,,,p549330,th9000778560,con4,,seg-1,"LOG","0","Resource
> manager YARN resource broker counted YARN cluster having total
> resource (1376256 MB, 168.00
> CORE).",,,0,,"resourcebroker_LIBYARN.c",776,
>
> 2017-11-11 23:21:40.944921
> UTC,,,p549330,th9000778560,con4,,seg-1,"LOG","0","Resource
> manager YARN resource broker counted HAWQ cluster now having (98304
> MB, 12.00 CORE) in a YARN cluster of total resource (1376256 MB,
> 168.00 CORE).",,,0,,"resourcebroker_LIBYARN.c",785,
>
> The second message shown above is ambiguous, After reading the sentence
> below it looks like that complete Hawq cluster in whole has only 98304 MB
> and 12 cores. However according to the configuration it should be 98304 MB
> and 12 cores per segment server.
>
> Resource manager YARN resource broker counted HAWQ cluster now having
> (98304 MB, 12.00 CORE) in a YARN cluster of total resource
> (1376256 MB, 168.00 CORE).
>
> Either the wrong variables are printed or we can correct the message to
> represent that the resources logged are per node. As this can confuse the
> user into thinking that hawq cluster does not have enough resources.
>
> Is this understanding correct ?
>
> Regards,
> Shubham Sharma
>

Re: New Committer: Amy Bai

[Wen Lin]

Her contribution includes (but not limited to):

*Direct contribution to code base:*

   - 23 commits in total with some major components in hawq involved,
   including contributions to hawq tde , hawq planner , hawq resource manager
   and some critical bugs. https://github.com/apach
   e/incubator-hawq/commits?author=amyrazz44
   <https://github.com/apache/incubator-hawq/commits?author=amyrazz44>
   - 21 closed PRs: https://github.com/apache/incubator-hawq/pulls?utf8=%E2
   %9C%93=is%3Apr%20is%3Aclosed%20author%3Aamyrazz44
   
<https://github.com/apache/incubator-hawq/pulls?utf8=%E2%9C%93=is%3Apr%20is%3Aclosed%20author%3Aamyrazz44>
   - Design doc for TDE:HAWQ_TDE_Design_ver0.2 .pdf
   
<https://issues.apache.org/jira/secure/attachment/12844669/HAWQ_TDE_Design_ver0.2%20.pdf>

   - 9 improvements including documentation, test, build, command line
   tools, code refactor.


   - HAWQ-1310 <https://issues.apache.org/jira/browse/HAWQ-1310> Reformat
  resource_negotiator().
  - HAWQ-1315 <https://issues.apache.org/jira/browse/HAWQ-1315> Fix
  wrong info in validateResourcePoolStatus().
  - HAWQ-1333 <https://issues.apache.org/jira/browse/HAWQ-1333> Change
  access mode of source files for HAWQ.
  - HAWQ-1323 <https://issues.apache.org/jira/browse/HAWQ-1323> Fix
  access mode of files in adding license header.
  - HAWQ-1323 <https://issues.apache.org/jira/browse/HAWQ-1323> Add
  license header added to each source file.
  - HAWQ-1126 <https://issues.apache.org/jira/browse/HAWQ-1126> Disable
  feature test case of ExternalTable.
  - HAWQ-960 <https://issues.apache.org/jira/browse/HAWQ-960> Remove
  the file BUILD_INSTRUCTIONS.md and move the content in it to README.md
  - HAWQ-1148 <https://issues.apache.org/jira/browse/HAWQ-1148> Update
  gtest-parallel to make sure test case can run in both parallel way and
  serial way.
  - HAWQ-955 <https://issues.apache.org/jira/browse/HAWQ-955> Add
  scripts for feature test running in parallel.
  -
  - 5 bug fixes including share input scan, resource manager, parser,
   planner components.
  - HAWQ-1371 <https://issues.apache.org/jira/browse/HAWQ-1371> Fix QE
  process hung in shared input scan.
  - HAWQ-1458 <https://issues.apache.org/jira/browse/HAWQ-1458> Fix
  share input scan bug for writer part.
  - HAWQ-1317 <https://issues.apache.org/jira/browse/HAWQ-1317> Fix
  some regex issues with out-of-range characters and large char ranges.
  - HAWQ-858 <https://issues.apache.org/jira/browse/HAWQ-858> Fix
  parser to understand case/when expression in group by.
  - HAWQ-1162 <https://issues.apache.org/jira/browse/HAWQ-1162>
  Resource manager does not reference dynamic minimum water level of each
  segment when it times out YARN containers.












   - 7 subtask mainly focus on hawq tde implementation.
  - HAWQ-1511 <https://issues.apache.org/jira/browse/HAWQ-1511>  Add
  TDE-related properties into hdfs-client.xml.
  - HAWQ-1509 <https://issues.apache.org/jira/browse/HAWQ-1509> Support
  TDE read function.
  - HAWQ-1502 <https://issues.apache.org/jira/browse/HAWQ-1502> Add
  verification to support TDE write function.
  - HAWQ-1502 <https://issues.apache.org/jira/browse/HAWQ-1502> Support
   TDE write function
  - HAWQ-1501 <https://issues.apache.org/jira/browse/HAWQ-1501> Support
  TDE by adding KmsClientProvider class.
  - HAWQ-1500 <https://issues.apache.org/jira/browse/HAWQ-1500> Support
  TDE by adding common class HttpClient.
  - HAWQ-1518 <https://issues.apache.org/jira/browse/HAWQ-1518> Add a
  UDF for showing whether the data directory is an encryption zone.











*Indirect contributions to code base: *
Provides a lot of valuable comments for PRs and help improve the quality of
the codes.
Reviewed for 29 closed RPs: https://github.com/apache
/incubator-hawq/pulls?utf8=%E2%9C%93=is%3Apr%20commenter%3Aamyrazz44

*Mailing List:*
About 16 emails until now, related to answering questions, discussions etc.
http://mail-archives.apache.org/mod_mbox/incubator-hawq-dev

*JIRA:*
29 JIRAs are created and assigned with most of them mapping to PRs above.
28 JIRAs are closed and 1 JIRAs are in progress (opened pull requests).
https://issues.apache.org/jira/browse/HAWQ-1511?jql=project%
20%3D%20HAWQ%20AND%20(reporter%20in%20(abai)%20OR%20assignee%20in%20(abai))


On Wed, Nov 1, 2017 at 2:02 PM, Wen Lin <w...@pivotal.io> wrote:

> Hi,
>
> The Project Management Committee (PMC) for Apache HAWQ (incubating) has
> invited Amy Bai to become a committer and we are pleased to announce that
> she has accepted.
> Being a committer enables easier contribution to the project since there
> is no need to go via the patch submission 

New Committer: Amy Bai

[Wen Lin]

Hi,

The Project Management Committee (PMC) for Apache HAWQ (incubating) has
invited Amy Bai to become a committer and we are pleased to announce that
she has accepted.
Being a committer enables easier contribution to the project since there is
no need to go via the patch submission process. This should enable better
productivity. Please join us in congratulating her and we are looking
forward to collaborating with her in the open source community. Her
contribution includes (but not limited to):
List contributions to code base, documentation, code review, discussion in
mailing list, JIRA, etc.

Regards!

Re: Re: [ANNOUNCE] Apache HAWQ 2.2.0.0-incubating Released

[Wen Lin]

Congratulations!
Thanks Ruilong for all the efforts on release!
Thanks Yi for volunteering for next release!

Regards!

On Thu, Jul 13, 2017 at 3:04 PM, Ed Espino  wrote:

> Yi,
>
> +1 to your offer to be the Release Manager for the next Apache HAWQ
> release. We all know your past contributions well. Thank you for
> volunteering.
>
> Regards,
> -=e
>
> On Wed, Jul 12, 2017 at 11:48 PM, Yi JIN  wrote:
>
> > Hi Ruilong,
> >
> > I would like to take this responsibility as a volunteer for the next
> > release. As a committer I used to contribute a lot of code to Apache
> HAWQ,
> > consequently besides code work, if possible I would like to contribute
> more
> > in another way and learn more about growing an Apache project.
> >
> > Best,
> > Yi (yjin)
> >
> > On Thu, Jul 13, 2017 at 4:43 PM, HuoRuilong  wrote:
> >
> > > Great step towards a mature hawq and active community! Thanks everyone
> > for
> > > making this real, especially the help from Ed!
> > >
> > > To make it a more successful apache project and community, we need to
> > keep
> > > the release cadence. Who would like to be volunteer for the next
> release
> > > manager and drive the effort? Thanks.
> > >
> > > Best regards,
> > > Ruilong Huo
> > >
> > >
> > > At 2017-07-13 14:39:21, "Lili Ma"  wrote:
> > > >Congratulations everyone :)
> > > >
> > > >We're stepping further towards graduation!
> > > >
> > > >Best Regards,
> > > >Lili
> > > >
> > > >2017-07-13 13:16 GMT+08:00 Ed Espino :
> > > >
> > > >> Congratulations to everyone on the first Apache HAWQ release with
> > > >> convenience binaries. Special thanks to Ruilong for his excellent
> > > release
> > > >> management guidance.
> > > >>
> > > >> I'm very proud to be part of a great dev team.
> > > >>
> > > >> Cheers,
> > > >> -=e
> > > >>
> > > >> On Wed, Jul 12, 2017 at 10:00 PM, 陶征霖  wrote:
> > > >>
> > > >> > Congrats!
> > > >> >
> > > >> > 2017-07-13 9:55 GMT+08:00 Yandong Yao :
> > > >> >
> > > >> > > Great achievement, Congrats!
> > > >> > >
> > > >> > > On Thu, Jul 13, 2017 at 8:46 AM, Lei Chang <
> > chang.lei...@gmail.com>
> > > >> > wrote:
> > > >> > >
> > > >> > > > Congrats!
> > > >> > > >
> > > >> > > > Cheers
> > > >> > > > Lei
> > > >> > > >
> > > >> > > >
> > > >> > > > On Wed, Jul 12, 2017 at 3:27 PM, Ruilong Huo  >
> > > >> wrote:
> > > >> > > >
> > > >> > > > > Hi All,
> > > >> > > > >
> > > >> > > > > The Apache HAWQ (incubating) Project Team is proud to
> announce
> > > >> > > > > the release of Apache HAWQ 2.2.0.0-incubating.
> > > >> > > > >
> > > >> > > > > This is a source code and binary release.
> > > >> > > > >
> > > >> > > > > ABOUT HAWQ
> > > >> > > > > Apache HAWQ (incubating) combines exceptional MPP-based
> > > analytics
> > > >> > > > > performance, robust ANSI SQL compliance, Hadoop ecosystem
> > > >> integration
> > > >> > > > > and manageability, and flexible data-store format support,
> all
> > > >> > > > > natively in Hadoop, no connectors required.
> > > >> > > > >
> > > >> > > > > Built from a decade’s worth of massively parallel processing
> > > (MPP)
> > > >> > > > > expertise developed through the creation of open source
> > > Greenplum®
> > > >> > > > > Database and PostgreSQL, HAWQ enables you to
> > > >> > > > > swiftly and interactively query Hadoop data, natively via
> > HDFS.
> > > >> > > > >
> > > >> > > > > FEATURES AND ENHANCEMENTS INCLUDED IN THIS RELEASE
> > > >> > > > > - CentOS 7.x support
> > > >> > > > > Apache HAWQ is improved to be compatible with CentOS 7.x
> along
> > > with
> > > >> > > 6.x.
> > > >> > > > >
> > > >> > > > > - Apache Ranger integration
> > > >> > > > > Integrate Apache HAWQ with Apache Ranger through HAWQ Ranger
> > > Plugin
> > > >> > > > Service
> > > >> > > > > which is a RESTful service. It enables users to use Apache
> > > Ranger
> > > >> to
> > > >> > > > > authorize
> > > >> > > > > user access to Apache HAWQ resources. It also manages all
> > Hadoop
> > > >> > > > > components’
> > > >> > > > > authorization policies with the same user interface, policy
> > > store,
> > > >> > and
> > > >> > > > > auditing
> > > >> > > > > stores.
> > > >> > > > >
> > > >> > > > > - PXF ORC profile
> > > >> > > > > Fully supports PXF with Optimized Row Columnar (ORC) file
> > > format.
> > > >> > > > >
> > > >> > > > > - Fixes and enhancements on Apache HAWQ resource manager,
> > query
> > > >> > > > execution,
> > > >> > > > > dispatcher,
> > > >> > > > > catalog, management utilities and more.
> > > >> > > > >
> > > >> > > > > JIRA GENERATED RELEASE NOTES
> > > >> > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > > >> > > > > projectId=12318826=12339641
> > > >> > > > >
> > > >> > > > > RELEASE ARTIFACTS ARE AVAILABLE AT
> > > >> > > > > http://apache.org/dyn/closer.cgi/incubator/hawq/2.2.0.0-
> > > incubating
> > > >> > > > >
> > > >> > > > > SHA256 & 

New committer: Xiang Sheng

[Wen Lin]

Hi,

The Project Management Committee (PMC) for Apache HAWQ (incubating) has
invited Xiang Sheng to become a committer and we are pleased to announce
that he has accepted.
Being a committer enables easier contribution to the project since there is
no need to go via the patch submission process. This should enable better
productivity. Please join us in congratulating him and we are looking
forward to collaborating with him in the open source community. His
contribution includes (but not limited to):
List contributions to code base, documentation, code review, discussion in
mailing list, JIRA, etc.

Regards!
Wen

Re: [VOTE] New committer: Xiang Sheng

[Wen Lin]

Sorry, I made a mistake.
Please ignore this email, I will resend it to private list.
Thank you for pointing out!

On Tue, May 9, 2017 at 4:16 PM, Lei Chang <chang.lei...@gmail.com> wrote:

> Hi Wen, I think this vote should be in private@ list.
>
> Cheers
> Lei
>
>
>
> On Tue, May 9, 2017 at 4:15 PM, Lei Chang <chang.lei...@gmail.com> wrote:
>
> >
> > +1.
> >
> > On Tue, May 9, 2017 at 3:54 PM, Yi Jin <y...@pivotal.io> wrote:
> >
> >> +1
> >>
> >> Xiang showed great code contribution in Apache HAWQ project including
> >> fixing bugs and contributing new features. His work shows his deep
> insight
> >> of this project and effective development work, he is also active in
> both
> >> user and dev mail list. Therefore, I think he deserves an Apache HAWQ
> >> committer.
> >>
> >> Best,
> >> Yi
> >>
> >> On Tue, May 9, 2017 at 5:40 PM, Lili Ma <lil...@apache.org> wrote:
> >>
> >> > +1 for Xiang!
> >> >
> >> > Xiang has contributed a lot to Apache HAWQ project, including Ranger
> >> > integration, Hawq Register implementation and Resource Manager bug
> fix.
> >> > Also Xiang answered a lot of questions for HAWQ in usr/dev mail list
> and
> >> > StackOverFlow channel.  He also shared the tech talk in Apache HAWQ
> >> Meetup.
> >> >
> >> > I think Xiang is well deserved to become a Apache HAWQ committer.
> >> >
> >> > Thanks
> >> > Lili
> >> >
> >> > 2017-05-09 14:18 GMT+08:00 Wen Lin <w...@pivotal.io>:
> >> >
> >> > > Hi All,
> >> > >
> >> > > This is a VOTE email for promoting candidate *Xiang Sheng* (with
> >> github
> >> > id
> >> > > *stanlyxiang*) from contributor to committer, who has been
> >> contributing
> >> > to
> >> > > Apache HAWQ (incubating) in last one and half years (from Nov. 2015
> to
> >> > May
> >> > > 2017). Please give +1, 0 or -1 with reasons in this email thread.
> >> > >
> >> > > His contribution includes (but not limited to):
> >> > > *Direct contribution to code base:*
> >> > >
> >> > >- 43 commits in total with some major components in hawq
> involved,
> >> > >including contributions to Apache Ranger integration, hawq
> register
> >> > >and command line tools, resource manager.
> >> > >https://github.com/apache/incubator-hawq/commits?author=stan
> >> lyxiang
> >> > >- 41 closed PRs: https://github.com/apache/incu
> >> bator-hawq/pulls?q=is%
> >> > >3Apr+is%3Aclosed+author%3Astanlyxiang
> >> > ><https://github.com/apache/incubator-hawq/pulls?q=is%
> >> > > 3Apr+is%3Aclosed+author%3Astanlyxiang>
> >> > >
> >> > >- 13 improvements including documentation, test, build, command
> >> line
> >> > >tools, code refactor.
> >> > >
> >> > >
> >> > >- HAWQ-140 <https://issues.apache.org/jira/browse/HAWQ-140>Add
> >> more
> >> > >   information in HAWQ build instructions file
> >> > >   - HAWQ-143 <https://issues.apache.org/jira/browse/HAWQ-143
> >Add
> >> > >   informations in Apache-HAWQ README.md
> >> > >   - HAWQ-154<https://issues.apache.org/jira/browse/HAWQ-154>
> >> Update
> >> > >   BUILD_INSTRUCTIONS file for dependencies install method and
> >> > > ambiguous word.
> >> > >   - HAWQ-203<https://issues.apache.org/jira/browse/HAWQ-203>Add
> a
> >> > guc
> >> > >   for debug metadata, datalocality time stat.
> >> > >   - HAWQ-265 <https://issues.apache.org/jira/browse/HAWQ-265
> >> > >   <https://issues.apache.org/jira/browse/HAWQ-279> >Change
> >> metadata
> >> > >   share memory flush strategy to prevent out of share memory
> >> problem
> >> > > when
> >> > >   create too many hdfs_file.
> >> > >   - HAWQ-279 <https://issues.apache.org/jira/browse/HAWQ-279>
> >>  Add
> >> > 2
> >> > >   guc in template-hawq-site
> >> > >   - HAWQ-284 <https://issues.apache.org/jira/browse/HAWQ-284>
> >> Add a
>

[VOTE] New committer: Xiang Sheng

[Wen Lin]

yContext(SQL Statement) and
  encapsulate these contents to JSON request to RPS.
  - HAWQ-1256 <https://issues.apache.org/jira/browse/HAWQ-1256> Enhance
  libcurl connection to Ranger Plugin Service, keep it as a long-live
  connection in session level.
  - HAWQ-1256 <https://issues.apache.org/jira/browse/HAWQ-1256
  > non-superuser connect to db will do aclcheck through ranger and
  curl handle haven’t been initialized.
  - HAWQ-1276 <https://issues.apache.org/jira/browse/HAWQ-1276> The
  error message is not friendly when ranger plugin service is unavailable.
  - HAWQ-1276 <https://issues.apache.org/jira/browse/HAWQ-1276> hawq
  should error out directly when ranger plugin service is
unavailable, not a
  warning.
  - HAWQ-1292 <https://issues.apache.org/jira/browse/HAWQ-1292> Change
  GUC enable_ranger(bool) to a text GUC(hawq_acl_type), which can
allow other
  kinds of ACL.
  - HAWQ-1329 <https://issues.apache.org/jira/browse/HAWQ-1329> pg_catalog
  view fallback failed
  - HAWQ-1350 <https://issues.apache.org/jira/browse/HAWQ-1350> Add
  —enable-rps option to build ranger-plugin when build hawq
  - HAWQ-1350 <https://issues.apache.org/jira/browse/HAWQ-1350> Fix
  ranger-plugin makefile error
  - HAWQ-1362 <https://issues.apache.org/jira/browse/HAWQ-1362> Add RPS
  start/stop script in HAWQ start/stop script
  - HAWQ-1362 <https://issues.apache.org/jira/browse/HAWQ-1362> remove
  SYS_CATALINA_HOME to stay consistent with build check CATALINA_HOME


*Indirect contributions to code base: *
Provides a lot of valuable comments for PRs and help improve the quality of
the codes.
Reviewed for 63 closed RPs: https://github.com/apache
/incubator-hawq/pulls?q=is%3Apr+commenter%3Astanlyxiang

*Mailing List:*
About 30 emails until now, related to answering questions, discussions etc.
http://mail-archives.apache.org/mod_mbox/incubator-hawq-dev

*JIRA:*
47 JIRAs are created and assigned with most of them mapping to PRs above.
38 JIRAs are closed and 1 JIRAs are in progress (opened pull requests).
https://issues.apache.org/jira/issues/?jql=project%20%3D%20H
AWQ%20AND%20(reporter%20in%20(xsheng)%20OR%20assignee%20in%20(xsheng)).


*Voting ends one week from today, i.e., 6:30 **UTC on 2017-5-9 *or whenever
we got enough votes from PPMC members.

Thanks and Regards,
Wen Lin

Re: [VOTE] Apache HAWQ Incubator "powered by" logo vote

[Wen Lin]

+1 for HAWQ logo, with "HAWQ" text, with drop shadow.

On Tue, Apr 25, 2017 at 2:31 AM, Alexander Denissov 
wrote:

> +1 for HAWQ logo, with "HAWQ" text, no drop shadow.
>
> On Mon, Apr 24, 2017 at 8:51 AM, Hong  wrote:
>
> > +1 for HAWQ logo, with "HAWQ" text, no drop shadow.
> >
> > 2017-04-21 13:48 GMT-04:00 Ed Espino :
> >
> > > I have uploaded a few variations of the Apache HAWQ "powered by" logo.
> > > Please select your preference via vote. This vote will remain open for
> at
> > > least 72 hours.
> > >
> > > [ ] HAWQ logo, no text, no drop shadow
> > > [ ] HAWQ logo, no text, with drop shadow
> > > [ ] HAWQ logo, with "HAWQ" text, no drop shadow
> > > [ ] HAWQ logo, with "HAWQ" text, with drop shadow
> > >
> > > The full variations and additional sizes can be viewed here:
> > > https://cwiki.apache.org/confluence/pages/viewpage.
> > action?pageId=69407067
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > --
> > > *Ed Espino*
> > >
> >
>

Re: [VOTE]: Apache HAWQ 2.1.0.0-incubating Release Candidate 3 (RC3)

[Wen Lin]

+1
built, installed, and ran feature tests successfully.

On Tue, Feb 7, 2017 at 12:15 PM, Radar Da lei  wrote:

> +1
>
> Downloaded & compiled & make simple tests successfully.
>
> Regards,
> Radar
>
> On Tue, Feb 7, 2017 at 12:13 PM, Ming Li  wrote:
>
> > +1. Built, ran feature test & installcheck-good successfully.
> >
> > On Tue, Feb 7, 2017 at 12:01 PM, Yi Jin  wrote:
> >
> > > Checked
> > >
> > > 1） download code, unzip and build
> > > 2)  ran basic test
> > > 3) read release notes
> > >
> > > +1 approve
> > >
> > > Yi
> > >
> > >
> > > On Mon, Feb 6, 2017 at 10:28 PM, Paul Guo  wrote:
> > >
> > > > Ed, I think we could extend the deadline a bit since I know some
> people
> > > was
> > > > just back from vacation.
> > > >
> > > > 2017-02-03 18:07 GMT+08:00 Ed Espino :
> > > >
> > > > > This is the vote for Apache HAWQ (incubating) 2.1.0.0-incubating
> > > Release
> > > > > Candidate 3 (RC3).  This is a Source only release.
> > > > >
> > > > > The vote will run for at least 72 hours and will close on Monday,
> > > > February
> > > > > 6, 2017.
> > > > >
> > > > > Issues fixed in RC3:
> > > > > * https://issues.apache.org/jira/browse/HAWQ-1308 - Fix Javadocs
> > > compile
> > > > > warnings on JDK-8 for PXF
> > > > > * https://issues.apache.org/jira/browse/HAWQ-1309 - PXF Service
> must
> > > > > default to port 51200 and user pxf
> > > > > * https://issues.apache.org/jira/browse/HAWQ-1306 - PXF no longer
> > > builds
> > > > > with JDK 1.8 (after commit 6fa1ced2)
> > > > > * https://issues.apache.org/jira/browse/HAWQ-1297 - Make PXF
> install
> > > > ready
> > > > > from source code
> > > > > * https://issues.apache.org/jira/browse/HAWQ-1228 - Use profile
> > based
> > > on
> > > > > file format in HCatalog integration(HiveRC, HiveText profiles)
> > > > > * https://issues.apache.org/jira/browse/HAWQ-1297 - Make PXF
> install
> > > > ready
> > > > > from source
> > > > >
> > > > > Issues fixed in RC2:
> > > > > * https://issues.apache.org/jira/browse/HAWQ-1267 - NOTICE file
> need
> > > to
> > > > be
> > > > > updated to reflect the right year for copyright
> > > > > * https://issues.apache.org/jira/browse/HAWQ-1268 - pom.xml need
> to
> > be
> > > > > updated to reflect the correct version for apache hawq
> > > 2.1.0.0-incubating
> > > > > release
> > > > > * https://issues.apache.org/jira/browse/HAWQ-1277 - The "make"
> > command
> > > > > generates an error on CentOS 7 when --with-perl is run in
> configure.
> > > > >
> > > > > Issues deferred from previous vote(s):
> > > > > * [RC1] https://issues.apache.org/jira/browse/HAWQ-1278 -
> > Investigate
> > > > > installcheck-good issue on Mac OSX
> > > > >
> > > > > Release Notes (Apache Jira generated):
> > > > > *
> > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > > > > projectId=12318826=12338900
> > > > >
> > > > > Release verification steps can be found at:
> > > > > *
> > > > > https://cwiki.apache.org/confluence/display/HAWQ/
> > > > > Release+Process%3A+Step+by+step+guide#ReleaseProcess:
> > Stepbystepguide-
> > > > > ValidatetheReleaseCandidate
> > > > >
> > > > > The tag to be voted on: 2.1.0.0-incubating-rc3
> > > > > (12c7df017551f1c3b0deb38c7243db3e018ef62c), located here:
> > > > > *
> > > > > https://git-wip-us.apache.org/repos/asf?p=incubator-hawq.
> > > git;a=commit;h=
> > > > > 12c7df017551f1c3b0deb38c7243db3e018ef62c
> > > > >
> > > > > Git release branch:
> > > > > *
> > > > > https://git-wip-us.apache.org/repos/asf?p=incubator-hawq.
> > > > > git;a=shortlog;h=refs/heads/2.1.0.0-incubating
> > > > >
> > > > > Source release package:
> > > > > *
> > > > > https://dist.apache.org/repos/dist/dev/incubator/hawq/2.1.0.
> > > > > 0-incubating.RC3/apache-hawq-src-2.1.0.0-incubating.tar.gz
> > > > >
> > > > > Source release verification:
> > > > >
> > > > > * PGP Signature:
> > > > >
> > > > > https://dist.apache.org/repos/dist/dev/incubator/hawq/2.1.0.
> > > > > 0-incubating.RC3/apache-hawq-src-2.1.0.0-incubating.tar.gz.asc
> > > > > * SHA256/MD5 Hash:
> > > > >
> > > > > https://dist.apache.org/repos/dist/dev/incubator/hawq/2.1.0.
> > > > > 0-incubating.RC3/apache-hawq-src-2.1.0.0-incubating.tar.gz.sha256
> > > > >
> > > > > https://dist.apache.org/repos/dist/dev/incubator/hawq/2.1.0.
> > > > > 0-incubating.RC3/apache-hawq-src-2.1.0.0-incubating.tar.gz.md5
> > > > >
> > > > > Keys to verify the signature of the release artifact are available
> > at:
> > > > > * https://dist.apache.org/repos/dist/dev/incubator/hawq/KEYS
> > > > >
> > > > > The artifact(s) has been signed with Key ID: 57325522
> > > > >
> > > > > Previous Vote Thread(s):
> > > > > * RC1 [VOTE] Thread
> > > > >
> > > > > https://lists.apache.org/thread.html/
> 882dc11880f8794fc5603aee470c0e
> > > > > 5f912f579a7c247c270dbeb9a4@%3Cdev.hawq.apache.org%3E
> > > > > * RC1 [RESULT][VOTE] Thread
> > > > >
> > > > > https://lists.apache.org/thread.html/
> 

Re: Publishing HAWQ dev docker images

[Wen Lin]

This is very helpful,
thanks, Richard!

On Wed, Dec 28, 2016 at 2:19 PM, Richard Guo  wrote:

> It takes several minutes to create a docker image from dockerfile in my
> computer. The time depends on network connection speed, as the process
> would download a lot of packages to prepare the environment.
>
> Best Regards,
> Richard
>
> On Wed, Dec 28, 2016 at 2:11 PM, Paul Guo  wrote:
>
> > Good job. My suggestion is that we could check in docker files (build
> shell
> > script or makefile if needed) into hawq repo at least. How long do you
> need
> > to create a docker image with your docker file? Thanks.
> >
> >
> >
> > 2016-12-28 13:55 GMT+08:00 Richard Guo :
> >
> > > Hi everyone,
> > >
> > > HAWQ team is building HAWQ dev docker images. The purpose is to provide
> > an
> > > out-of-box way for developers to setup build and test environment for
> > HAWQ.
> > >
> > > A demo could be found on github hawq-docker
> > >  . Currently only
> CentOS
> > 7
> > > is supported. CentOS 6 will be supported soon. It is based on Zhanwei's
> > > work.
> > >
> > > The idea is to predefine all the environment setup steps in the
> > dockerfile
> > > and then build the image from dockerfile with tools provided by docker.
> > > After that, users can simply create containers with the docker image
> and
> > > then do the HAWQ build and test jobs.
> > > Also a Makefile is provided to simplify this process. Please refer to
> the
> > > README in the github repository for more details.
> > >
> > > Regarding the place to host the docker images and dockerfiles, does
> > anyone
> > > have any idea? Comments and discussions are welcomed.
> > >
> > > Best Regards,
> > > Richard
> > >
> >
>

Re: Apache HAWQ release manager volunteer

[Wen Lin]

+1. Thank you, Ed!

On Wed, Dec 7, 2016 at 12:10 PM, Paul Guo  wrote:

> +1. Thanks.
>
> 2016-12-07 11:12 GMT+08:00 Ed Espino :
>
> > HAWQ dev community,
> >
> > I am volunteering to be the release manager for the second Apache HAWQ
> > release.  I will be following the guidelines set forth in the project's
> > Release Management wiki section as well as the Apache documentation:
> >
> > o Apache HAWQ wiki Release Management:
> > https://cwiki.apache.org/confluence/display/HAWQ/Release+Management
> >
> > o A Guide To Release Management During Incubation (DRAFT):
> > http://incubator.apache.org/guides/releasemanagement.html
> >
> > NOTE: As needed, I will be updating the project's Release Management wiki
> > to reflect updates and filling possible gaps to the release process.
> >
> > Assuming there are no objections, I will do my best to send a release
> plan
> > within 48 hours.
> >
> > Regards,
> > -=e
> >
> > --
> > *Ed Espino*
> > *esp...@apache.org *
> >
>

Re: HAWQ standby master sync process

[Wen Lin]

Kyle,

When HAWQ cluster is initialized, if a standby master is configured in
hawq-site.xml, the HAWQ scripts will initialize standby master on one node,
and register it into master's gp_segment_configuration table. So the master
knows standby master from this catalog table.
Unlike segment instance, which is register itself by sending heartbeat
message to master, standby master has no heartbeat message.
It's not possible to have two standby masters running together, if you
initialize another standby master, the first one in
gp_segment_configuration table will be removed.

Regards!

Wen

On Tue, Sep 13, 2016 at 5:32 AM, Kyle Dunn <kd...@pivotal.io> wrote:

> Hey Ming -
>
> Am I understanding correctly that a standby master will register
> automagically to the active master, based on the contents of hawq-site.xml?
>
> What would happen if two different standby masters on different nodes both
> tried registering with the same active master? I ask because this is the
> exact situation that would be useful for having a passive DR site with HAWQ
> installed, querying for new WALs in the same flow as a local standby.
>
> As for "daisy chaining" masters, which I believe is what you described in
> (2) above: Master -> WAL -> Standby -> DR node, I think this may be less
> desirable than multiple "normal" standby client nodes, as losing the
> standby node becomes a cascading failure into DR.
>
> Anytime we can make use of the DFS available (I say DFS, rather than HDFS,
> as the hope is eventually this would be S3, Azure blob, Ceph, etc) - we
> should!  (unrelated to DR) In my mind, this includes propagating the
> system catalog to segment nodes via the underlying DFS, rather than
> transmitting as part of each query.
>
> Thank you for the helpful insight and discussion!
>
>
> -Kyle
>
> On Thu, Sep 8, 2016 at 10:55 PM Ming Li <m...@pivotal.io> wrote:
>
>> Hi Kyle,
>>
>> As for your question how to config standby host, when standby nodes(which
>> is config in hawq-site.xml) started, it will auto registered it's info in
>> the system table gp_segment_configuration(
>> there is system table:
>> http://hdb.docs.pivotal.io/20/reference/catalog/gp_segment_
>> configuration.html),
>> so that hawq can use this info internally in catalog.  if you need more
>> details about it, @wen lin can help you.
>>
>> Then standby will report the LSN of WALs it synched to master node, master
>> node according to this LSN to test the gap between master and node is
>> still
>> in xlog file or it is overwritten (because xlog file recycled). If the gap
>> is not in the xlog file, we cannot do further just report "out of sync",
>> which need to manually run hawq init standby to recreate standby node;
>> else
>> we just push the WAL after this LSN to standby node, and redo them. All
>> related standby script problem can ask @radar for help.
>>
>> In most cases the standby should be less workload than master, so I
>> suggestion maybe we can implement it as:
>> (1) Master push WAL to standby node, when standby received them, it
>> firstly
>> write to file, then report successfully to master so that no blocking
>> transaction commit.
>> (2) standby node redo them on this node, and at the same time, it need to
>> guarantee that the WAL should be transferred to the remote DR node, we can
>> set different sync policy (whether need to guarantee WAL transferred to
>> remote node when transaction committed ) in case of different transaction
>> commit latency and different data loss acceptance at remote node.
>>
>> More to discussed:
>> (1) If standby "report out of sync" and gap is not available on master
>> node, we need to reinit standby manually, which need to shutdown master
>> node. We need to think an stronger policy for this scenario, e.g. just
>> push
>> WAL to other nodes, and write as duplicate file? or we can further to
>> write
>> into hdfs directly?
>> (2) If multiple master feature implemented, maybe the design need to be
>> changed. I don't take time on it.
>>
>> Any comments or suggestions are welcomed. Thanks.
>>
>>
>> On Fri, Sep 9, 2016 at 1:22 AM, Kyle Dunn <kd...@pivotal.io> wrote:
>>
>> > Ming -
>> >
>> > Thank you for the info, this is very helpful in understanding how WAL
>> > shipment happens.
>> >
>> > One question I have is: if/where the destination host is configured in
>> > walsendserver.c? Alternatively, does a standby master client initiate
>> the
>> > request rather than the active master pushing out WAL

Re: HAWQ Ranger Integration Design Doc

[Wen Lin]

It's very strange, I attached a PDF file in the email, but it seems be
blocked.
The document can be downloaded from
https://issues.apache.org/jira/browse/HAWQ-256

On Thu, Jul 28, 2016 at 4:04 PM, Ruilong Huo <r...@pivotal.io> wrote:

> Any link to the doc?
>
> Best regards,
> Ruilong Huo
>
> On Thu, Jul 28, 2016 at 4:04 PM, Wen Lin <w...@pivotal.io> wrote:
>
> >
> >
> > On Thu, Jul 28, 2016 at 4:03 PM, Wen Lin <w...@pivotal.io> wrote:
> >
> >> Hi,
> >>
> >> A design document for HAWQ Ranger Integration has been worked out by
> >> Hubert, Lili and me.
> >> Please share your suggestion or comments.
> >>
> >> Regards!
> >>
> >
> >
>

Re: HAWQ Ranger Integration Design Doc

[Wen Lin]

On Thu, Jul 28, 2016 at 4:03 PM, Wen Lin <w...@pivotal.io> wrote:

> Hi,
>
> A design document for HAWQ Ranger Integration has been worked out by
> Hubert, Lili and me.
> Please share your suggestion or comments.
>
> Regards!
>

HAWQ Ranger Integration Design Doc

[Wen Lin]

Hi,

A design document for HAWQ Ranger Integration has been worked out by
Hubert, Lili and me.
Please share your suggestion or comments.

Regards!

Re: [Propose] Create a new HAWQ roadmap page

[Wen Lin]

Hi,

Glad to see this roadmap page is created. I am wondering if some
enhancements for Hadoop YARN related can be added into this roadmap.
1. support "YARN label based scheduling" in HAWQ and libyarn,
https://issues.apache.org/jira/browse/HAWQ-742
2. support Fair Scheduler in HAWQ and libyarn;
3. support resource preemption with Capacity Scheduler in HAWQ and libyarn,
http://hortonworks.com/blog/better-slas-via-resource-preemption-in-yarns-capacityscheduler/

It seems no jira is created for number 2 and 3.

Thanks!

On Thu, Jul 14, 2016 at 8:54 AM, Lei Chang  wrote:

> Added to the page:
> https://cwiki.apache.org/confluence/display/HAWQ/HAWQ+Roadmap
>
> It can be incrementally updated.
>
> Cheers
> Lei
>
>
>
>
> On Wed, Jun 29, 2016 at 5:23 PM, Lei Chang  wrote:
>
> >
> > I classified the items into the following categories. appreciate your
> > comments.
> >
> > Cloud related:
> > [HAWQ-308] - S3 Integration
> > [HAWQ-310] - Snapshot support
> >
> > Data Management Functionality Enhancement
> > [HAWQ-786] - Framework to support pluggable formats and file systems
> > [HAWQ-864] - Support ORC as a native file format
> > [HAWQ-150] - External tables can be designated for both READ and WRITE
> > [HAWQ-304] - Support update and delete on non-heap tables
> > [HAWQ-401] - json type support
> > [HAWQ-319] - REST API for HAWQ
> > [HAWQ-312] - Multiple active master support
> >
> > Performance enhancement
> > [HAWQ-303] - Index support for non-heap tables
> >
> > Languages & Analytics
> > [HAWQ-321] - Support plpython3u
> >
> > Ecosystem:
> > [HAWQ-256] - Integrate Security with Apache Ranger
> > [HAWQ-29] - Refactor HAWQ InputFormat to support Spark/Scala
> >
> > Management & Build
> > [HAWQ-8] - Installing the HAWQ Software thru the Apache Ambari
> > [HAWQ-311] - Data Transfer tool
> > [HAWQ-326] - Support RPM build for HAWQ
> >
> > Cheers
> > Lei
> >
> >
> >
> >
> > On Fri, Jun 24, 2016 at 5:10 PM, Lei Chang  wrote:
> >
> >>
> >> Nice, I created a page and we can discuss the items and put them on the
> >> page.
> >>
> >> For the items, I think it makes sense to add at least items in the jira
> >> roadmap panel, here are some major ones I extracted from the panel.
> looks
> >> better to classify them into categories.
> >>
> >> [HAWQ-786] - Framework to support pluggable formats and file systems
> >> [HAWQ-864] - Support ORC as a native file format
> >> [HAWQ-308] - S3 Integration
> >> [HAWQ-256] - Integrate Security with Apache Ranger
> >> [HAWQ-150] - External tables can be designated for both READ and WRITE
> >> [HAWQ-303] - Index support for non-heap tables
> >> [HAWQ-304] - Support update and delete on non-heap tables
> >> [HAWQ-310] - Snapshot support
> >> [HAWQ-312] - Multiple active master support
> >> [HAWQ-319] - REST API for HAWQ
> >> [HAWQ-321] - Support plpython3u
> >> [HAWQ-29] - Refactor HAWQ InputFormat to support Spark/Scala
> >> [HAWQ-311] - Data Transfer tool
> >> [HAWQ-326] - Support RPM build for HAWQ
> >> [HAWQ-8] - Installing the HAWQ Software thru the Apache Ambari
> >> [HAWQ-752] - build pxf compatible with Apache Hadoop
> >> [HAWQ-401] - json type support
> >>
> >> Cheers
> >> Lei
> >>
> >>
> >>
> >> On Thu, Jun 23, 2016 at 11:23 PM, Vineet Goel 
> wrote:
> >>
> >>> +1 too
> >>>
> >>> I can help start a draft on the wiki based on historical user requests
> >>> and
> >>> trends in the ecosystem. And of course, the roadmap is a living and
> >>> breathing document which will continue to evolve over time based on
> >>> continuous feedback, and more.
> >>>
> >>> -Vineet
> >>>
> >>>
> >>> On Thu, Jun 23, 2016 at 8:18 AM, Kavinder Dhaliwal <
> kdhali...@pivotal.io
> >>> >
> >>> wrote:
> >>>
> >>> > +1 I'm in favor of this. The Zeppelin roadmap is very community
> driven
> >>> and
> >>> > having something similar for HAWQ will go a long way to getting more
> >>> > feedback about the overall direction and goals of HAWQ.
> >>> >
> >>> > On Thu, Jun 23, 2016 at 2:02 AM, Lei Chang 
> >>> wrote:
> >>> >
> >>> > > Hi Guys,
> >>> > >
> >>> > > I noticed there are a lot of requests about hawq roadmaps coming
> >>> from the
> >>> > > offline hawq activities (meetup et al).
> >>> > >
> >>> > > Although we have the list of backlog JIRAs on our JIRA page
> >>> > > <
> >>> > >
> >>> >
> >>>
> https://issues.apache.org/jira/browse/HAWQ/?selectedTab=com.atlassian.jira.jira-projects-plugin:roadmap-panel
> >>> > > >.
> >>> > > But it does not give a high level description. A good example from
> >>> other
> >>> > > communities is here:
> >>> > >
> >>> https://cwiki.apache.org/confluence/display/ZEPPELIN/Zeppelin+Roadmap
> >>> > >
> >>> > > So I am proposing we have a similar HAWQ Roadmap page maintained on
> >>> our
> >>> > > wiki page.
> >>> > >
> >>> > > Thoughts?
> >>> > >
> >>> > > Cheers
> >>> > > Lei
> >>> > >
> >>> >
> >>>
> >>
> >>
> >
>

Re: About *.out files in test/feature

[Wen Lin]

Very cool!

On Tue, Jul 12, 2016 at 12:18 PM, Hong Wu  wrote:

> BTW, since the make system is a little tricky in HAWQ, I think `make
> distclean` should also delete third-party libraries cloned during "make"
> such as orca and so on.
>
> In conclusion, `make` should delete files generated from compiling. `make
> distclean` should return back to the original codebase to users as it was
> cloned.
>
> Best
> xunzhang
>
> 2016-07-12 11:57 GMT+08:00 Gmail :
>
> > I agree with ruilong.
> > Make clean should clean files generated only in make process.
> >
> > Make distclean should clean files generated in make and configure
> process.
> >
> > xunzhang
> > Send from my iPhone
> >
> >
> >
> > 发自我的 iPhone
> > > 在 2016年7月12日，11:40，Ruilong Huo  写道：
> > >
> > > I think "make clean" should only clean the files generated during
> compile
> > > process.
> > > The *.out files are created by feature test code with psql, so I would
> > > prefer keep them.
> > > If you would like to clean them, that should be in feature test
> > framework,
> > > not in makefile.
> > >
> > > Any comments?
> > >
> > > Best regards,
> > > Ruilong Huo
> > >
> > >> On Tue, Jul 12, 2016 at 11:37 AM, Xiang Sheng 
> > wrote:
> > >>
> > >> We indeed should delete these useless files after runing make clean or
> > make
> > >> distclean.
> > >>
> > >>> On Tue, Jul 12, 2016 at 11:35 AM, Ming Li  wrote:
> > >>>
> > >>> LGTM. +1.
> > >>>
> >  On Tue, Jul 12, 2016 at 11:32 AM, Paul Guo 
> wrote:
> > 
> >  I'd mask the output files after running feature tests in .gitignore
> > and
> >  clean it up after running "make clean" or "make distclean". Anyone
> has
> > >>> any
> >  suggestions? Thanks.
> > 
> >  diff --git a/src/test/feature/.gitignore
> b/src/test/feature/.gitignore
> >  index a2e6bd4..c7332b2 100644
> >  --- a/src/test/feature/.gitignore
> >  +++ b/src/test/feature/.gitignore
> >  @@ -1 +1,2 @@
> >  doc/
> >  +**/*.out
> > 
> >  diff --git a/src/test/feature/Makefile b/src/test/feature/Makefile
> >  index adc6acc..e0985d1 100644
> >  --- a/src/test/feature/Makefile
> >  +++ b/src/test/feature/Makefile
> >  @@ -35,6 +35,7 @@ doc:
> > doxygen doxygen_template
> > 
> >  clean distclean: sharelibclean
> >  +   find . -type f -name "*.out" |xargs rm -f
> > $(RM) feature-test
> > $(RM) feature-test.dSYM
> > >>
> > >>
> > >>
> > >> --
> > >> Best Regards,
> > >> Xiang Sheng
> > >>
> >
>

Re: About the commit

[Wen Lin]

agree!
To make commit messages more clear.

On Mon, Jun 27, 2016 at 4:15 PM, Ruilong Huo  wrote:

> +1 with strong agree. Squash or even manual merge so that the message
> follows the convention would definitely help other contributors to
> understand each commit.
>
> Best regards,
> Ruilong Huo
>
> On Sat, Jun 25, 2016 at 9:43 AM, hong wu  wrote:
>
> > FYI: In new version of github
> > , committers could
> > handle
> > the squash process much more convenient.
> >
> > xunzhang
> >
> > 2016-06-25 9:30 GMT+08:00 hong wu :
> >
> > > Hi HAWQ committers,
> > >
> > > I notice that there are some informal commits of recent check-in. For
> > > example:
> > > ce3f7c6b5c0315b97298d651f5d5f7383000491a
> > > <
> >
> https://github.com/apache/incubator-hawq/commit/ce3f7c6b5c0315b97298d651f5d5f7383000491a
> > >
> > > 4d44097085fd139002a255b1032082dc0b030414
> > > <
> >
> https://github.com/apache/incubator-hawq/commit/4d44097085fd139002a255b1032082dc0b030414
> > >
> > > 817249a4605abd4415fc0de8e6a545bf88d2aa2e
> > > <
> >
> https://github.com/apache/incubator-hawq/commit/817249a4605abd4415fc0de8e6a545bf88d2aa2e
> > >
> > > ...
> > >
> > > I think we'd better ensure a commit info starting with `HAWQ-#JIRA`. In
> > > the following, I try to list the senses developer should pay attention
> > to:
> > > 1. To avoid commits generated automatically(for example
> > > ce3f7c6b5c0315b97298d651f5d5f7383000491a
> > > <
> >
> https://github.com/apache/incubator-hawq/commit/ce3f7c6b5c0315b97298d651f5d5f7383000491a
> > >),
> > > do not merge or pull from upstream after local commits. Sync with
> > upstream
> > > master before your local commits.
> > > 2. Check commits info before pushing into master. If you found a commit
> > > with empty code change, rebase then squash it.
> > > 3. If a pull request contains some informal commits(maybe for code
> review
> > > convenience), committers should rebase then squash this temporary
> commits
> > > before pushing into master.
> > > 4. If you local develop branch is behind of upstream, it's ok. There
> are
> > > some acceptable reasons for that. For example, during the discuss of
> your
> > > pull request, the master branch is updated. But in this case, the
> asfgit
> > >  could not resolve it which means
> developers
> > > should close your pull request manually. So I recommend to attach the
> > pull
> > > request number in your commits comments(For example
> > > a57cc9523f97e471a69b658556c989d13ad88661
> > > <
> >
> https://github.com/apache/incubator-hawq/commit/a57cc9523f97e471a69b658556c989d13ad88661
> > >
> > > ).
> > >
> > > Best
> > > xunzhang
> > >
> > >
> >
>

Re: HAWQ YARN RPC Errors

[Wen Lin]

Hi, Gagan,

Where is the log? There is no attachment in your email.

Thanks!

On Sun, May 15, 2016 at 1:24 AM, Gagan Brahmi <gaganbra...@gmail.com> wrote:

> Hi Wen,
>
> Please find attached logs which has a few instances of the occurrence
> of the error.
>
>
> Regards,
> Gagan Brahmi
>
> On Thu, May 12, 2016 at 7:33 PM, Wen Lin <w...@pivotal.io> wrote:
> > Hi, Gagan,
> >
> > It seems a sync failure between QD and Resource Manager. Not related to
> > libyarn 's RPC.
> > Would you like to attach the master's log file? Thanks!
> >
> > On Fri, May 13, 2016 at 12:58 AM, Gagan Brahmi <gaganbra...@gmail.com>
> > wrote:
> >
> >> Hi Team,
> >>
> >> Do we have some recommended tuning for the RPC warning/errors
> >> encountered intermittently?
> >>
> >> The error which is seen is the following:
> >>
> >> WARNING:  Sync RPC framework (inet) finds exception raised.
> >> ERROR:  failed to return resource to resource manager, failed to
> >> receive content (pquery.c:991)
> >>
> >> This error however, disappears when we retry the query. There are
> >> cases when the query is to be retried more than once.
> >>
> >> The error looks to be invoked when COMM2RM_CLIENT_FAIL_RECV is
> encountered.
> >>
> >> The setup is using YARN resource manager. And the following is the
> >> yarn-client configuration used:
> >>
> >> 
> >>
> >> 
> >>   hadoop.security.authentication
> >>   kerberos
> >> 
> >>
> >> 
> >>   rpc.client.connect.retry
> >>   10
> >> 
> >>
> >> 
> >>   rpc.client.connect.tcpnodelay
> >>   true
> >> 
> >>
> >> 
> >>   rpc.client.connect.timeout
> >>   60
> >> 
> >>
> >> 
> >>   rpc.client.max.idle
> >>   1
> >> 
> >>
> >> 
> >>   rpc.client.ping.interval
> >>   1
> >> 
> >>
> >> 
> >>   rpc.client.read.timeout
> >>   360
> >> 
> >>
> >> 
> >>   rpc.client.socket.linger.timeout
> >>   -1
> >> 
> >>
> >> 
> >>   rpc.client.timeout
> >>   360
> >> 
> >>
> >> 
> >>   rpc.client.write.timeout
> >>   360
> >> 
> >>
> >> 
> >>   yarn.client.failover.max.attempts
> >>   15
> >> 
> >>
> >>   
> >>
> >> I would appreciate some recommendations.
> >>
> >>
> >> Regards,
> >> Gagan Brahmi
> >>
>

Re: HAWQ YARN RPC Errors

[Wen Lin]

Hi, Gagan,

It seems a sync failure between QD and Resource Manager. Not related to
libyarn 's RPC.
Would you like to attach the master's log file? Thanks!

On Fri, May 13, 2016 at 12:58 AM, Gagan Brahmi 
wrote:

> Hi Team,
>
> Do we have some recommended tuning for the RPC warning/errors
> encountered intermittently?
>
> The error which is seen is the following:
>
> WARNING:  Sync RPC framework (inet) finds exception raised.
> ERROR:  failed to return resource to resource manager, failed to
> receive content (pquery.c:991)
>
> This error however, disappears when we retry the query. There are
> cases when the query is to be retried more than once.
>
> The error looks to be invoked when COMM2RM_CLIENT_FAIL_RECV is encountered.
>
> The setup is using YARN resource manager. And the following is the
> yarn-client configuration used:
>
> 
>
> 
>   hadoop.security.authentication
>   kerberos
> 
>
> 
>   rpc.client.connect.retry
>   10
> 
>
> 
>   rpc.client.connect.tcpnodelay
>   true
> 
>
> 
>   rpc.client.connect.timeout
>   60
> 
>
> 
>   rpc.client.max.idle
>   1
> 
>
> 
>   rpc.client.ping.interval
>   1
> 
>
> 
>   rpc.client.read.timeout
>   360
> 
>
> 
>   rpc.client.socket.linger.timeout
>   -1
> 
>
> 
>   rpc.client.timeout
>   360
> 
>
> 
>   rpc.client.write.timeout
>   360
> 
>
> 
>   yarn.client.failover.max.attempts
>   15
> 
>
>   
>
> I would appreciate some recommendations.
>
>
> Regards,
> Gagan Brahmi
>

Re: Error building libyarn

[Wen Lin]

I haven't met with such building error. Libyarn is built with GCC 4.4.2 and
tested here. So that's probably compiler difference causes this error.
It seems rethrow_if_nested is ambiguous to the compiler.
Would you like to add a "std::" before "rethrow_if_nested" and have a try?

Thanks!

On Fri, Mar 4, 2016 at 5:43 AM, Gagan Brahmi  wrote:

> Hi Guys,
>
> Wondering if anyone has got around this problem earlier.
>
> I am getting the following error when I try to make libyarn.
>
> [ 32%] Building CXX object
> src/CMakeFiles/libyarn-shared.dir/libyarnclient/ApplicationClient.cpp.o
> cd /root/HAWQ/incubator-hawq-master/depends/libyarn/build/src &&
> /usr/bin/g++   -DTEST_HDFS_PREFIX=\"./\" -D_GNU_SOURCE
> -D__STDC_FORMAT_MACROS -Dlibyarn_shared_EXPORTS
> -fno-omit-frame-pointer -msse4.2 -Wl,--export-dynamic -std=c++0x -Wall
> -O2 -g -DNDEBUG -fPIC -I/usr/local/include
> -I/root/HAWQ/incubator-hawq-master/depends/libyarn/src
> -I/root/HAWQ/incubator-hawq-master/depends/libyarn/src/common
> -I/root/HAWQ/incubator-hawq-master/depends/libyarn/build/src
> -I/usr/include/libxml2
> -I/root/HAWQ/incubator-hawq-master/depends/libyarn/mock-o
> CMakeFiles/libyarn-shared.dir/libyarnclient/ApplicationClient.cpp.o -c
>
> /root/HAWQ/incubator-hawq-master/depends/libyarn/src/libyarnclient/ApplicationClient.cpp
>
> /root/HAWQ/incubator-hawq-master/depends/libyarn/src/libyarnclient/ApplicationClient.cpp:
> In constructor ‘libyarn::ApplicationClient::ApplicationClient(std::string&,
> std::string&, std::string&)’:
>
> /root/HAWQ/incubator-hawq-master/depends/libyarn/src/libyarnclient/ApplicationClient.cpp:72:9:
> warning: format ‘%d’ expects argument of type ‘int’, but argument 4
> has type ‘std::vector::size_type {aka long unsigned
> int}’ [-Wformat]
>
> /root/HAWQ/incubator-hawq-master/depends/libyarn/src/libyarnclient/ApplicationClient.cpp:
> In function ‘void libyarn::HandleYarnFailoverException(const
> Yarn::YarnFailoverException&)’:
>
> /root/HAWQ/incubator-hawq-master/depends/libyarn/src/libyarnclient/ApplicationClient.cpp:139:28:
> error: call of overloaded ‘rethrow_if_nested(const
> Yarn::YarnFailoverException&)’ is ambiguous
>
> /root/HAWQ/incubator-hawq-master/depends/libyarn/src/libyarnclient/ApplicationClient.cpp:139:28:
> note: candidates are:
> In file included from
>
> /usr/local/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.4/../../../../include/c++/4.7.4/exception:155:0,
>  from
>
> /usr/local/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.4/../../../../include/c++/4.7.4/ios:40,
>  from
>
> /usr/local/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.4/../../../../include/c++/4.7.4/ostream:40,
>  from
>
> /usr/local/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.4/../../../../include/c++/4.7.4/iostream:40,
>  from
>
> /root/HAWQ/incubator-hawq-master/depends/libyarn/src/libyarnclient/ApplicationClient.cpp:20:
>
> /usr/local/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.4/../../../../include/c++/4.7.4/bits/nested_exception.h:146:5:
> note: void std::rethrow_if_nested(const _Ex&) [with _Ex =
> Yarn::YarnFailoverException]
> In file included from
>
> /root/HAWQ/incubator-hawq-master/depends/libyarn/src/common/SessionConfig.h:24:0,
>  from
>
> /root/HAWQ/incubator-hawq-master/depends/libyarn/src/libyarnclient/ApplicationClient.cpp:25:
>
> /root/HAWQ/incubator-hawq-master/depends/libyarn/src/common/ExceptionInternal.h:103:20:
> note: void Yarn::rethrow_if_nested(const T&) [with T =
> Yarn::YarnFailoverException]
> make[2]: ***
> [src/CMakeFiles/libyarn-shared.dir/libyarnclient/ApplicationClient.cpp.o]
> Error 1
> make[2]: Leaving directory
> `/root/HAWQ/incubator-hawq-master/depends/libyarn/build'
> make[1]: *** [src/CMakeFiles/libyarn-shared.dir/all] Error 2
> make[1]: Leaving directory
> `/root/HAWQ/incubator-hawq-master/depends/libyarn/build'
> make: *** [all] Error 2
>
> Any help or hints would be appreciated!
>
>
>
> Regards,
> Gagan Brahmi
>

Re: Adjusting license headers

[Wen Lin]

Thank you, Roman!
Once you finish review, I will modify them together.

On Tue, Dec 29, 2015 at 3:13 PM, Roman Shaposhnik 
wrote:

> Hi!
>
> I've committed my patch on top of the latest master.
> I think we're really close to getting done with all this!
>
> Thanks for adjusting the pom file. One thing I've noticed
> while reviewing that change is that I actually need
> to review it more ;-)
>
> Basically, it boils down to things like:
>
> https://github.com/apache/incubator-hawq/commit/c9fc57a3a32d7ec9f522ec5936027d6be290e553#diff-a27afae6b1039ded0f71ad9967e1c67eL2
>
> being absolutely THE RIGHT thing to do, while
> things like:
>
> https://github.com/apache/incubator-hawq/commit/c9fc57a3a32d7ec9f522ec5936027d6be290e553#diff-b012d7bca7f50ded58242c01ffa5c52b
>
> does NOT look right to me.
>
> The difference boils down to files that contain code
> that originated from Greenplum/EMC/Pivotal vs.
> files that are unchanged between HAWQ and
> PostgreSQL.
>
> Give me a day or two to review a bit further. We're
> not quite out of the woods yet, but we're REALLY close.
>
> Thanks,
> Roman.
>

Re: license headers for hawq

[Wen Lin]

Hi, Roman,

Thank you for the suggestion!
I've put a pom.xml for hawq, it contains the exclude list.
run "mvn apache-rat:rat" can get a report. No unknown Licenses now.

Regards,

On Tue, Dec 8, 2015 at 6:52 AM, Roman Shaposhnik <ro...@shaposhnik.org>
wrote:

> On Mon, Dec 7, 2015 at 3:13 AM, Wen Lin <w...@pivotal.io> wrote:
> > Hi, Caleb,
> > The files you mentioned that need to be removed is recorded in Jira:
> > https://issues.apache.org/jira/browse/HAWQ-186
> > Lirong is working on this.
> >
> > Hi, Roman,
> > Today I add license headers for some source files. The latest status is :
> > Summary
> > ---
> > Generated at: 2015-12-07T18:59:07+08:00
> > Notes: 78
> > Binaries: 786
> > Archives: 20
> > Standards: 5834
> >
> > Apache Licensed: 1435
> > Generated Documents: 0
> >
> > JavaDocs are generated and so license header is optional
> > Generated files do not required license headers
> >
> > 4383 Unknown Licenses
> >
> > Some files are from 3rd party, like postgres, gtest, gmock, etc. I am
> > wondering, is it correct to add license header for these files?
> > Some other files are .out, .ans, .sgml etc, when use apache-rat to add
> > license header for a folder, these files will be ignored by apache-rat
>
> You don't need to change license headers on files coming from
> other projects (in fact you can NOT do that legally).
>
> You do, however, need to compile a very detailed list of exceptions
> that you will pass to rat via -E/-e option:
> http://creadur.apache.org/rat/apache-rat/index.html
>
> I would like these exception to be grouped and documented somehow
> so that anybody reviewing the release can make sure why things
> are on the list of exceptions.
>
> Thanks,
> Roman.
>

Re: license headers for hawq

[Wen Lin]

Hi, Caleb,
The files you mentioned that need to be removed is recorded in Jira:
https://issues.apache.org/jira/browse/HAWQ-186
Lirong is working on this.

Hi, Roman,
Today I add license headers for some source files. The latest status is :
Summary
---
Generated at: 2015-12-07T18:59:07+08:00
Notes: 78
Binaries: 786
Archives: 20
Standards: 5834

Apache Licensed: 1435
Generated Documents: 0

JavaDocs are generated and so license header is optional
Generated files do not required license headers

4383 Unknown Licenses

Some files are from 3rd party, like postgres, gtest, gmock, etc. I am
wondering, is it correct to add license header for these files?
Some other files are .out, .ans, .sgml etc, when use apache-rat to add
license header for a folder, these files will be ignored by apache-rat

Thanks!

On Thu, Dec 3, 2015 at 3:29 AM, Caleb Welton  wrote:

> From my quick investigation there are:
>
> 1. The stupid stuff that can simply be deleted
>
> depends/libyarn/releng/bin/templates/.core-site-secure-ha.xml.swp
> tools/bin/ext/yaml/__init__.pyc
> tools/bin/ext/yaml/composer.pyc
> tools/bin/ext/yaml/constructor.pyc
> tools/bin/ext/yaml/cyaml.pyc
> tools/bin/ext/yaml/dumper.pyc
> tools/bin/ext/yaml/emitter.pyc
> tools/bin/ext/yaml/error.pyc
> tools/bin/ext/yaml/events.pyc
> tools/bin/ext/yaml/loader.pyc
> tools/bin/ext/yaml/nodes.pyc
> tools/bin/ext/yaml/parser.pyc
> tools/bin/ext/yaml/reader.pyc
> tools/bin/ext/yaml/representer.pyc
> tools/bin/ext/yaml/resolver.pyc
> tools/bin/ext/yaml/scanner.pyc
> tools/bin/ext/yaml/serializer.pyc
> tools/bin/ext/yaml/tokens.pyc
>
>
> 2. Data files used in testing
>
>   The most substantial list being the ones under
>   contrib/hawq-hadoop/hawq-mapreduce-tool/test-data/hawq-data/gpseg1
>
>
> 3. Jar files and tar files of various dependencies, including:
>
> contrib/hawq-hadoop/hawq-mapreduce-tool/lib/parquet-column-1.1.0.jar
> contrib/hawq-hadoop/hawq-mapreduce-tool/lib/parquet-common-1.1.0.jar
> contrib/hawq-hadoop/hawq-mapreduce-tool/lib/parquet-encoding-1.1.0.jar
> contrib/hawq-hadoop/hawq-mapreduce-tool/lib/parquet-format-1.0.0.jar
> contrib/hawq-hadoop/hawq-mapreduce-tool/lib/parquet-hadoop-1.1.0.jar
> contrib/hawq-hadoop/hawq-mapreduce-tool/lib/postgresql-9.2-1003-jdbc4.jar
> contrib/hawq-hadoop/hawq-mapreduce-tool/lib/snakeyaml-1.12.jar
> contrib/hawq-hadoop/hawq-mapreduce-tool/lib/snappy-java-1.1.0.jar
> pxf/gradle/wrapper/gradle-wrapper.jar
> tools/bin/pythonSrc/epydoc-3.0.1.tar.gz
> tools/bin/pythonSrc/lockfile-0.9.1.tar.gz
> tools/bin/pythonSrc/logilab-astng-0.20.1.tar.gz
> tools/bin/pythonSrc/logilab-common-0.50.1.tar.gz
> tools/bin/pythonSrc/paramiko-1.7.6-9.tar.gz
> tools/bin/pythonSrc/PSI-0.3b2_gp.tar.gz
> tools/bin/pythonSrc/pychecker-0.8.18.tar.gz
> tools/bin/pythonSrc/pycrypto-2.0.1.tar.gz
> tools/bin/pythonSrc/pylint-0.21.0.tar.gz
> tools/bin/pythonSrc/unittest2-0.5.1.tar.gz
> tools/bin/src/netperf-2.4.3.tar.gz
>
> 4. a cacerts file
>
> depends/libyarn/releng/make/dependencies/cacerts
>
> 5. a couple image files
>
> src/port/win32.ico
> src/tools/backend/flow.gif
>
>
> The above was identified from the following
>
> find * -type f -exec file '{}' + | grep -v text | grep -v empty >
> /tmp/binfiles
>
> There were additionally some false positives such as
>src/test/regress/data/mpp12839_1.data
>
> Which are basically text but include some unusual characters.
>
> Regards,
>   Caleb
>
>
>
> On Tue, Dec 1, 2015 at 6:45 PM, Roman Shaposhnik 
> wrote:
>
> > On Mon, Nov 30, 2015 at 10:24 PM, Owen O'Malley 
> > wrote:
> > > Can I ask what the binary files are? It is hard to include binary files
> > in
> > > Apache releases, because they are hard to inspect or validate.
> >
> > I actually have that very same question. I was hoping that comments in
> > the RAT exclusion file will provide us with some insight ;-)
> >
> > Thanks,
> > Roman.
> >
>

license headers for hawq

[Wen Lin]

Hi, HAWQ Mentors,

I have added license headers to apache-hawq, here is the rules I follow.
1. will not add license headers to source files from or modified from
Postgres.
2. will not add license headers 3rd-party tools/libraries, like
contrib/orafce, pgcrypto, pljava, etc.
3. add  license headers to files from greenplum, or created by hawq(use
apache-rat to add license header for source files).

*
Summary
---
Generated at: 2015-11-30T10:40:35+08:00
Notes: 79
Binaries: 796
Archives: 20
Standards: 5870

Apache Licensed: 1366
Generated Documents: 0

JavaDocs are generated and so license header is optional
Generated files do not required license headers

4488 Unknown Licenses

***

Please help me verify it.
Thanks,

Re: Problem about Hawq on YARN

[Wen Lin]

Hi, Leon,

The log you attached maybe known bug in hawq dev team or an unknown one.
It's hard to conclude based on it.
Would you please send a full log? Then we can know what happened before the
error message was printed.

Thank you!

On Fri, Nov 27, 2015 at 3:23 PM, Leon Zhang  wrote:

> Hi, HAWQ Developers:
>
>  Thanks for the friendly hawq community, they help me to setup the
> hawq-2.0 on YARN. After I execute the TPC-DS queries, I encounter some yarn
> error for part of queries.
>
>  For instance, the query1:
> -- start query 12 in stream 0 using template query1.tpl and seed 1711160884
> with customer_total_return as
> (select sr_customer_sk as ctr_customer_sk
> ,sr_store_sk as ctr_store_sk
> ,sum(SR_REVERSED_CHARGE) as ctr_total_return
> from store_returns
> ,date_dim
> where sr_returned_date_sk = d_date_sk
> and d_year =1998
> group by sr_customer_sk
> ,sr_store_sk)
>  select  c_customer_id
> from customer_total_return ctr1
> ,store
> ,customer
> where ctr1.ctr_total_return > (select avg(ctr_total_return)*1.2
> from customer_total_return ctr2
> where ctr1.ctr_store_sk = ctr2.ctr_store_sk)
> and s_store_sk = ctr1.ctr_store_sk
> and s_state = 'TN'
> and ctr1.ctr_customer_sk = c_customer_sk
> order by c_customer_id
> limit 100;
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:502: WARNING:  Sync RPC
> framework (inet) finds exception raised.
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:502: ERROR:  failed to return
> resource to HAWQ resource manager because of failing to receive content.
> (pquery.c:983)
>
>  And many other errors like this:
> $ grep psql:/mnt/xiaolin/query/scale_1/query_0.sql
> hawq_yarn/log_1/log_1.log
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:502: WARNING:  Sync RPC
> framework (inet) finds exception raised.
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:502: ERROR:  failed to return
> resource to HAWQ resource manager because of failing to receive content.
> (pquery.c:983)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:1103: WARNING:  Sync RPC
> framework (inet) finds exception raised.
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:1103: ERROR:  failed to return
> resource to HAWQ resource manager because of failing to receive content.
> (pquery.c:983)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:1132: ERROR:  failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:1997: WARNING:  Sync RPC
> framework (inet) finds exception raised.
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:1997: ERROR:  failed to return
> resource to HAWQ resource manager because of failing to receive content.
> (pquery.c:983)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2150: WARNING:  Sync RPC
> framework (inet) finds exception raised.
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2150: ERROR:  failed to return
> resource to HAWQ resource manager because of failing to receive content.
> (pquery.c:983)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2455: ERROR:  insufficient
> memory reserved for statement (execHHashagg.c:732)  (seg12 dserver2:40404
> pid=331172) (dispatcher.c:1701)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2543: WARNING:  Sync RPC
> framework (inet) finds exception raised.
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2543: ERROR:  failed to return
> resource to HAWQ resource manager because of failing to receive content.
> (pquery.c:983)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2639: WARNING:  Sync RPC
> framework (inet) finds exception raised.
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2639: ERROR:  failed to return
> resource to HAWQ resource manager because of failing to receive content.
> (pquery.c:983)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2661: ERROR:  failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2720: ERROR:  failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2742: ERROR:  failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2794: ERROR:  failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2902: ERROR:  failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2935: ERROR:  failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:3182: WARNING:  Sync RPC
> framework (inet) finds exception raised.
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:3182: ERROR:  failed to return
> resource to HAWQ resource manager because of failing to receive content.
> (pquery.c:983)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:3286: ERROR:  failed to acquire
> resource because of too many unavailable