Re: vectorized executor

[Hongxu Ma]

Hi Yuliang
Thanks for your attention to the progress of VE. Currently, we (pivotal) 
has no plans to support more operator.

But I think the VE framework is a good start, hope to improve it with 
community efforts.
So let's wait for some contributions from the community in future.

?? 04/08/2018 10:17,  :
> Hello, everyone:
>   
> I am very glad to see the new vectorized executor function in the 
> version of Apache HAWQ 2.4.0.0-incubating Release. A preliminary 
> understanding of the current support for vectorized scan, vectorized AGG, 
> will support more oprator in the future?

-- 
Regards,
Hongxu.

Re: [DISCUSS] Graduate Apache HAWQ (incubating) as a TLP

[Hongxu Ma]

es the key technological advantages of MPP database
>> with the scalability and convenience of Hadoop.
>>
>> NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>> Committee (PMC), to be known as the "Apache HAWQ Project",
>> be and hereby is established pursuant to Bylaws of the
>> Foundation; and be it further
>>
>> RESOLVED, that the Apache HAWQ Project be and hereby is
>> responsible for the creation and maintenance of software
>> related to Hadoop native SQL query engine that
>> combines the key technological advantages of MPP database
>> with the scalability and convenience of Hadoop;
>> and be it further
>>
>> RESOLVED, that the office of "Vice President, Apache HAWQ" be
>> and hereby is created, the person holding such office to
>> serve at the direction of the Board of Directors as the chair
>> of the Apache HAWQ Project, and to have primary responsibility
>> for management of the projects within the scope of
>> responsibility of the Apache HAWQ Project; and be it further
>>
>> RESOLVED, that the persons listed immediately below be and
>> hereby are appointed to serve as the initial members of the
>> Apache HAWQ Project:
>>
>>  * Alan Gates   
>>  * Alexander Denissov   
>>  * Amy Bai  
>>  * Atri Sharma  
>>  * Bhuvnesh Chaudhary   
>>  * Bosco
>>  * Chunling Wang
>>  * David Yozie  
>>  * Ed Espino
>>  * Entong Shen  
>>  * Foyzur Rahman
>>  * Goden Yao
>>  * Gregory Chase
>>  * Hong Wu  
>>  * Hongxu Ma
>>  * Hubert Zhang 
>>  * Ivan Weng
>>  * Jesse Zhang  
>>  * Jiali Yao
>>  * Jun Aoki 
>>  * Kavinder Dhaliwal
>>  * Lav Jain 
>>  * Lei Chang
>>  * Lili Ma  
>>  * Lirong Jian  
>>  * Lisa Owen
>>  * Ming Li  
>>  * Mohamed Soliman  
>>  * Newton Alex  
>>  * Noa Horn 
>>  * Oleksandr Diachenko  
>>  * Paul Guo 
>>  * Radar Da Lei 
>>  * Roman Shaposhnik 
>>  * Ruilong Huo  
>>  * Shivram Mani 
>>  * Shubham Sharma   
>>  * Tushar Pednekar  
>>  * Venkatesh Raghavan   
>>  * Vineet Goel  
>>  * Wen Lin  
>>  * Xiang Sheng  
>>  * Yi Jin   
>>  * Zhanwei Wang 
>>  * Zhenglin Tao 
>>
>> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Lei Chang
>> be appointed to the office of Vice President, Apache HAWQ, to
>> serve in accordance with and subject to the direction of the
>> Board of Directors and the Bylaws of the Foundation until
>> death, resignation, retirement, removal or disqualification,
>> or until a successor is appointed; and be it further
>>
>> RESOLVED, that the initial Apache HAWQ PMC be and hereby is
>> tasked with the creation of a set of bylaws intended to
>> encourage open development and increased participation in the
>> Apache HAWQ Project; and be it further
>>
>> RESOLVED, that the Apache HAWQ Project be and hereby
>> is tasked with the migration and rationalization of the Apache
>> Incubator HAWQ podling; and be it further
>>
>> RESOLVED, that all responsibilities pertaining to the Apache
>> Incubator HAWQ podling encumbered upon the Apache Incubator
>> Project are hereafter discharged.

-- 
Regards,
Hongxu.

Re: Vectorized Execution plugin lib has been released

[Hongxu Ma]

Good job, thank you very much. @Weinan @Shujie

Glad to see HAWQ has this modern feature and hope to continue to improve 
it with the community efforts.

在 19/06/2018 14:18, Weinan Wang 写道:
> Greetings,
>
> I am glad to announce that Vectorized Execution plugin has been completed 
> released in HAWQ.
>
> At present, it supports 7 types of data trigger the vectorized function and 
> reach two to eight times
> acceleration comparing with regular execution in different performance test 
> queries. Furthermore,
> it has an extended friendly framework, you can patch other types into as your 
> need.
>
> You can acquire the introduction document  as follow:
> https://cwiki.apache.org/confluence/display/HAWQ/Vectorized+Execution+Introduction
>  
> 
>
>
>
>
> Regards,
>
> WANG Weinan

-- 
Regards,
Hongxu.

[jira] [Created] (HAWQ-1627) Support setting the max protocol message size when talking with HDFS

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1627:
---

 Summary: Support setting the max protocol message size when 
talking with HDFS
 Key: HAWQ-1627
 URL: https://issues.apache.org/jira/browse/HAWQ-1627
 Project: Apache HAWQ
  Issue Type: Improvement
  Components: libhdfs
Reporter: Hongxu Ma
Assignee: Radar Lei
 Fix For: 2.4.0.0-incubating


Now, the max size of protocol message in libhdfs is 64MB, and it cannot be 
adjusted.
When the max size is reached (accessing a very big HDFS table/file), will see 
the following lines in hawq master log:
{code}
2018-06-20 11:21:56.768003 
CST,,,p75703,th-8481004160,,,seg-1,"LOG","0","3rd party error 
log:
[libprotobuf ERROR google/protobuf/io/coded_stream.cc:208] A protocol message 
was rejected because it was too big (more than 67108864 bytes).  To increase 
the limit (or to
disable these warnings), see CodedInputStream::SetTotalBytesLimit() in 
google/protobuf/io/coded_stream.h.""SysLoggerMain","syslogger.c",518,
2018-06-20 11:21:56.771657 
CST,,,p75703,th-8481004160,,,seg-1,"LOG","0","3rd party error 
log:
2018-06-20 11:21:56.771492, p75751, th0x7fffcd7303c0, ERROR Failed to invoke 
RPC call ""getFsStats"" on server ""localhost:9000"":
RpcChannel.cpp: 783: HdfsRpcException: RPC channel to ""localhost:9000"" got 
protocol mismatch: RPC channel cannot parse response header.
@   Hdfs::Internal::RpcChannelImpl::readOneResponse(bool)
@   Hdfs::Internal::RpcChannelImpl::checkOneResponse()
@   
Hdfs::Internal::RpcChannelImpl::invokeInternal(std::__1::shared_ptr)
@   
Hdfs::Internal::RpcChannelImpl:""SysLoggerMain","syslogger.c",518,
2018-06-20 11:21:56.771711 
CST,,,p75703,th-8481004160,,,seg-1,"LOG","0","3rd party error 
log:
:invoke(Hdfs::Internal::RpcCall const&)
@   Hdfs::Internal::NamenodeImpl::invoke(Hdfs::Internal::RpcCall const&)
@   Hdfs::Internal::NamenodeImpl::getFsStats()
@   Hdfs::Internal::NamenodeProxy::getFsStats()
@   Hdfs::Internal::FileSystemImpl::getFsStats()
@   Hdfs::Internal::FileSystemImpl::connect()
@   Hdfs::FileSystem::connect(char const*, char const*, char const*)
@   Hdfs::FileSystem::connect(char const*)
@   hdfsBuilderConnect
@   gpfs_hdfs_connect
@   HdfsConnect
@   HdfsGetConnection
@   HdfsGetFileBlockLocati""SysLoggerMain","syslogger.c",518,
{code}

Considering HDFS has a guc *"ipc.maximum.data.length"* to set it, HAWQ also 
should has a guc about it.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Graduate Apache HAWQ (incubating) as a TLP

[Hongxu Ma]

tion of the Board of Directors as the chair
> of the Apache HAWQ Project, and to have primary responsibility
> for management of the projects within the scope of
> responsibility of the Apache HAWQ Project; and be it further
>
> RESOLVED, that the persons listed immediately below be and
> hereby are appointed to serve as the initial members of the
> Apache HAWQ Project:
>
>  * Alan Gates   
>  * Alexander Denissov   
>  * Amy Bai  
>  * Atri Sharma  
>  * Bhuvnesh Chaudhary   
>  * Bosco
>  * Chunling Wang
>  * David Yozie  
>  * Ed Espino
>  * Entong Shen  
>  * Foyzur Rahman
>  * Goden Yao
>  * Gregory Chase
>  * Hong Wu  
>  * Hongxu Ma
>  * Hubert Zhang 
>  * Ivan Weng
>  * Jesse Zhang  
>  * Jiali Yao
>  * Jun Aoki 
>  * Kavinder Dhaliwal
>  * Lav Jain 
>  * Lei Chang
>  * Lili Ma  
>  * Lirong Jian  
>  * Lisa Owen
>  * Ming Li  
>  * Mohamed Soliman  
>  * Newton Alex  
>  * Noa Horn 
>  * Oleksandr Diachenko  
>  * Paul Guo 
>  * Radar Da Lei 
>  * Roman Shaposhnik 
>  * Ruilong Huo  
>  * Shivram Mani 
>  * Shubham Sharma   
>  * Tushar Pednekar  
>  * Venkatesh Raghavan   
>  * Vineet Goel  
>  * Wen Lin  
>  * Xiang Sheng  
>  * Yi Jin   
>  * Zhanwei Wang 
>  * Zhenglin Tao 
>
> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Lei Chang
> be appointed to the office of Vice President, Apache HAWQ, to
> serve in accordance with and subject to the direction of the
> Board of Directors and the Bylaws of the Foundation until
> death, resignation, retirement, removal or disqualification,
> or until a successor is appointed; and be it further
>
> RESOLVED, that the initial Apache HAWQ PMC be and hereby is
> tasked with the creation of a set of bylaws intended to
> encourage open development and increased participation in the
> Apache HAWQ Project; and be it further
>
> RESOLVED, that the Apache HAWQ Project be and hereby
> is tasked with the migration and rationalization of the Apache
> Incubator HAWQ podling; and be it further
>
> RESOLVED, that all responsibilities pertaining to the Apache
> Incubator HAWQ podling encumbered upon the Apache Incubator
> Project are hereafter discharged.
>

-- 
Regards,
Hongxu.

[jira] [Created] (HAWQ-1618) Segment panic at workfile_mgr_close_file() when transaction ROLLBACK

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1618:
---

 Summary: Segment panic at workfile_mgr_close_file() when 
transaction ROLLBACK
 Key: HAWQ-1618
 URL: https://issues.apache.org/jira/browse/HAWQ-1618
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Query Execution
Reporter: Hongxu Ma
Assignee: Lei Chang
 Fix For: 2.4.0.0-incubating


Log:

{code}

2018-05-23 15:49:14.843058 
UTC,"user","db",p179799,th401824032,"172.31.6.17","6935",2018-05-23 15:47:39 
UTC,1260445558,con25148,cmd7,seg21,slice82,,x1260445558,sx1,"ERROR","25M01","*canceling
 MPP operation*",,"INSERT INTO ...

2018-05-23 15:49:15.253671 UTC,,,p179799,th0,,,2018-05-23 15:47:39 
UTC,0,con25148,cmd7,seg21,slice82"PANIC","XX000","Unexpected internal 
error: Segment process r

eceived signal SIGSEGV",,,0"1    0x8ce2a3 postgres gp_backtrace + 0xa3

2    0x8ce491 postgres  + 0x8ce491

3    0x7f2d147ae7e0 libpthread.so.0  + 0x147ae7e0

4    0x91f4ad postgres workfile_mgr_close_file + 0xd

5    0x90bc84 postgres  + 0x90bc84

6    0x4e6b60 postgres AbortTransaction + 0x240

7    0x4e75c5 postgres AbortCurrentTransaction + 0x25

8    0x7ed81a postgres PostgresMain + 0x6ea

9    0x7a0c50 postgres  + 0x7a0c50

10   0x7a3a19 postgres PostmasterMain + 0x759

11   0x4a5309 postgres main + 0x519

12   0x7f2d13cead1d libc.so.6 __libc_start_main + 0xfd

13   0x4a5389 postgres  + 0x4a5389"

{code}

 

Core stack:

{code}

(gdb) bt

#0  0x7f2d147ae6ab in raise () from libpthread.so.0

#1  0x008ce552 in SafeHandlerForSegvBusIll (postgres_signal_arg=11, 
processName=) at elog.c:4573

#2  

#3  *workfile_mgr_close_file* (work_set=0x0, file=0x7f2ce96d2de0, 
canReportError=canReportError@entry=0 '\000') at workfile_file.c:129

#4  0x0090bc84 in *ntuplestore_cleanup* (fNormal=0 '\000', 
canReportError=0 '\000', ts=0x21f4810) at tuplestorenew.c:654

#5  XCallBack_NTS (event=event@entry=XACT_EVENT_ABORT, nts=nts@entry=0x21f4810) 
at tuplestorenew.c:674

#6  0x004e6b60 in CallXactCallbacksOnce (event=) at 
xact.c:3660

#7  AbortTransaction () at xact.c:2871

#8  0x004e75c5 in AbortCurrentTransaction () at xact.c:3377

#9  0x007ed81a in PostgresMain (argc=, argv=, argv@entry=0x182c900, username=0x17ddcd0 "user") at postgres.c:4648

#10 0x007a0c50 in BackendRun (port=0x17cfb10) at postmaster.c:5915

#11 BackendStartup (port=0x17cfb10) at postmaster.c:5484

#12 ServerLoop () at postmaster.c:2163

#13 0x007a3a19 in PostmasterMain (argc=, argv=) at postmaster.c:1454

#14 0x004a5309 in main (argc=9, argv=0x1785d10) at main.c:226

{code}

 

Repro:

{code}

# create test table

drop table if exists testsisc; 
create table testsisc (i1 int, i2 int, i3 int, i4 int); 
insert into testsisc select i, i % 1000, i % 10, i % 75 from 
generate_series(0,1) i;


drop table if exists to_insert_into; 
create table to_insert_into as 
with ctesisc as 
 (select count(i1) as c1,i3 as c2 from testsisc group by i3)
select t1.c1 as c11, t1.c2 as c12, t2.c1 as c21, t2.c2 as c22
from ctesisc as t1, ctesisc as t2
where t1.c1 = t2.c2
limit 10;

# run a long time query

begin;

set gp_simex_run=on;
set gp_cte_sharing=on;

insert into to_insert_into
with ctesisc as 
 (select count(i1) as c1,i3 as c2 from testsisc group by i3)
select *
from ctesisc as t1, ctesisc as t2
where t1.c1 = t2.c2;

commit;

{code}

Kill one segment process when the second query is running. Then will find panic 
log in segment log.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Remain with HAWQ project or not?

[Hongxu Ma]

thanks Radar!

在 26/05/2018 00:26, Ed Espino 写道:
> Radar, Nice work coordinating this important project task!
>
> 谢谢 (Thank you),
> -=e
>
> On Fri, May 25, 2018 at 12:18 AM, Radar Lei <r...@pivotal.io> wrote:
>
>> Thanks all for your responses! It's really great to see so many people
>> willing to remain with HAWQ project!
>>
>> Now we will end this thread since more than two weeks passed.
>>
>> As a summary of all the responses:
>>
>> - We have 42 committers are willing to remain with HAWQ project.
>> - There are 19 committers either respond 'NO' or do not have responses.
>> - We will have two mentors join the PMC: Roman and Alan have confirmed
>> to stay for helping after graduation.
>>
>>
>> Below is the detail list of who will remain in Apache HAWQ project as
>> committer/PMC members:
>>
>> Alan Gates
>> Alexander Denissov
>> Amy Bai
>> Atri Sharma
>> Bhuvnesh Chaudhary
>> Chunling Wang
>> David Yozie
>> Ed Espino
>> Entong Shen
>> Foyzur Rahman
>> Goden Yao
>> Gregory Chase
>> Hong Wu
>> Hongxu Ma
>> Hubert Zhang
>> Ivan Weng
>> Jesse Zhang
>> Jiali Yao
>> Jun Aoki
>> Kavinder Dhaliwal
>> Lav Jain
>> Lei Chang
>> Lili Ma
>> Lirong Jian
>> Lisa Owen
>> Ming Li
>> Mohamed Soliman
>> Newton Alex
>> Noa Horn
>> Oleksandr Diachenko
>> Paul Guo
>> Radar Da Lei
>> Roman Shaposhnik
>> Ruilong Huo
>> Shivram Mani
>> Shubham Sharma
>> Tushar Pednekar
>> Venkatesh Raghavan
>> Vineet Goel
>> Wen Lin
>> Xiang Sheng
>> Yi Jin
>> Zhanwei Wang
>> Zhenglin Tao
>>
>> Please feel free to let me know if you have any questions with this result.
>> Thanks.
>>
>> Regards,
>> Radar
>>
>> On Tue, May 22, 2018 at 2:43 PM, Radar Lei <r...@pivotal.io> wrote:
>>
>>> Forward responds I received in individual mails in recent days:
>>>
>>> Yes:
>>> Vineet Goel
>>> Jesse Zhang
>>>
>>> No:
>>> Owen O'Malley
>>>
>>>
>>>
>>> Regards,
>>> Radar
>>>
>>> On Mon, May 21, 2018 at 12:20 PM, Greg Chase <gch...@gmail.com> wrote:
>>>
>>>> Yes :-)
>>>>
>>>> This email encrypted by tiny buttons & fat thumbs, beta voice
>>>> recognition, and autocorrect on my iPhone.
>>>>
>>>>> On May 14, 2018, at 9:41 PM, Radar Lei <r...@pivotal.io> wrote:
>>>>>
>>>>> FWD responds I received in individual mails:
>>>>>
>>>>> Yes:
>>>>> Goden Yao
>>>>> Kavinder Dhaliwal
>>>>> Mohamed Soliman
>>>>> Newton Alex
>>>>> Oleksandr Diachenko
>>>>> Tushar Pednekar
>>>>> Venkatesh Raghavan
>>>>> David Yozie
>>>>>
>>>>> No:
>>>>> Mel Kiyama
>>>>>
>>>>>
>>>>> Regards,
>>>>> Radar
>>>>>
>>>>>> On Mon, May 14, 2018 at 6:18 AM, Entong Shen <ent...@apache.org>
>>>> wrote:
>>>>>> Yes.
>>>>>>
>>>>>>> On 2018/05/07 08:11:53, Radar Lei <r...@pivotal.io> wrote:
>>>>>>> HAWQ committers,
>>>>>>>
>>>>>>> Per the discussion in "Apache HAWQ graduation from incubator?" [1],
>> we
>>>>>> want
>>>>>>> to setup the PMC as part of HAWQ graduation resolution.
>>>>>>>
>>>>>>> So we'd like to confirm whether you want to remain as a
>> committer/PMC
>>>>>>> member of Apache HAWQ project?
>>>>>>>
>>>>>>> If you'd like to remain with HAWQ project, it's welcome and please
>>>>>> *respond**
>>>>>>> 'Yes'* in this thread, or *respond 'No'* if you are not interested
>> in
>>>> any
>>>>>>> more. Thanks.
>>>>>>>
>>>>>>> This thread will be available for at least 72 hours, after that, we
>>>> will
>>>>>>> send individual confirm emails.
>>>>>>>
>>>>>>> [1]
>>>>>>> https://lists.apache.org/thread.html/b4a0b5671ce377b3d51c9b7ab00496
>>>>>> a1eebfcbf1696ce8b67e078c64@%3Cdev.hawq.apache.org%3E
>>>>>>> Regards,
>>>>>>> Radar
>>>>>>>
>>>

-- 
Regards,
Hongxu.

Re: Remain with HAWQ project or not?

[Hongxu Ma]

Yes, let's make HAWQ better.

Thanks.

在 07/05/2018 16:11, Radar Lei 写道:
> HAWQ committers,
>
> Per the discussion in "Apache HAWQ graduation from incubator?" [1], we want
> to setup the PMC as part of HAWQ graduation resolution.
>
> So we'd like to confirm whether you want to remain as a committer/PMC
> member of Apache HAWQ project?
>
> If you'd like to remain with HAWQ project, it's welcome and please *respond**
> 'Yes'* in this thread, or *respond 'No'* if you are not interested in any
> more. Thanks.
>
> This thread will be available for at least 72 hours, after that, we will
> send individual confirm emails.
>
> [1]
> https://lists.apache.org/thread.html/b4a0b5671ce377b3d51c9b7ab00496a1eebfcbf1696ce8b67e078c64@%3Cdev.hawq.apache.org%3E
>
> Regards,
> Radar
>

-- 
Regards,
Hongxu.

Re: New Committer: Lav Jain

[Hongxu Ma]

Congrats!

在 25/04/2018 03:26, Shivram Mani 写道:
> The Project Management Committee (PMC) for Apache HAWQ (incubating)
> has invited *Lav Jain* to become a committer and we are pleased to
> announce that he has accepted.
> Being a committer enables easier contribution to the project since
> there is no need to go via the patch submission process. This should
> enable better productivity.
> Please join us in congratulating him and we are looking forward to
> collaborating with him in the open source community.
> His contribution includes (but not limited to):
>
>
> His contribution includes (but not limited to):
>- *12 commits* with some major components in hawq involved,
> including contributions to *PXF* and* Ranger Plugin Service*
>- Numerous other contributions including documentation, test, build,
> command line
> tools, code refactor, etc
>- *300+ commits* to HAWQ and PXF *CI and testing*
>
> HAWQ-1584. Propogate bridge endIteration exception during write
> HAWQ-1580. Set PXF port number to 5888 for GPDB
> HAWQ-1545. Fix pxf-private.classpath file for Cloudera tarball distro
> HAWQ-1533. Fix compiler warnings when building PXF code
> HAWQ-1523. Provide ability to bundle PXF artifacts into a single tarbal
> HAWQ-1492. Enable packaging for PXF JDBC plugin.
> HAWQ-1461. Improve partition parameters validation for PXF-JDBC plugin.
> HAWQ-1454. Exclude certain jars from Ranger Plugin Service packaging
> HAWQ-1415. Set the default_value of JAVA_HOME for running RPS
> HAWQ-1420. Enable build of Ranger plugin open source RPM
> HAWQ-1415. Set the default_value of JAVA_HOME for running RPS
>
> **JIRA*:*
> *11 JIRAs* were created or assigned to Lav
> 10 JIRAs are closed and 1 JIRA is still open
> https://issues.apache.org/jira/browse/HAWQ-1492?jql=
> project%20%3D%20HAWQ%20AND%20(assignee%20in%20(lavjain)%
> 20OR%20reporter%20in%20(lavjain))
>
> *Related Contributions*
> Lav has also been a very active contributor to the *HAWQ Ambari project*
> *40 JIRAs* were created or assigned to Lav
> https://issues.apache.org/jira/browse/AMBARI-21377?jql=
> project%20%3D%20AMBARI%20AND%20(assignee%20in%20(lavjain)%
> 20OR%20reporter%20in%20(lavjain))
>
>
>
>

-- 
Regards,
Hongxu.

Re: New Committer: Shubham Sharma

[Hongxu Ma]

Congrats!

在 25/04/2018 03:28, Shivram Mani 写道:
> The Project Management Committee (PMC) for Apache HAWQ (incubating)
> has invited *Shubham Sharma* to become a committer and we are pleased
> to announce that he has accepted.
> Being a committer enables easier contribution to the project since
> there is no need to go via the patch submission process. This should
> enable better productivity.
> Please join us in congratulating him and we are looking forward to
> collaborating with him in the open source community.
>
>
> His contribution includes (but not limited to):
>
>
>
> 15 commits to components in hawq, including contributions to PXF and master
> branch CI
> - Reference - https://github.com/apache/incubator-hawq/pulls?utf8=%E2%
> 9C%93=is%3Apr+author%3Aoutofmem0ry
>
> HAWQ-1572: Fix travis ci build failure. Thrift/Boost incompatibility
> HAWQ-1559: Fix travis ci build failure due to missing apr package
> HAWQ-1553: Add option to hawq extract to specify log directory
> HAWQ-1368: Add option to hawq register to specify log directory
> HAWQ-1549: Fix standby resync when using -M smart/fast/immediate
> HAWQ-1548: Correct ambiguous message while logging hawq utilization
> HAWQ-1527: Enabled partition filtering for integral data types
> HAWQ-1524:  Fix travis ci build failure caused after protobuf upgrade to
> 3.4
> HAWQ-1198: Fix to filter out irrelevant fragments while accessing Hive
> HAWQ-1273:  Removed incorrect references from gplogfilter
> HAWQ-1504:  Fixed namenode hang during docker container restart
> HAWQ-1503: Fixed ftp link causing build failure on centos-6 docker
> HAWQ-1495: Corrected answer file to match insert statement
> HAWQ-1373: Added feature to reload GUC values using hawq reload-config
> HAWQ-1480: Added feature for packing a core file
>
> 4 commits to hawq documentation (https://github.com/apache/incubator-hawq
> -docs/pulls?utf8=%E2%9C%93=is%3Apr+author%3Aoutofmem0ry)
>   - Two closed and two open pull requests
>
> HAWQ-1562: Fixed incorrect path to default log directory
> HAWQ-1368: Add option to hawq register to specify log directory
> HAWQ-1480: Added documentation for packcore utility
> HAWQ-1553: Add option to hawq extract to specify log directory
>
> *JIRA:*
> 12 JIRAs were created by Shubham. Pull requests for all 13 issues has been
> merged to master.
> https://issues.apache.org/jira/browse/HAWQ-1572?jql=
> project%20%3D%20HAWQ%20AND%20reporter%20in%20(outofmemory)
> 3 JIRAs were assigned to Shubham. Pull requests for these issues has been
> merged to master. [HAWQ-1198, HAWQ-1368, HAWQ-1373 ]
>
> Other contributions include code review(https://github.com/apache/incubator-
> hawq/pull/1318) and email discussions.
>

-- 
Regards,
Hongxu.

Re: Apache HAWQ 2.4.0.0-incubating Release Plan/Timeframe

[Hongxu Ma]

Thanks Radar, wonderful features!
 

在 02/04/2018 16:19, Radar Lei 写道:
> Hi Apache HAWQ community,
>
> We delivered latest Apache HAWQ release two weeks ago, I think it's time to
> get the next release moving along.
>
> I drafted a plan for the release Apache HAWQ 2.4.0.0-incubating, now it's
> open for discussion, please help to provide feedback.
>
> Release Version:
>
> Apache HAWQ 2.4.0.0-incubating
>
>
> Plan Scope:
>
> New Feature: Pluggable Vectorized Execution Engine on HAWQ.
> New Feature: Support Runtime Filter for HAWQ local hash join.
> New Feature: Support accessing Hive table data by Pluggable Storage
> Framework.
> Bug fixes.
>
>
> Release Type:
>
> Source release and binary release.
>
>
> Timeline:
>
> Target to end of June 2018 : branch cut
> Target to end of July 2018 : release
>
>
>
> For details, you can check in the wiki page:
>
> https://cwiki.apache.org/confluence/display/HAWQ/Apache+HAWQ+2.4.0.0-incubating+Release
>
>
> Please feel free to discuss and provide feedback, thanks.
>
> Regards,
> Radar
>

-- 
Regards,
Hongxu.

Re: [ANNOUNCE] Apache HAWQ 2.3.0.0-incubating Release

[Hongxu Ma]

+1 for radar,  

在 23/03/2018 10:04, Radar Lei 写道:
> Thanks Yi for your great work on Apache HAWQ 2.3.0.0-incubating release.
>
> I would like to volunteer to be next release manager of Apache HAWQ.
>
> As a committer, I used to work close with previous excellent release
> managers to prepare the releases, now I want to contribute more on Apache
> HAWQ releases. I hope we can keep HAWQ release regularly and always keep
> Apache HAWQ project in a healthy status. Thanks.
>
>
> Regards,
> Radar
>
> On Fri, Mar 23, 2018 at 8:48 AM, Yi JIN  wrote:
>
>> Thanks everyone for making this real, especially the help from Ruilong and
>> Radar!
>>
>> To make it a more successful apache project and community, we need to keep
>> the release cadence. Who would like to be volunteer for the next
>> release manager
>> and drive the effort? Thanks.
>>
>> Best,
>> Yi (yjin)
>>
>> On Fri, Mar 23, 2018 at 11:40 AM, Lei Chang 
>> wrote:
>>
>>> congrats!
>>>
>>> Cheers
>>> Lei
>>>
>>>
>>>
>>> On Wed, Mar 21, 2018 at 1:58 PM, Yi JIN  wrote:
>>>
 Thanks John, I fixed it and resent the announcement.

 Best,
 Yi (yjin)

 On Wed, Mar 21, 2018 at 4:14 PM, Amy Bai  wrote:

> Congratulations! Thanks Yi for this release!
>
> On Wed, Mar 21, 2018 at 11:15 AM, John D. Ament <
>> johndam...@apache.org
> wrote:
>
>> Dropping announce@.
>>
>> On Tue, Mar 20, 2018 at 10:28 PM Yi JIN  wrote:
>>
>>> Apache HAWQ (incubating) Project Team is proud to announce Apache
>>> HAWQ 2.3.0.0-incubating has been released.
>>>
>>> Apache HAWQ (incubating) combines exceptional MPP-based analytics
>>> performance, robust ANSI SQL compliance, Hadoop ecosystem
>>> integration and manageability, and flexible data-store format
>>> support, all natively in Hadoop, no connectors required. Built
>>> from a decade’s worth of massively parallel processing (MPP)
>>> expertise developed through the creation of the Pivotal
>>> Greenplum® enterprise database and open source PostgreSQL, HAWQ
>>> enables to you to swiftly and interactively query Hadoop data,
>>> natively via HDFS.
>>>
>>> *Download Link*:
>>>
>>> https://dist.apache.org/repos/dist/release/incubator/hawq/2.
>> 3.0.0-incubating/
>>>
>> This has been a hot topic as of late.  This link is incorrect.
>>> Please
>> review http://www.apache.org/dev/release-publishing#
>>> distribution_dist
> and
>> related pages and fix the link.  Please resend this email once
>> you've
>> corrected it.
>>
>>
>>> *About this release*
>>> This is a release having both source code and binary
>>>
>>> All changes:
>>>
>>> https://cwiki.apache.org/confluence/display/HAWQ/
 Apache+HAWQ+2.3.0.0-
>> incubating+Release
>>>
>>> *HAWQ Resources:*
>>>
>>> - JIRA: https://issues.apache.org/jira/browse/HAWQ
>>> - Wiki:
>>> https://cwiki.apache.org/confluence/display/HAWQ/
>> Apache+HAWQ+Home
>>> - Mailing list(s): dev@hawq.incubator.apache.org
>>>u...@hawq.incubator.apache.org
>>>
>>> *Know more about HAWQ:*
>>> http://hawq.apache.org
>>>
>>> - Apache HAWQ (incubating) Team
>>>
>>> =
>>> *Disclaimer*
>>>
>>> Apache HAWQ (incubating) is an effort undergoing incubation at
>> The
>>> Apache Software Foundation (ASF), sponsored by the name of Apache
>>> Incubator PMC. Incubation is required of all newly accepted
>>> projects until a further review indicates that the
>>> infrastructure, communications, and decision making process have
>>> stabilized in a manner consistent with other successful ASF
>>> projects. While incubation status is not necessarily a reflection
>>> of the completeness or stability of the code, it does indicate
>>> that the project has yet to be fully endorsed by the ASF.
>>>

-- 
Regards,
Hongxu.

Re: [ANNOUNCE] Apache HAWQ 2.3.0.0-incubating Release

[Hongxu Ma]

Congrats!
Thanks Yi~

在 21/03/2018 10:27, Yi JIN 写道:
> Apache HAWQ (incubating) Project Team is proud to announce Apache
> HAWQ 2.3.0.0-incubating has been released.
>
> Apache HAWQ (incubating) combines exceptional MPP-based analytics
> performance, robust ANSI SQL compliance, Hadoop ecosystem
> integration and manageability, and flexible data-store format
> support, all natively in Hadoop, no connectors required. Built
> from a decade’s worth of massively parallel processing (MPP)
> expertise developed through the creation of the Pivotal
> Greenplum® enterprise database and open source PostgreSQL, HAWQ
> enables to you to swiftly and interactively query Hadoop data,
> natively via HDFS.
>
> *Download Link*:
> https://dist.apache.org/repos/dist/release/incubator/hawq/2.3.0.0-incubating/
>
> *About this release*
> This is a release having both source code and binary
>
> All changes:
> https://cwiki.apache.org/confluence/display/HAWQ/Apache+HAWQ+2.3.0.0-incubating+Release
>
>
> *HAWQ Resources:*
>
> - JIRA: https://issues.apache.org/jira/browse/HAWQ
> - Wiki: https://cwiki.apache.org/confluence/display/HAWQ/Apache+HAWQ+Home
> - Mailing list(s): dev@hawq.incubator.apache.org
>u...@hawq.incubator.apache.org
>
> *Know more about HAWQ:*
> http://hawq.apache.org
>
> - Apache HAWQ (incubating) Team
>
> =
> *Disclaimer*
>
> Apache HAWQ (incubating) is an effort undergoing incubation at The
> Apache Software Foundation (ASF), sponsored by the name of Apache
> Incubator PMC. Incubation is required of all newly accepted
> projects until a further review indicates that the
> infrastructure, communications, and decision making process have
> stabilized in a manner consistent with other successful ASF
> projects. While incubation status is not necessarily a reflection
> of the completeness or stability of the code, it does indicate
> that the project has yet to be fully endorsed by the ASF.
>

-- 
Regards,
Hongxu.

[jira] [Created] (HAWQ-1594) Memory leak in standby master (gpsyncagent process)

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1594:
---

 Summary: Memory leak in standby master (gpsyncagent process)
 Key: HAWQ-1594
 URL: https://issues.apache.org/jira/browse/HAWQ-1594
 Project: Apache HAWQ
  Issue Type: Bug
Reporter: Hongxu Ma
Assignee: Radar Lei
 Fix For: backlog


In a high workload scenario, the gpsyncagent process of standby master consumes 
memory continuously until restart it.

There are some Memory leak happened.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE]: Apache HAWQ 2.3.0.0-incubating Release (RC2)

[Hongxu Ma]

+1

Both source and rpm package are verified in my environment: Red Hat 
Enterprise Linux Server release 7.2
Include installation and execute some simple queries.

在 27/02/2018 12:19, Yi JIN 写道:
> Hi All,
>
> This is the vote for Apache HAWQ (incubating) 2.3.0.0-incubating Release
> Candidate 2 (RC2). It is a source release for HAWQ core, PXF, and Ranger;
> and binary release for HAWQ core,  PXF and Ranger. We have rpm package
> involved for the binary release.
>
> The vote will run for at least 72 hours and will close on Saturday, March
> 3rd, 2017. Thanks.
>
> 1. Wiki page of the release:
> *https://cwiki.apache.org/confluence/display/HAWQ/Apache+HAWQ+2.3.0.0-incubating+Release
> *
>
>
> 2. Release Notes (Apache Jira generated):
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> version=12340262=Html=12318826
>
>
> 3. Release verification steps can be found at:
> For source tarball: https://cwiki.apache.org/confluence/display/HAWQ/
> Release+Process%3A+Step+by+step+guide#ReleaseProcess:
> Stepbystepguide-ValidatetheReleaseCandidate
> For rpm package: https://cwiki.apache.org/confluence/display/HAWQ/
> Build+Package+and+Install+with+RPM
>
>
> 4. Git release branch:
> https://git-wip-us.apache.org/repos/asf?p=incubator-hawq.
> git;a=shortlog;h=refs/heads/2.3.0.0-incubating
>
> 5. Source and Binary release balls with signare:
> https://dist.apache.org/repos/dist/dev/incubator/hawq/2.3.0.
> 0-incubating.RC2/
>
>
> 6. Keys to verify the signature of the release artifact are available at:
> https://dist.apache.org/repos/dist/dev/incubator/hawq/KEYS
>
>
> 7. The artifact(s) has been signed with Key ID: CE60F90D1333092A
>
> 8. Fixed issues in RC2.
> https://issues.apache.org/jira/browse/HAWQ-1589
> https://issues.apache.org/jira/browse/HAWQ-1590
>
> REMINDER: Please provide details of what you have tried and verified before
> your vote conclusion. Thanks!
>
>
> Please vote accordingly:
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove (and reason why)
>
>
> Best regards,
> Yi (yjin)
>

-- 
Regards,
Hongxu.

Re: [jira] [Created] (HAWQ-1591) Common tuple batch structure for vectorized execution

[Hongxu Ma]

Thank you very much! Lirong

在 27/02/2018 11:31, Ivan Weng 写道:
> Thanks Lirong. It's a good suggestion. Will evaluate the benefit and cost
> before making decision.
>
> Also I think it's better to make the tuple batch interfaces more general
> and its internal format could be replaced if possible.
>
>
> Regards,
> Ivan
>
> On Mon, Feb 26, 2018 at 6:59 PM, Lirong Jian <jianlir...@gmail.com> wrote:
>
>> Have you guy consider using Apache Arrow format (http://arrow.apache.org/)
>> as the in-memory tuple batch structure for vectorized execution? I think
>> the goal of Apache Arrow project matches the one of vectorized execution
>> perfectly, and its community is quite active, although the implementation
>> of Apache Arrow format is far away from the implementation of
>> PostgreSQL/GPDB/HAWQ tuple structure , which means the engineering efforts
>> needed would be huge.
>>
>> Just my two cents.
>>
>> Lirong
>>
>> Lirong Jian
>> HashData Inc.
>>
>> 2018-02-26 13:40 GMT+08:00 Hongxu Ma (JIRA) <j...@apache.org>:
>>
>>> Hongxu Ma created HAWQ-1591:
>>> ---
>>>
>>>   Summary: Common tuple batch structure for vectorized
>> execution
>>>   Key: HAWQ-1591
>>>           URL: https://issues.apache.org/jira/browse/HAWQ-1591
>>>   Project: Apache HAWQ
>>>Issue Type: Sub-task
>>>Components: Query Execution
>>>  Reporter: Hongxu Ma
>>>  Assignee: Lei Chang
>>>   Fix For: backlog
>>>
>>>
>>> A common tuple batch structure for vectorized execution, holds the tuples
>>> which be transfered between vectorized operators.
>>>
>>>
>>>
>>>
>>>
>>> --
>>> This message was sent by Atlassian JIRA
>>> (v7.6.3#76005)
>>>

-- 
Regards,
Hongxu.

[jira] [Created] (HAWQ-1591) Common tuple batch structure for vectorized execution

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1591:
---

 Summary: Common tuple batch structure for vectorized execution
 Key: HAWQ-1591
 URL: https://issues.apache.org/jira/browse/HAWQ-1591
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: Query Execution
Reporter: Hongxu Ma
Assignee: Lei Chang
 Fix For: backlog


A common tuple batch structure for vectorized execution, holds the tuples which 
be transfered between vectorized operators.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE]: Apache HAWQ 2.3.0.0-incubating Release (RC1)

[Hongxu Ma]

+1

Both source and rpm package are verified in my environment: Red Hat 
Enterprise Linux Server release 7.2
Include installation and execute some simple queries.

在 20/02/2018 10:42, Yi JIN 写道:
> Hi All,
>
> This is the vote for Apache HAWQ (incubating) 2.3.0.0-incubating Release
> Candidate 1 (RC1). It is a source release for HAWQ core, PXF, and Ranger;
> and binary release for HAWQ core,  PXF and Ranger. We have rpm package
> involved for the binary release.
>
> The vote will run for at least 72 hours and will close on Saturday, Feb 24,
> 2017. Thanks.
>
> 1. Wiki page of the release:
> *https://cwiki.apache.org/confluence/display/HAWQ/Apache+HAWQ+2.3.0.0-incubating+Release
> *
>
>
> 2. Release Notes (Apache Jira generated):
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12340262=Html=12318826
>
>
> 3. Release verification steps can be found at:
> For source tarball: https://cwiki.apache.org/confluence/display/HAWQ
> /Release+Process%3A+Step+by+step+guide#ReleaseProcess:Stepbystepguide-
> ValidatetheReleaseCandidate
> For rpm package: https://cwiki.apache.org/confluence/display/HAWQ
> /Build+Package+and+Install+with+RPM
>
>
> 4. Git release branch:
> https://git-wip-us.apache.org/repos/asf?p=incubator-hawq.git;a=shortlog;h=refs/heads/2.3.0.0-incubating
>
> 5. Source and Binary release balls with signare:
> https://dist.apache.org/repos/dist/dev/incubator/hawq/2.3.0.0-incubating.RC1/
>
>
> 6. Keys to verify the signature of the release artifact are available at:
> https://dist.apache.org/repos/dist/dev/incubator/hawq/KEYS
>
>
> 7. The artifact(s) has been signed with Key ID: CE60F90D1333092A
>
>
> Please vote accordingly:
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove (and reason why)
>
>
> Best regards,
> Yi (yjin)
>

-- 
Regards,
Hongxu.

[jira] [Created] (HAWQ-1583) Add vectorized executor extension and GNC

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1583:
---

 Summary: Add vectorized executor extension and GNC
 Key: HAWQ-1583
 URL: https://issues.apache.org/jira/browse/HAWQ-1583
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: Query Execution
Reporter: Hongxu Ma
Assignee: Lei Chang
 Fix For: backlog


The vectorized executor will be implemented as a extension (located at contrib 
directory).

And use a GNC to enable vectorized executor, e.g:

{code}

postgres=# set vectorized_executor_enable to on;

// run the new vectorized executor

postgres=# set vectorized_executor_enable to off;

// run the original HAWQ executor


{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Podling Report Reminder - January 2018

[Hongxu Ma]

Hi Lei
The format is broken...

在 02/01/2018 14:38, Lei Chang 写道:

Hi Guys,

The following is the Podling Report. Please review and give your comments.

Cheers
Lei





HAWQHAWQ is an advanced enterprise SQL on Hadoop analytic engine built
around arobust and high-performance massively-parallel processing
(MPP) SQL frameworkevolved from Greenplum Database.HAWQ has been
incubating since 2015-09-04.Three most important issues to address in
the move towardsgraduation:  1. Continue to improve the project's
release cadence. To this end we plan on expanding automation
services to support increased developer participation.Any issues
that the Incubator PMC (IPMC) or ASF Board wish/needto be aware of?
 Nothing urgent at this time.How has the community developed since the
last report?1. Conference Talks (2): * The nature of cloud database.
The 7th Data Technology Carnival (Speaker: Lei Chang, Nov 17, 2017)  *
New Data Warehouse: Apache HAWQ. 2017 Global Internet Technology
Conference (Speaker: Lei Chang, Nov 24, 2017) 2. Active contributions
from approximately 20 different community   contributors since the
last report.3. Three committer candidates passed the voting process:
1) Amy BAI   2) ChunLing WANG3) Hongxu MA   How has the project
developed since the last report?1. The scope of 2.3 release is
finalized, and is under development  1) New Feature: HAWQ Ranger
supports RPS HA.  (Done)  2) New Feature: HAWQ Ranger supports
Kerberos authentication. (Done)  3) New Feature: HAWQ Core supports
plugable external storage framework. (Almost Done HAWQ-786)  4) New
Feature: HAWQ Core supports HDFS TDE (Transparent Data Encryption)
through libHdfs3. (Done, HAWQ-1193)  5) Licenses: Fix PXF license
files located in PXF jar files. (Done HAWQ-1496)  6) Licenses: Check
Apache HAWQ mandatory libraries to match LC20, LC30 license criteria.
(Not started HAWQ-1512)  7) Build: Release build project (On going
HAWQ-127)  8) Bug fixes. (On going)Project page link:
https://cwiki.apache.org/confluence/display/HAWQ/Apache+HAWQ+2.3.0.0-incubating+Release
How would you assess the podling's maturity?Please feel free to
add your own commentary.  [ ] Initial setup  [ ] Working towards first
release  [ ] Community building  [X] Nearing graduation  [ ]
Other:Date of last release:  2017-07-12, Apache HAWQ 2.2.0.0  When
were the last committers or PPMC members elected?   1) Amy BAI:  Nov
1, 2017   2) ChunLing WANG: Nov 1, 2017   3) Hongxu MA: Nov 4,
2017Signed-off-by:  [ ](hawq) Alan Gates Comments:  [ ](hawq)
Konstantin Boudnik Comments:  [ ](hawq) Justin Erenkrantz
Comments:  [ ](hawq) Thejas Nair Comments:  [ ](hawq) Roman
Shaposhnik Comments: IPMC/Shepherd notes:


On Sat, Dec 30, 2017 at 10:30 PM, 
<johndam...@apache.org><mailto:johndam...@apache.org> wrote:



Dear podling,

This email was sent by an automated system on behalf of the Apache
Incubator PMC. It is an initial reminder to give you plenty of time to
prepare your quarterly board report.

The board meeting is scheduled for Wed, 17 January 2018, 10:30 am PDT.
The report for your podling will form a part of the Incubator PMC
report. The Incubator PMC requires your report to be submitted 2 weeks
before the board meeting, to allow sufficient time for review and
submission (Wed, January 03).

Please submit your report with sufficient time to allow the Incubator
PMC, and subsequently board members to review and digest. Again, the
very latest you should submit your report is 2 weeks prior to the board
meeting.

Thanks,

The Apache Incubator PMC

Submitting your Report

--

Your report should contain the following:

*   Your project name
*   A brief description of your project, which assumes no knowledge of
the project or necessarily of its field
*   A list of the three most important issues to address in the move
towards graduation.
*   Any issues that the Incubator PMC or ASF Board might wish/need to be
aware of
*   How has the community developed since the last report
*   How has the project developed since the last report.
*   How does the podling rate their own maturity.

This should be appended to the Incubator Wiki page at:

https://wiki.apache.org/incubator/January2018

Note: This is manually populated. You may need to wait a little before
this page is created from a template.

Mentors
---

Mentors should review reports for their project(s) and sign them off on
the Incubator wiki page. Signing off reports shows that you are
following the project - projects that are not signed may raise alarms
for the Incubator PMC.

Incubator PMC







--
Regards,
Hongxu.

[jira] [Created] (HAWQ-1544) prompt file count doesn't match hash bucket number when reorganize table

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1544:
---

 Summary: prompt file count doesn't match hash bucket number when 
reorganize table
 Key: HAWQ-1544
 URL: https://issues.apache.org/jira/browse/HAWQ-1544
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Catalog
Reporter: Hongxu Ma
Assignee: Radar Lei


Reproduce:
{code}
postgres=# create table t_vm ( i int ) distributed randomly;
CREATE TABLE
postgres=# insert into t_vm select generate_series(1,1);
INSERT 0 1
postgres=# show default_hash_table_bucket_number ;
 default_hash_table_bucket_number
--
 6
(1 row)

postgres=# set default_hash_table_bucket_number=8;
SET
postgres=# show default_hash_table_bucket_number ;
 default_hash_table_bucket_number
--
 8
(1 row)

postgres=# alter table t_vm set with(reorganize=true) distributed by (i) ;
ALTER TABLE
postgres=# select count(*) from t_vm;
ERROR:  file count 8 in catalog is not in proportion to the bucket number 6 of 
hash table with oid=16619, some data may be lost, if you still want to continue 
the query by considering the table as random, set GUC 
allow_file_count_bucket_num_mismatch to on and try again. 
(cdbdatalocality.c:3776)
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: New committer: Hongxu Ma

[Hongxu Ma]

Thanks all nice buddy!
It's a great honor for me.

在 02/11/2017 00:05, Lei Chang 写道:
> Congrats!
>
> Cheers
> Lei
>
>
> On Wed, Nov 1, 2017 at 11:33 PM, Brian Lu <b...@pivotal.io> wrote:
>
>> Congrats Hongxu!
>>
>> Best Regards,
>> Brian
>>
>> On Wed, Nov 1, 2017 at 10:44 PM, Shubham Sharma <ssha...@pivotal.io>
>> wrote:
>>
>>> Congrats Hongxu !
>>>
>>> On Wed, Nov 1, 2017 at 3:15 AM, stanly sheng <stanly.sxi...@gmail.com>
>>> wrote:
>>>
>>>> Congrats, Hongxu!
>>>>
>>>> 2017-11-01 14:16 GMT+08:00 Hubert Zhang <hzh...@pivotal.io>:
>>>>
>>>>> Congrats to Hongxu!
>>>>>
>>>>> On Wed, Nov 1, 2017 at 2:06 PM, Radar Lei <r...@pivotal.io> wrote:
>>>>>
>>>>>> The Project Management Committee (PMC) for Apache HAWQ (incubating)
>>> has
>>>>>> invited Hongxu Ma to become a committer and we are pleased to
>>> announce
>>>>> that
>>>>>> he has accepted.
>>>>>> Being a committer enables easier contribution to the project since
>>>> there
>>>>> is
>>>>>> no need to go via the patch submission process. This should enable
>>>> better
>>>>>> productivity.
>>>>>> Please join us in congratulating him and we are looking forward to
>>>>>> collaborating with him in the open source community.
>>>>>>
>>>>>> His contribution includes (but not limited to):
>>>>>>
>>>>>> *Direct contribution to code base:*
>>>>>>
>>>>>> - 26 commits in total with some major components in hawq
>> involved,
>>>>>> including contributions to Apache Ranger integration, TDE
>> support
>>>>>> and command line tools.
>>>>>> *https://github.com/apache/incubator-hawq/commits?author=interma
>>>>>> <https://github.com/apache/incubator-hawq/commits?author=
>> interma
>>>> *
>>>>>> - 27 closed PRs:
>>>>>> *https://github.com/apache/incubator-hawq/pulls?utf8=%E2%
>>>>>> 9C%93=is%3Apr%20is%3Aclosed%20author%3Ainterma
>>>>>> <https://github.com/apache/incubator-hawq/pulls?utf8=%E2%
>>>>>> 9C%93=is%3Apr%20is%3Aclosed%20author%3Ainterma>*
>>>>>> - 11 Jiras on Apache Ranger integration
>>>>>>- https://issues.apache.org/jira/browse/HAWQ-1493 Integrate
>>>> Ranger
>>>>>>lookup JAAS configuration in ranger-admin plugin jar
>>>>>>- https://issues.apache.org/jira/browse/HAWQ-1485 Use
>>>>> user/password
>>>>>>instead of credentials cache in Ranger lookup for HAWQ with
>>>>> Kerberos
>>>>>>enabled.
>>>>>>- https://issues.apache.org/jira/browse/HAWQ-1477
>>> Ranger-plugin
>>>>>>connect to Ranger admin under kerberos security.
>>>>>>- https://issues.apache.org/jira/browse/HAWQ-1456 Copy RPS
>>>>>>configuration files to standby in specific scenarios
>>>>>>- https://issues.apache.org/jira/browse/HAWQ-1436 Implement
>>> RPS
>>>>> High
>>>>>>availability on HAWQ
>>>>>>- https://issues.apache.org/jira/browse/HAWQ-1410 Add basic
>>> test
>>>>>> case
>>>>>>for hcatalog with ranger
>>>>>>- https://issues.apache.org/jira/browse/HAWQ-1405 'hawq stop
>>>>>>--reload' should not stop RPS
>>>>>>- https://issues.apache.org/jira/browse/HAWQ-1393 'hawq stop
>>>>>> cluster'
>>>>>>failed when rps.sh have some path errors (e.g. CATALINA_HOME)
>>>>>>- https://issues.apache.org/jira/browse/HAWQ-1365 Print out
>>>>> detailed
>>>>>>schema information for tables which the user doesn't have
>>>>> privileges
>>>>>>- https://issues.apache.org/jira/browse/HAWQ-1279 Force to
>>>>> recompute
>>>>>>namespace_path when enable_ranger
>>>>>>- https://issues.apache.org/jira/browse/HAWQ-1257 If user
>>>> doesn't
>>>>>>have privileges on ce

Re: Podling Report Reminder - October 2017

[Hongxu Ma]

Hi Dr.Chang
> 4) New Feature: HAWQ Core supports HDFS TDE (Transparent Data Encryption)
> through libHdfs3. (On going HAWQ-1193)
HAWQ-1193 is already finished now, we forgot close this jira before.
Thanks!

-- 
Regards,
Hongxu.

Re: Travis CI build failing

[Hongxu Ma]

I noted this issue yesterday and haven't find how to fix it yet...
Thanks Shubham & Ed!

But I think solution2 is better:
The root cause is code issue (low version protobuf make it passed by 
coincidence), let's open a PR to fix it, not many work.

Thanks.

在 31/08/2017 11:00, stanly sheng 写道:
> Hi Shubham,
>
> I prefer solution 1 for the longer stable status and this can avoid
> frequently updating code due to library changes.
>
> Thanks.
>
> 2017-08-31 8:51 GMT+08:00 Shubham Sharma :
>
>> Hi,
>>
>> The current Travis CI build is failing as seen from
>> https://travis-ci.org/apache/incubator-hawq?branch=master
>>
>> Ed and I discussed about it, he found out that this might have happened due
>> to a recent protobuf update in brew. The last successful build had a
>> protobuf version of 3.2 and the latest(failing) is 3.4
>>
>> Am working on setting up a local dev branch on my mac to test the failure
>> due to upgrade(till now I have been using centos).
>>
>>  From the travis CI error message it looks like it is a problem with
>> namespace resolution while declaring stringstream and ostringstream
>>
>> Error message -
>>
>> /Users/travis/build/apache/incubator-hawq/depends/
>> libyarn/src/libyarnclient/LibYarnClient.cpp:248:9:
>> error: unknown type name 'stringstream'; did you mean
>> 'std::stringstream'?
>>
>>  stringstream ss;
>>
>>  ^~~~
>>
>>  std::stringstream
>>
>> /Applications/Xcode.app/Contents/Developer/Toolchains/
>> XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/iosfwd:153:38:
>> note: 'std::stringstream' declared here
>>
>> typedef basic_stringstream stringstream;
>>
>> /Users/travis/build/apache/incubator-hawq/depends/
>> libyarn/src/libyarnclient/LibYarnClient.cpp:299:13:
>> error: unknown type name 'ostringstream'; did you mean
>> 'std::ostringstream'?
>>
>>  ostringstream key;
>>
>>  ^
>>
>>  std::ostringstream
>>
>> /Applications/Xcode.app/Contents/Developer/Toolchains/
>> XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/iosfwd:152:38:
>> note: 'std::ostringstream' declared here
>>
>> typedef basic_ostringstreamostringstream;
>>
>>
>>
>> Here are the possible solutions -
>>
>> 1 - To update travis.yml to use protobuf 3.2 only
>> 2 -  Use namespace while declaring stringstream or ostringstream
>>
>> Also, trying to get to the root cause of the problem, if it is due to
>> protobuf upgrade, the only change that I see in protobuf regarding std
>> namespace is, in the recent version they moved it out of common/stubs.h -
>> https://github.com/google/protobuf/blob/master/CHANGES.txt#L61, not sure
>> if
>> this is the problem but only thing I could find relative.
>>
>> Will appreciate your input on this.
>>
>> --
>> Regards,
>> Shubham
>>
>
>

-- 
Regards,
Hongxu.

[jira] [Created] (HAWQ-1520) gpcheckhdfs should skip hdfs trash directory

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1520:
---

 Summary: gpcheckhdfs should skip hdfs trash directory
 Key: HAWQ-1520
 URL: https://issues.apache.org/jira/browse/HAWQ-1520
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: Command Line Tools
Reporter: Hongxu Ma
Assignee: Radar Lei
 Fix For: 2.3.0.0-incubating


When enable hdfs trash feature, there is a *Trash* directory under the 
encryption zone.
Example:

{code}
[gpadmin@test1 hawq]$ sudo -u hdfs hdfs crypto -listZones
/hawq/hawq-1503886333  tdekey
[gpadmin@test1 hawq]$ hdfs dfs -ls /hawq/hawq-1503886333/
Found 1 items
drwxrwxrwt   - hdfs hdfs  0 2017-08-27 23:59 
/hawq/hawq-1503886333/.Trash
{code}

But gpcheckhdfs scrpit doesn't consider it(/hawq/hawq-1503886333) as an empty 
directory.
We should fix it to skip this trash directory.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HAWQ-1518) Add a GUC for showing whether the data directory is an encryption zone

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1518:
---

 Summary: Add a GUC for showing whether the data directory is an 
encryption zone
 Key: HAWQ-1518
 URL: https://issues.apache.org/jira/browse/HAWQ-1518
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: Catalog
Reporter: Hongxu Ma
Assignee: Radar Lei
 Fix For: 2.3.0.0-incubating


A GUC(read only) for showing whether the data directory is an encryption zone.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HAWQ-1511) Add TDE-related properties into function-test.xml

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1511:
---

 Summary: Add TDE-related properties into function-test.xml
 Key: HAWQ-1511
 URL: https://issues.apache.org/jira/browse/HAWQ-1511
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: libhdfs
Reporter: Hongxu Ma
Assignee: Radar Lei
 Fix For: 2.3.0.0-incubating


include:
* dfs.encryption.key.provider.uri
* hadoop.security.crypto.buffer.size



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HAWQ-1510) Add TDE-related functionality into hawq command line tools

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1510:
---

 Summary: Add TDE-related functionality into hawq command line tools
 Key: HAWQ-1510
 URL: https://issues.apache.org/jira/browse/HAWQ-1510
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: Command Line Tools
Reporter: Hongxu Ma
Assignee: Radar Lei
 Fix For: 2.3.0.0-incubating


1, hawq init
the only way to enable tde in hawq:
user should give a key name(already created by hadoop key command) parameter 
when execuate the init command, it makes the whole hawq_default directory as an 
encryption zone.

note:
cannot support transfer the existed(and non-empty) hawq_default directory into 
an encryption zone.

2, hawq state
show the encryption zone info if user enable tde in hawq.

3, hawq register 
cannot register file in different encryption zones / un-encryption zones.

4, hawq extract
give user a warning of the table data is stored in encryption zone if user 
enable tde in hawq.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HAWQ-1509) Support TDE read function

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1509:
---

 Summary: Support TDE read function
 Key: HAWQ-1509
 URL: https://issues.apache.org/jira/browse/HAWQ-1509
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: libhdfs
Reporter: Hongxu Ma
Assignee: Radar Lei
 Fix For: 2.3.0.0-incubating


Currently, we have already supported TDE write.
Then will support TDE read in this JIRA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HAWQ-1508) Fix travis broken caused by libssl path

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1508:
---

 Summary: Fix travis broken caused by libssl path
 Key: HAWQ-1508
 URL: https://issues.apache.org/jira/browse/HAWQ-1508
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: libhdfs
Reporter: Hongxu Ma
Assignee: Radar Lei
 Fix For: 2.3.0.0-incubating


https://travis-ci.org/apache/incubator-hawq/builds/257989127?utm_source=github_status_medium=notification
{code}
CMake Error at 
/usr/local/Cellar/cmake/3.6.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:148
 (message):

  Could NOT find SSL (missing: SSL_INCLUDE_DIR)

Call Stack (most recent call first):

  
/usr/local/Cellar/cmake/3.6.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:388
 (_FPHSA_FAILURE_MESSAGE)

  CMake/FindSSL.cmake:24 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)

  CMakeLists.txt:24 (FIND_PACKAGE)

-- Configuring incomplete, errors occurred!

See also 
"/Users/travis/build/apache/incubator-hawq/depends/libhdfs3/build/CMakeFiles/CMakeOutput.log".

failed to configure the project

make[1]: *** [pre-config] Error 1

make: *** [all] Error 2
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HAWQ-1493) Integrate Ranger lookup JAAS configuration in ranger-admin plugin jar

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1493:
---

 Summary: Integrate Ranger lookup JAAS configuration in 
ranger-admin plugin jar
 Key: HAWQ-1493
 URL: https://issues.apache.org/jira/browse/HAWQ-1493
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: Security
Reporter: Hongxu Ma
Assignee: Radar Lei
 Fix For: 2.3.0.0-incubating


For support ranger lookup a kerberized HAWQ, we modify java.security file and 
add a .java.login.config file into ranger account now.
But both of the two files are global, have influenced other program, so we need 
integrate the JAAS configuration in a private scope.

After investigation, `setProperty("java.security.auth.login.config")` in 
ranger-admin plugin code is a good solution.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HAWQ-1485) Use user/password instead of credentials cache in Ranger lookup for HAWQ with Kerberos enabled.

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1485:
---

 Summary: Use user/password instead of credentials cache in Ranger 
lookup for HAWQ with Kerberos enabled.
 Key: HAWQ-1485
 URL: https://issues.apache.org/jira/browse/HAWQ-1485
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: Security
Reporter: Hongxu Ma
Assignee: Radar Lei
 Fix For: 2.3.0.0-incubating


When used credentials cache:
Try error password in Ranger UI doesn't destroy the existed kerberos 
credentials (created by last success kinit command)
It's a strange behavior to user.

So we should use user/password for kerberos authentication.
Core logic:
{code}
Properties props = new Properties();
if (connectionProperties.containsKey(AUTHENTICATION) && 
connectionProperties.get(AUTHENTICATION).equals(KERBEROS)) {
//kerberos mode
props.setProperty("kerberosServerName", 
connectionProperties.get("principal"));
props.setProperty("jaasApplicationName", "pgjdbc");
}

String url = String.format("jdbc:postgresql://%s:%s/%s", 
connectionProperties.get("hostname"), connectionProperties.get("port"), db);
props.setProperty("user", connectionProperties.get("username"));
props.setProperty("password", connectionProperties.get("password"));

return DriverManager.getConnection(url, props);
{code}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Apache HAWQ incubator - Travis CI is back to green state.

[Hongxu Ma]

Thanks ed!
It makes our "clean freak" guys happy. 


在 07/06/2017 06:15, Ed Espino 写道:
> HAWQ Dev,
>
> I have corrected the Apache HAWQ incubator Travis CI builds (
> https://travis-ci.org/apache/incubator-hawq/builds/240133714). Going
> forward, please make an effort to ensure both master and Pull Requests
> (PRs) builds remain in a "passing" state.
>
> Regards,
> -=e
>

-- 
Regards,
Hongxu.

[jira] [Created] (HAWQ-1477) Ranger-plugin connect to Ranger admin under kerberos security.

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1477:
---

 Summary: Ranger-plugin connect to Ranger admin under kerberos 
security.
 Key: HAWQ-1477
 URL: https://issues.apache.org/jira/browse/HAWQ-1477
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: Security
Reporter: Hongxu Ma
Assignee: Ed Espino
 Fix For: backlog


Ranger-plugin connect to Ranger admin server under kerberos security.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HAWQ-1456) Copy RPS configuration files to standby in specific scenarios

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1456:
---

 Summary: Copy RPS configuration files to standby in specific 
scenarios
 Key: HAWQ-1456
 URL: https://issues.apache.org/jira/browse/HAWQ-1456
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: Security
Reporter: Hongxu Ma
Assignee: Ed Espino
 Fix For: backlog


Copy rps releated configuration files under file /usr/local/hawq/ranger/etc/ 
from HAWQ master to standby master in specific scenarios.

* scenario 1: hawq init standby
may does not has a standby before, so should copy latest confguration files to 
standby.

* scenario 2: execute enable-ranger-plugin.sh 
this script will modify rps confguration files, so should copy updated 
confguration files to standby.





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HAWQ-1436) Implement RPS High availability on HAWQ

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1436:
---

 Summary: Implement RPS High availability on HAWQ
 Key: HAWQ-1436
 URL: https://issues.apache.org/jira/browse/HAWQ-1436
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: Security
Reporter: Hongxu Ma
Assignee: Ed Espino
 Fix For: backlog


Once Ranger is configured, HAWQ will rely on RPS to connect to Ranger. A single 
point RPS may influence the robustness of HAWQ. Thus We need to investigate and 
design out the way to implement RPS High availability. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HAWQ-1410) add basic test case for hcatalog with ranger

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1410:
---

 Summary: add basic test case for hcatalog with ranger
 Key: HAWQ-1410
 URL: https://issues.apache.org/jira/browse/HAWQ-1410
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: Security
Reporter: Hongxu Ma
Assignee: Ed Espino
 Fix For: 2.2.0.0-incubating


add test case:
basic operations on hcatalog with ranger

more pxf test with ranger will be added in future.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HAWQ-1405) 'hawq stop --reload' should not stop RPS

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1405:
---

 Summary: 'hawq stop --reload' should not stop RPS
 Key: HAWQ-1405
 URL: https://issues.apache.org/jira/browse/HAWQ-1405
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Command Line Tools
Reporter: Hongxu Ma
Assignee: Ed Espino
 Fix For: 2.2.0.0-incubating


Currently, running 'hawq stop cluster --reload', RPS is stopped, but not 
restarted.

We should not stop RPS when running 'hawq stop --reload'.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HAWQ-1393) 'hawq stop cluster' failed when rps.sh have some path errors (e.g. CATALINA_HOME)

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1393:
---

 Summary: 'hawq stop cluster' failed when rps.sh have some path 
errors (e.g. CATALINA_HOME)
 Key: HAWQ-1393
 URL: https://issues.apache.org/jira/browse/HAWQ-1393
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Command Line Tools
Reporter: Hongxu Ma
Assignee: Ed Espino
 Fix For: 2.2.0.0-incubating


example (hawq master already stopped)
{code}
$ hawq stop cluster -a
20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[INFO]:-Prepare to do 'hawq 
stop'
20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[INFO]:-You can find log in:
20170320:16:43:23:038375 
hawq_stop:interma-mbp:hma-[INFO]:-/Users/hma/hawqAdminLogs/hawq_stop_20170320.log
20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[INFO]:-GPHOME is set to:
20170320:16:43:23:038375 
hawq_stop:interma-mbp:hma-[INFO]:-/Users/hma/hawq/install
20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[INFO]:-Stop hawq with args: 
['stop', 'cluster']
20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[INFO]:-No standby host 
configured
20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[WARNING]:-Failed to connect 
to database, cannot get hawq_acl_type
20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[INFO]:-Stop hawq cluster
20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[WARNING]:-HAWQ process is 
not running on localhost, skip
20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[WARNING]:-Master is not 
running, skip
20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[INFO]:-Master stopped 
successfully
20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[WARNING]:-try to stop RPS 
when hawq_acl_type is unknown
20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[INFO]:-Stop Ranger plugin 
service
20170320:16:43:24:038375 hawq_stop:interma-mbp:hma-[INFO]:-Cannot find 
/usr/lib/bigtop-tomcat/bin/setclasspath.sh
This file is needed to run this program
FATAL: Failed to stop HAWQ Ranger Plugin Service. Check 
/Users/hma/hawq/install/ranger/plugin-service/logs/catalina.out for details.
20170320:16:43:24:038375 hawq_stop:interma-mbp:hma-[ERROR]:-Ranger plugin 
service stop failed, exit
{code}

We wish to continue to run the remain logic of hawq stop.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HAWQ-1385) hawq_ctl stop failed when master is down

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1385:
---

 Summary: hawq_ctl stop failed when master is down
 Key: HAWQ-1385
 URL: https://issues.apache.org/jira/browse/HAWQ-1385
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Command Line Tools
Reporter: Hongxu Ma
Assignee: Ed Espino
 Fix For: 2.2.0.0-incubating


example (master is down):
hawq activate standby
hawq stop allsegments

error output:
{code}
20170314:03:49:36:005905 
hawq_activate:centos7-datanode3:gpadmin-[INFO]:-20170314:03:49:36:005933 
hawq_stop:centos7-datanode3:gpadmin-[ERROR]:-Failed to connect to database, 
this script can only be run when the database is up
{code}




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HAWQ-1381) Core dump when execute 'select * from hawq_toolkit.__hawq_log_master_ext;' on macOS

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1381:
---

 Summary: Core dump when execute 'select * from 
hawq_toolkit.__hawq_log_master_ext;' on macOS
 Key: HAWQ-1381
 URL: https://issues.apache.org/jira/browse/HAWQ-1381
 Project: Apache HAWQ
  Issue Type: Bug
Reporter: Hongxu Ma
Assignee: Ed Espino
 Fix For: 2.2.0.0-incubating


macOS 10.12.1

{code}
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib  0x7fffded60dda __pthread_kill + 10
1   libsystem_pthread.dylib 0x7fffdee4c787 pthread_kill + 90
2   libsystem_c.dylib   0x7fffdecc6420 abort + 129
3   libsystem_c.dylib   0x7fffdecc6592 abort_report_np + 181
4   libsystem_c.dylib   0x7fffdececf28 __chk_fail + 48
5   libsystem_c.dylib   0x7fffdececef8 __chk_fail_overflow + 16
6   libsystem_c.dylib   0x7fffdeced413 __sprintf_chk + 199
7   postgres0x00010e39e394 external_set_env_vars + 
916
8   postgres0x00010e39c335 
open_external_readable_source + 181
9   postgres0x00010e39cba6 external_getnext + 70
10  postgres0x00010e62a42e ExternalNext + 110
11  postgres0x00010e606518 ExecScan + 72
12  postgres0x00010e62a3af ExecExternalScan + 31
13  postgres0x00010e5f4bd3 ExecProcNode + 739
14  postgres0x00010e5e88e2 ExecutePlan + 722
15  postgres0x00010e5e82af ExecutorRun + 1471
16  postgres0x00010e83531d PortalRunSelect + 317
17  postgres0x00010e834dc8 PortalRun + 952
18  postgres0x00010e82ad0f exec_simple_query + 2367
19  postgres0x00010e828eeb PostgresMain + 7979
20  postgres0x00010e7c4198 BackendRun + 1048
21  postgres0x00010e7c0f35 BackendStartup + 373
22  postgres0x00010e7bde10 ServerLoop + 1248
23  postgres0x00010e7bc3cb PostmasterMain + 4859
24  postgres0x00010e69f22c main + 940
25  libdyld.dylib   0x7fffdec32255 start + 1
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HAWQ-1380) Keep hawq_toolkit schema check in HAWQ native side

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1380:
---

 Summary: Keep hawq_toolkit schema check in HAWQ native side
 Key: HAWQ-1380
 URL: https://issues.apache.org/jira/browse/HAWQ-1380
 Project: Apache HAWQ
  Issue Type: Sub-task
Reporter: Hongxu Ma
Assignee: Ed Espino
 Fix For: 2.2.0.0-incubating


Given that *hawq_toolkit* is an administrative schema, let's keep it in HAWQ 
Native, outside of Ranger. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HAWQ-1365) Print out detailed schema information for tables which the user doesn't have privileges

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1365:
---

 Summary: Print out detailed schema information for tables which 
the user doesn't have privileges
 Key: HAWQ-1365
 URL: https://issues.apache.org/jira/browse/HAWQ-1365
 Project: Apache HAWQ
  Issue Type: Improvement
Reporter: Hongxu Ma
Assignee: Ed Espino
 Fix For: 2.2.0.0-incubating


Business Value:
Current output information for the table name which user doesn't have 
privileges doesn't include schema output.  We should print out the schema 
information out, otherwise it's difficult for users to identify the identified 
table.

{code}
postgres=# select * from public.a, s1.a;
ERROR:  permission denied for relation(s): a, a
{code}

Criteria:
Should print out the schema information for the table.
{code}
postgres=# select * from public.a, s1.a;
ERROR:  permission denied for relation(s): public.a, s1.a
{code}




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HAWQ-1286) Reduce all unnecessary calls of namespace check when run \d

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1286:
---

 Summary: Reduce all unnecessary calls of namespace check when run 
\d
 Key: HAWQ-1286
 URL: https://issues.apache.org/jira/browse/HAWQ-1286
 Project: Apache HAWQ
  Issue Type: Sub-task
Reporter: Hongxu Ma
Assignee: Ed Espino


After HAWQ-1279 is done, current schema is no cached in current session.
But it cause too many calls of namespace check to send in run \d , most of them 
are unnecessary (e.g. repeat check usage right of public schema).

So we should reduct those unnecessary calls.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-1279) Force to recompute namespace_path when enable_ranger

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1279:
---

 Summary: Force to recompute namespace_path when enable_ranger
 Key: HAWQ-1279
 URL: https://issues.apache.org/jira/browse/HAWQ-1279
 Project: Apache HAWQ
  Issue Type: Sub-task
Reporter: Hongxu Ma
Assignee: Ed Espino


namespace_path is cached in each psql session and it the cache invalidation is 
triggered by Grant/Revoke SQL.

When enable_ranger, Grant/Revoke SQL is no longer use, so the cache prevent a 
ack-check request sending.

Example:
```
// create table t(i int); => failed
[{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
[{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
[{""resource"":{""database"":""postgres"",""schema"":""public""},""privileges"":[""usage""],""allowed"":false}]

// grant usage and create permissions to public schema in ranger and try again 
=> failed again
[{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
[{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
// why not send a request for USAGE??
```



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-1193) TDE support in HAWQ

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1193:
---

 Summary: TDE support in HAWQ
 Key: HAWQ-1193
 URL: https://issues.apache.org/jira/browse/HAWQ-1193
 Project: Apache HAWQ
  Issue Type: New Feature
  Components: libhdfs
Reporter: Hongxu Ma
Assignee: Lei Chang
 Fix For: backlog


TDE(transparently data encrypted) has been supported after hadoop 2.6:
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html
https://issues.apache.org/jira/browse/HDFS-6134

Use tde can promise：
1, hdfs file is encrypted.
2, network transfer is encrypted.

So hawq will update libhdfs3 to support it.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-1194) Design document

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1194:
---

 Summary: Design document
 Key: HAWQ-1194
 URL: https://issues.apache.org/jira/browse/HAWQ-1194
 Project: Apache HAWQ
  Issue Type: Sub-task
Reporter: Hongxu Ma
Assignee: Lei Chang






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Help for HAWQ wiki/jira permission

[Hongxu Ma]

Hi everyone,

I dived into HAWQ recently, wish to share some ideas to wiki (e.g. build 
and install), and assign jira issues(e.g. HAWQ-513) to myself.


Can who give me the permission to edit access the HAWQ wiki/jira?

Thanks, below is my info:

JIRA:

 * username: hongxu ma
 * mail: inte...@outlook.com <mailto:inte...@outlook.com>

Confluence:

 * username: interma
 * mail: inte...@outlook.com <mailto:inte...@outlook.com>


--
Regards,
Hongxu.