[jira] [Resolved] (HAWQ-894) Add feature test for polymorphism with new test framework

[Yi Jin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yi Jin resolved HAWQ-894.
-
Resolution: Fixed

> Add feature test for polymorphism with new test framework
> -
>
> Key: HAWQ-894
> URL: https://issues.apache.org/jira/browse/HAWQ-894
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Tests
>Reporter: Lin Wen
>Assignee: Yi Jin
> Fix For: 2.0.1.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (HAWQ-894) Add feature test for polymorphism with new test framework

[Yi Jin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yi Jin closed HAWQ-894.
---

> Add feature test for polymorphism with new test framework
> -
>
> Key: HAWQ-894
> URL: https://issues.apache.org/jira/browse/HAWQ-894
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Tests
>Reporter: Lin Wen
>Assignee: Yi Jin
> Fix For: 2.0.1.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-974) PXF Bridge api should stream data as pages

[Oleksandr Diachenko (JIRA)]

Oleksandr Diachenko created HAWQ-974:


 Summary: PXF Bridge api should stream data as pages
 Key: HAWQ-974
 URL: https://issues.apache.org/jira/browse/HAWQ-974
 Project: Apache HAWQ
  Issue Type: Improvement
  Components: PXF
Reporter: Oleksandr Diachenko
Assignee: Goden Yao


As for now PXF Bridge api returns whole fragment, which might be big amount of 
data sometimes. So instead it should be able to stream fixed size pages.
Criteria of choosing page size - GC activity should be in some reasonable range 
0-5%.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq issue #820: HAWQ-953 hawq pxf-hive support partition column f...

[shivzone]

Github user shivzone commented on the issue:

https://github.com/apache/incubator-hawq/pull/820
  
Please refer to comment posted in jira 
https://issues.apache.org/jira/browse/HAWQ-779?focusedCommentId=15403051.
Combine all the required fixes under one PR against HAWQ-779. Sorry for the 
inconvenience.
-1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (HAWQ-779) support more pxf filter pushdwon

[Shivram Mani (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403051#comment-15403051
 ] 

Shivram Mani commented on HAWQ-779:
---

[~jiadx] Since all the issues are related to this jira. Please submit one pull 
request with all the required patches for CR against this jira.
Do not use HAWQ-953 or HAWQ-950 as they have already been reverted.

>  support more pxf filter pushdwon
> -
>
> Key: HAWQ-779
> URL: https://issues.apache.org/jira/browse/HAWQ-779
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: PXF
>Reporter: Devin Jia
>Assignee: Shivram Mani
> Fix For: 2.0.1.0-incubating
>
>
> When I use the pxf hawq, I need to read a traditional relational database 
> systems and solr by way of the external table. The project 
> :https://github.com/Pivotal-Field-Engineering/pxf-field/tree/master/jdbc-pxf-ext,
>  only "WriteAccessor ",so I developed 2 plug-ins, the projects: 
> https://github.com/inspur-insight/pxf-plugin , But these two plug-ins need to 
> modified HAWQ:
> 1. When get a list of fragment from pxf services, push down the 
> 'filterString'. modify the backend / optimizer / plan / createplan.c of 
> create_pxf_plan methods:
> segdb_work_map = map_hddata_2gp_segments (uri_str,
> total_segs, segs_participating,
> relation, ctx-> root-> parse-> jointree-> quals);
> 2. modify pxffilters.h and pxffilters.c, support TEXT types LIKE operation, 
> Date type data operator, Float type operator.
> 3. Modify org.apache.hawq.pxf.api.FilterParser.java, support the LIKE 
> operator.
> I already created a feature branch in my local ,and tested.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq pull request #829: HAWQ-972.Setup multi handle after setting ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-hawq/pull/829


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #829: HAWQ-972.Setup multi handle after setting all con...

[shivzone]

Github user shivzone commented on the issue:

https://github.com/apache/incubator-hawq/pull/829
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #830: HAWQ-973. Check pi_varList before adding proj hea...

[shivzone]

Github user shivzone commented on the issue:

https://github.com/apache/incubator-hawq/pull/830
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #830: HAWQ-973. Check pi_varList before adding proj hea...

[sansanichfb]

Github user sansanichfb commented on the issue:

https://github.com/apache/incubator-hawq/pull/830
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #830: HAWQ-973. Check pi_varList before adding p...

[kavinderd]

GitHub user kavinderd opened a pull request:

https://github.com/apache/incubator-hawq/pull/830

HAWQ-973. Check pi_varList before adding proj headers



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kavinderd/incubator-hawq HAWQ-973

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/830.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #830


commit 1f30d4139914f692cd42fc32915193edc0fa6505
Author: Kavinder Dhaliwal 
Date:   2016-08-01T22:53:40Z

HAWQ-973. Check pi_varList before adding proj headers




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #829: HAWQ-972.Setup multi handle after setting all con...

[sansanichfb]

Github user sansanichfb commented on the issue:

https://github.com/apache/incubator-hawq/pull/829
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #829: HAWQ-972.Setup multi handle after setting ...

[kavinderd]

GitHub user kavinderd opened a pull request:

https://github.com/apache/incubator-hawq/pull/829

HAWQ-972.Setup multi handle after setting all conf



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kavinderd/incubator-hawq HAWQ-972

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/829.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #829


commit 117bb5bf6607e35046622e4cc79b4b8cf3e0392a
Author: Kavinder Dhaliwal 
Date:   2016-08-01T22:24:19Z

HAWQ-972.Setup multi handle after setting all conf




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (HAWQ-958) LICENSE file missing checklist

[Goden Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Goden Yao updated HAWQ-958:
---
Description: 
>From [~jmclean] IPMC release VOTE feedback

{quote}
- Please use the short form of the license linking to a license files in LICENSE
- BSD licensed code [3] 3. ./tools/bin/pythonSrc/unittest2-0.5.1/setup.py  [GY] 
[~rlei] can you check this one as you're the last committer through HAWQ-837
- BSD license code [7] 7. ./depends/thirdparty/thrift/compiler/cpp/src/md5.? 
[GY] [~xunzhang] , this one was you through HAWQ-735
- license for this file [9] 9. ./src/backend/port/dynloader/ultrix4.h   [GY] 
original import, this one seems to be postgres license  plus a couple others.
- license for this file [10] Are we OK this was taken form GNU C? 10. 
./src/port/inet_aton.c   [GY] original import
- MIT license PSI [11] 11. ./tools/bin/pythonSrc/PSI-0.3b2_gp/ [GY] [~rlei] 
this is from HAWQ-837 a month ago, MIT license
- BSD licensed code [12] 12. ./src/port/snprintf.c
- BSD licensed code [13] 13 ./src/port/crypt.c  Is this regard as cryptography 
code? [14] 14. http://www.apache.org/dev/crypto.html
- BSD licensed code [15][16] 15. ./src/port/memcmp.c , 16. 
./src/backend/utils/mb/wstrcmp.c
- license for this file [17] 17. ./src/port/rand.c
- license of these files [18][19] 18. ./src/backend/utils/adt/inet_net_ntop.c
19. ./src/backend/utils/adt/inet_net_pton.c
- license of this file [20] 20 ./src/port/strlcpy.c
- regex license [21] 21. ./src/backend/regex/COPYRIGHT
- How are these files licensed? [22] + others copyright AEG Automation GmbH 22. 
./src/backend/port/qnx4/shm.c
- How is this file licensed? [23] 23. ./src/backend/port/beos/shm.c
- BSD licensed libpq [24]. 24. ./src/backend/libpq/sha2.?
Is this considered crypto code and may need an export license?
- pgdump [25] 25. ./src/bin/pg_dump/
- license for this file [26] 26. ./src/port/gettimeofday.c
- license for this file [27] Look like an ASF header may of been incorrectly 
added to this. 27. 
./depends/thirdparty/thrift/lib/cpp/src/thrift/windows/SocketPair.cpp
- This BSD licensed file [36] 36. ./src/bin/pg_controldata/pg_controldata.c
- license for these files [37][38] and others in [39]
37. ./depends/thirdparty/thrift/aclocal/ax_cxx_compile_stdcxx_11.m4
38. ./depends/thirdparty/thrift/aclocal/ax_boost_base.m4
39. ./depends/thirdparty/thrift/aclocal
- This BSD licensed file [40]
40. ./depends/thirdparty/thrift/build/cmake/FindGLIB.cmake
- This BSD licensed file [41]
41. ./tools/bin/pythonSrc/unittest2-0.5.1/setup.py
- BSD licensed pychecker [42]
42. ./tools/bin/pythonSrc/pychecker-0.8.18/
- licenses for all of these files [43]
43. ./src/interfaces/libpq/po/*.po
- BSD license pg800 [44]
44. ./tools/bin/ext/pg8000/*
- how is this file licensed? [45]
45. ./src/backend/utils/mb/Unicode/UCS_to_GB18030.pl
- license for this file [47] 47 
./tools/bin/pythonSrc/lockfile-0.9.1/lockfile/pidlockfile.py
- Python license for this file [48]. Is this an Apache comparable license? 48 
./tools/bin/pythonSrc/pychecker-0.8.18/pychecker2/symbols.py
- How are these files licensed? [49] Note multiple copyright owners and missing 
headers.
49.  ./src/backend/utils/mb/Unicode/*
- BSD licensed fig leaf. [50] Note that files incorrectly has had ASF headers 
applied.
50. ./tools/bin/ext/figleaf/*
- This BSD licensed file [51]
51. ./depends/thirdparty/thrift/lib/py/compat/win32/stdint.h
- This public domain style sheet [52]
52. ./tools/bin/pythonSrc/PyGreSQL-4.0/docs/default.css
- This file [53]
53. ./src/test/locale/test-ctype.c
- License for unit test2 [54]
54 ./tools/bin/pythonSrc/unittest2-0.5.1/unittest2/
- MIT licensed lock file [55]
55. ./tools/bin/pythonSrc/lockfile-0.9.1/LICENSE
- JSON code here [56]
56. ./src/include/catalog/JSON
- License for this file [57]
57. ./src/pl/plperl/ppport.h

Looks like GPL/LPGL licensed code may be included [4][5][6] in the release.
4. ./depends/thirdparty/thrift/debian/copyright (end of file)
5. ./depends/thirdparty/thrift/doc/licenses/lgpl-2.1.txt
6. ./tools/bin/gppylib/operations/test/test_package.py

This file [8] and others(?) may incorrectly have an ASF headers on it. Also why 
does this file have an ASF header with copyright line? [46]
8. ./tools/sbin/hawqstandbywatch.py
46. 
./contrib/hawq-hadoop/hawq-mapreduce-tool/src/test/resources/log4j.properties

Code includes code licensed under the 4 clause BSD license which is not 
compatible with the Apache 2.0 license. [28][29][30][31][32][33] It may be that 
this clause has been rescinded [35] and it is OK to include but that needs to 
be checked.
28. ./src/backend/port/dynloader/freebsd.c
29. ./src/backend/port/dynloader/netbsd.c
30. ./src/backend/port/dynloader/openbsd.c
31. ./src/bin/gpfdist/src/gpfdist/glob.c
32. ./src/bin/gpfdist/src/gpfdist/include/glob.h
33. ./src/include/port/win32_msvc/glob.h
34. ./src/port/glob.c -- [Goden] was not in original Justin's feedback but 
given the context, I 

[jira] [Updated] (HAWQ-973) Check pi_varList in pxfheaders before setting headers

[Kavinder Dhaliwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kavinder Dhaliwal updated HAWQ-973:
---
Description: "add_projection_desc_httpheader()" extracts the indices of 
columns to project from pi_varNumbers, but calculates the length based on 
pi_targetList. This is similar to how ExecVariableList handles projection. 
However there are cases where length(pi_targetList) >0 and 
length(pi_varNumbers) == 0, so a condition on pi_varList needs to be added 
before invoking add_projection_desc_httpheader as is done in ExecProject.  
(was: {code}
add_projection_desc_httpheader
{code}

extracts the indices of columns to project from pi_varNumbers, but calculates 
the length based on pi_targetList. This is similar to how ExecVariableList 
handles projection. However there are cases where length(pi_targetList) >0 and 
length(pi_varNumbers) == 0, so a condition on pi_varList needs to be added 
before invoking add_projection_desc_httpheader as is done in ExecProject.)

> Check pi_varList in pxfheaders before setting headers
> -
>
> Key: HAWQ-973
> URL: https://issues.apache.org/jira/browse/HAWQ-973
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: PXF
>Reporter: Kavinder Dhaliwal
>Assignee: Kavinder Dhaliwal
>
> "add_projection_desc_httpheader()" extracts the indices of columns to project 
> from pi_varNumbers, but calculates the length based on pi_targetList. This is 
> similar to how ExecVariableList handles projection. However there are cases 
> where length(pi_targetList) >0 and length(pi_varNumbers) == 0, so a condition 
> on pi_varList needs to be added before invoking 
> add_projection_desc_httpheader as is done in ExecProject.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HAWQ-973) Check pi_varList in pxfheaders before setting headers

[Kavinder Dhaliwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kavinder Dhaliwal reassigned HAWQ-973:
--

Assignee: Kavinder Dhaliwal  (was: Goden Yao)

> Check pi_varList in pxfheaders before setting headers
> -
>
> Key: HAWQ-973
> URL: https://issues.apache.org/jira/browse/HAWQ-973
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: PXF
>Reporter: Kavinder Dhaliwal
>Assignee: Kavinder Dhaliwal
>
> {code}
> add_projection_desc_httpheader
> {code}
> extracts the indices of columns to project from pi_varNumbers, but calculates 
> the length based on pi_targetList. This is similar to how ExecVariableList 
> handles projection. However there are cases where length(pi_targetList) >0 
> and length(pi_varNumbers) == 0, so a condition on pi_varList needs to be 
> added before invoking add_projection_desc_httpheader as is done in 
> ExecProject.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-973) Check pi_varList in pxfheaders before setting headers

[Kavinder Dhaliwal (JIRA)]

Kavinder Dhaliwal created HAWQ-973:
--

 Summary: Check pi_varList in pxfheaders before setting headers
 Key: HAWQ-973
 URL: https://issues.apache.org/jira/browse/HAWQ-973
 Project: Apache HAWQ
  Issue Type: Bug
  Components: PXF
Reporter: Kavinder Dhaliwal
Assignee: Goden Yao


{code}
add_projection_desc_httpheader
{code}

extracts the indices of columns to project from pi_varNumbers, but calculates 
the length based on pi_targetList. This is similar to how ExecVariableList 
handles projection. However there are cases where length(pi_targetList) >0 and 
length(pi_varNumbers) == 0, so a condition on pi_varList needs to be added 
before invoking add_projection_desc_httpheader as is done in ExecProject.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HAWQ-972) Curl upload field should be set before sending request

[Kavinder Dhaliwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kavinder Dhaliwal reassigned HAWQ-972:
--

Assignee: Kavinder Dhaliwal  (was: Goden Yao)

> Curl upload field should be set before sending request
> --
>
> Key: HAWQ-972
> URL: https://issues.apache.org/jira/browse/HAWQ-972
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables, PXF
>Reporter: Kavinder Dhaliwal
>Assignee: Kavinder Dhaliwal
>
> HAWQ-932 refactored out common code from curl_init_upload and 
> curl_init_download into curl_init, however curl_init now invokes 
> setup_multi_handle which performs the async curl request before 
> curl_init_upload sets the upload parameter to false. This leads to a 
> non-deterministic error where GET requests are made to PXF endpoints that 
> only accept POST requests leading to a 405 status return code from PXF.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-972) Curl upload field should be set before sending request

[Kavinder Dhaliwal (JIRA)]

Kavinder Dhaliwal created HAWQ-972:
--

 Summary: Curl upload field should be set before sending request
 Key: HAWQ-972
 URL: https://issues.apache.org/jira/browse/HAWQ-972
 Project: Apache HAWQ
  Issue Type: Bug
  Components: External Tables, PXF
Reporter: Kavinder Dhaliwal
Assignee: Goden Yao


HAWQ-932 refactored out common code from curl_init_upload and 
curl_init_download into curl_init, however curl_init now invokes 
setup_multi_handle which performs the async curl request before 
curl_init_upload sets the upload parameter to false. This leads to a 
non-deterministic error where GET requests are made to PXF endpoints that only 
accept POST requests leading to a 405 status return code from PXF.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-965) Error reporting is misleading with incorrect table/file in PXF location in HA cluster

[Goden Yao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402749#comment-15402749
 ] 

Goden Yao commented on HAWQ-965:


can you clarify where the incorrect table/file path is in your sample?
{code} pxf://singlecluster/... {code} is considered a format of HA uri (without 
port) in the current logic. 

> Error reporting is misleading with incorrect table/file in PXF location in HA 
> cluster
> -
>
> Key: HAWQ-965
> URL: https://issues.apache.org/jira/browse/HAWQ-965
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: PXF
>Reporter: Shivram Mani
>Assignee: Goden Yao
> Fix For: backlog
>
>
> When we create an external table with PXF protocol and use an incorrect path, 
> we get a deceiving error about Standby namenode.
> eg:
> {code}
> template1=# create external table hive_txt1 (a int, b text) location 
> ('pxf://singlecluster/testtxt?profile=Hive') format 'custom' 
> (formatter='pxfwritable_import');
> CREATE EXTERNAL TABLE
> template1=# select * from hive_txt1;  
>   
>   ERROR:  Standby NameNode of HA nameservice singlecluster was not found 
> after call to Active NameNode failed - failover aborted (pxfmasterapi.c:257)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-967) Extend Projection info to include filter attributes

[Goden Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Goden Yao updated HAWQ-967:
---
Description: 
HAWQ-927 includes query projection columns as part of the projection info sent 
from HAWQ to PXF.
For queries where filter attributes are different from projection attributes, 
PXF would return data with NULL values in the filter attributes.
e.g. a table "test" has 2 columns say: c1 int, c2 int
{code}
select c1 from test where c2 > 0;
{code}
In the case above, as c2 is not in column projection, pxf will return records 
like (1, NULL), (2, NULL) ... as part of the implementation in HAWQ-927 
Due to this HAWQ wouldn't have the necessary data to carry out filters once it 
receives data back from the underlying external dataset. via PXF and wrong 
result will be returned to users

The projection information must be a union of the internal HAWQ projection info 
and the attributes in the filters.

  was:
HAWQ-927 includes query projection columns as part of the projection info sent 
from HAWQ to PXF.
For queries where filter attributes are different from projection attributes, 
PXF would return data with NULL values in the filter attributes.
e.g.
{code}
select c1 from table where c2 > 0;
{code}

Due to this HAWQ wouldn't have the necessary data to carry out filters once it 
receives data back from the underlying external dataset. via PXF

The projection information must be a union of the internal HAWQ projection info 
and the attributes in the filters.


> Extend Projection info to include filter attributes
> ---
>
> Key: HAWQ-967
> URL: https://issues.apache.org/jira/browse/HAWQ-967
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: PXF
>Reporter: Shivram Mani
>Assignee: Oleksandr Diachenko
> Fix For: backlog
>
>
> HAWQ-927 includes query projection columns as part of the projection info 
> sent from HAWQ to PXF.
> For queries where filter attributes are different from projection attributes, 
> PXF would return data with NULL values in the filter attributes.
> e.g. a table "test" has 2 columns say: c1 int, c2 int
> {code}
> select c1 from test where c2 > 0;
> {code}
> In the case above, as c2 is not in column projection, pxf will return records 
> like (1, NULL), (2, NULL) ... as part of the implementation in HAWQ-927 
> Due to this HAWQ wouldn't have the necessary data to carry out filters once 
> it receives data back from the underlying external dataset. via PXF and wrong 
> result will be returned to users
> The projection information must be a union of the internal HAWQ projection 
> info and the attributes in the filters.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-967) Extend Projection info to include filter attributes

[Goden Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Goden Yao updated HAWQ-967:
---
Priority: Blocker  (was: Major)

> Extend Projection info to include filter attributes
> ---
>
> Key: HAWQ-967
> URL: https://issues.apache.org/jira/browse/HAWQ-967
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: PXF
>Reporter: Shivram Mani
>Assignee: Oleksandr Diachenko
>Priority: Blocker
> Fix For: backlog
>
>
> HAWQ-927 includes query projection columns as part of the projection info 
> sent from HAWQ to PXF.
> For queries where filter attributes are different from projection attributes, 
> PXF would return data with NULL values in the filter attributes.
> e.g. a table "test" has 2 columns say: c1 int, c2 int
> {code}
> select c1 from test where c2 > 0;
> {code}
> In the case above, as c2 is not in column projection, pxf will return records 
> like (1, NULL), (2, NULL) ... as part of the implementation in HAWQ-927 
> Due to this HAWQ wouldn't have the necessary data to carry out filters once 
> it receives data back from the underlying external dataset. via PXF and wrong 
> result will be returned to users
> The projection information must be a union of the internal HAWQ projection 
> info and the attributes in the filters.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-967) Extend Projection info to include filter attributes

[Goden Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Goden Yao updated HAWQ-967:
---
Description: 
HAWQ-927 includes query projection columns as part of the projection info sent 
from HAWQ to PXF.
For queries where filter attributes are different from projection attributes, 
PXF would return data with NULL values in the filter attributes.
e.g.
{code}
select c1 from table where c2 > 0;
{code}

Due to this HAWQ wouldn't have the necessary data to carry out filters once it 
receives data back from the underlying external dataset. via PXF

The projection information must be a union of the internal HAWQ projection info 
and the attributes in the filters.

  was:
HAWQ-927 includes query projection columns as part of the projection info sent 
from HAWQ to PXF.
For queries where filter attributes are different from projection attributes, 
PXF would return data with NULL values in the filter attributes.
Due to this HAWQ wouldn't have the necessary data to carry out filters once it 
receives data back from the underlying external dataset. via PXF

The projection information must be a union of the internal HAWQ projection info 
and the attributes in the filters.


> Extend Projection info to include filter attributes
> ---
>
> Key: HAWQ-967
> URL: https://issues.apache.org/jira/browse/HAWQ-967
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: PXF
>Reporter: Shivram Mani
>Assignee: Oleksandr Diachenko
> Fix For: backlog
>
>
> HAWQ-927 includes query projection columns as part of the projection info 
> sent from HAWQ to PXF.
> For queries where filter attributes are different from projection attributes, 
> PXF would return data with NULL values in the filter attributes.
> e.g.
> {code}
> select c1 from table where c2 > 0;
> {code}
> Due to this HAWQ wouldn't have the necessary data to carry out filters once 
> it receives data back from the underlying external dataset. via PXF
> The projection information must be a union of the internal HAWQ projection 
> info and the attributes in the filters.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HAWQ-971) Update PXF Resolvers to return partial content

[Shivram Mani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shivram Mani reassigned HAWQ-971:
-

Assignee: Shivram Mani  (was: Goden Yao)

> Update PXF Resolvers to return partial content
> --
>
> Key: HAWQ-971
> URL: https://issues.apache.org/jira/browse/HAWQ-971
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: PXF
>Reporter: Shivram Mani
>Assignee: Shivram Mani
> Fix For: backlog
>
>
> Most of the data formats on HDFS don't support projecting specific columns. 
> For such formats we can still avoid resolving every individual field in the 
> record and instead only resolve fields that need to be projected and send 
> NULLs for other fields.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-971) Update PXF Resolvers to return partial content

[Shivram Mani (JIRA)]

Shivram Mani created HAWQ-971:
-

 Summary: Update PXF Resolvers to return partial content
 Key: HAWQ-971
 URL: https://issues.apache.org/jira/browse/HAWQ-971
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: PXF
Reporter: Shivram Mani
Assignee: Goden Yao


Most of the data formats on HDFS don't support projecting specific columns. For 
such formats we can still avoid resolving every individual field in the record 
and instead only resolve fields that need to be projected and send NULLs for 
other fields.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-970) Provide More Accurate Information When LibYARN Meets an Exception

[Goden Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Goden Yao updated HAWQ-970:
---
Fix Version/s: backlog

> Provide More Accurate Information When LibYARN Meets an Exception
> -
>
> Key: HAWQ-970
> URL: https://issues.apache.org/jira/browse/HAWQ-970
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: libyarn
>Reporter: Lin Wen
>Assignee: Lei Chang
> Fix For: backlog
>
>
> Sometimes when an exception happens in libyarn, the log information is not 
> accurate enough. For example, below is an exception related to kerberos 
> ticket expiration, but we can't know from this log. 
> {code}
> 2016-07-06 01:47:51.945902 
> BST,,,p182375,th1403270400,con4,,seg-1,"WARNING","01000","YARN 
> mode resource broker failed to get container report. 
> LibYarnClient::getContainerReports, Catch the Exception:YarnIOException: 
> Unexpected exception: when calling ApplicationCl
> ientProtocol::getContainers in 
> /data1/pulse2-agent/agents/agent1/work/LIBYARN-main-opt/rhel5_x86_64/src/libyarnserver/ApplicationClientProtocol.cpp:
>  195",,,0,,"resourcebroker_LIBYARN_proc.c",1748,
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-970) Provide More Accurate Information When LibYARN Meets an Exception

[Goden Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Goden Yao updated HAWQ-970:
---
Description: 
Sometimes when an exception happens in libyarn, the log information is not 
accurate enough. For example, below is an exception related to kerberos ticket 
expiration, but we can't know from this log. 
{code}
2016-07-06 01:47:51.945902 
BST,,,p182375,th1403270400,con4,,seg-1,"WARNING","01000","YARN mode 
resource broker failed to get container report. 
LibYarnClient::getContainerReports, Catch the Exception:YarnIOException: 
Unexpected exception: when calling ApplicationCl
ientProtocol::getContainers in 
/data1/pulse2-agent/agents/agent1/work/LIBYARN-main-opt/rhel5_x86_64/src/libyarnserver/ApplicationClientProtocol.cpp:
 195",,,0,,"resourcebroker_LIBYARN_proc.c",1748,
{code}

  was:
Sometimes when an exception happens in libyarn, the log information is not 
accurate enough. For example, below is an exception related to kerberos ticket 
expiration, but we can't know from this log. 

2016-07-06 01:47:51.945902 
BST,,,p182375,th1403270400,con4,,seg-1,"WARNING","01000","YARN mode 
resource broker failed to get container report. 
LibYarnClient::getContainerReports, Catch the Exception:YarnIOException: 
Unexpected exception: when calling ApplicationCl
ientProtocol::getContainers in 
/data1/pulse2-agent/agents/agent1/work/LIBYARN-main-opt/rhel5_x86_64/src/libyarnserver/ApplicationClientProtocol.cpp:
 195",,,0,,"resourcebroker_LIBYARN_proc.c",1748,


> Provide More Accurate Information When LibYARN Meets an Exception
> -
>
> Key: HAWQ-970
> URL: https://issues.apache.org/jira/browse/HAWQ-970
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: libyarn
>Reporter: Lin Wen
>Assignee: Lei Chang
> Fix For: backlog
>
>
> Sometimes when an exception happens in libyarn, the log information is not 
> accurate enough. For example, below is an exception related to kerberos 
> ticket expiration, but we can't know from this log. 
> {code}
> 2016-07-06 01:47:51.945902 
> BST,,,p182375,th1403270400,con4,,seg-1,"WARNING","01000","YARN 
> mode resource broker failed to get container report. 
> LibYarnClient::getContainerReports, Catch the Exception:YarnIOException: 
> Unexpected exception: when calling ApplicationCl
> ientProtocol::getContainers in 
> /data1/pulse2-agent/agents/agent1/work/LIBYARN-main-opt/rhel5_x86_64/src/libyarnserver/ApplicationClientProtocol.cpp:
>  195",,,0,,"resourcebroker_LIBYARN_proc.c",1748,
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-966) Adjust Libyarn Output Log

[Goden Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Goden Yao updated HAWQ-966:
---
Fix Version/s: 2.0.1.0-incubating

> Adjust Libyarn Output Log
> -
>
> Key: HAWQ-966
> URL: https://issues.apache.org/jira/browse/HAWQ-966
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: libyarn
>Reporter: Lin Wen
>Assignee: Lin Wen
> Fix For: 2.0.1.0-incubating
>
>
> While HAWQ is running, libyarn generates a lot of logs. Some of them are 
> useless or duplicate to HAWQ users, should be compressed or reduced, so that 
> more meaningful log message can be provided for HAWQ users. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-965) Error reporting is misleading with incorrect table/file in PXF location in HA cluster

[Goden Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Goden Yao updated HAWQ-965:
---
Fix Version/s: backlog

> Error reporting is misleading with incorrect table/file in PXF location in HA 
> cluster
> -
>
> Key: HAWQ-965
> URL: https://issues.apache.org/jira/browse/HAWQ-965
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: PXF
>Reporter: Shivram Mani
>Assignee: Goden Yao
> Fix For: backlog
>
>
> When we create an external table with PXF protocol and use an incorrect path, 
> we get a deceiving error about Standby namenode.
> eg:
> {code}
> template1=# create external table hive_txt1 (a int, b text) location 
> ('pxf://singlecluster/testtxt?profile=Hive') format 'custom' 
> (formatter='pxfwritable_import');
> CREATE EXTERNAL TABLE
> template1=# select * from hive_txt1;  
>   
>   ERROR:  Standby NameNode of HA nameservice singlecluster was not found 
> after call to Active NameNode failed - failover aborted (pxfmasterapi.c:257)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-968) Incorrect free in url_close

[Goden Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Goden Yao updated HAWQ-968:
---
Fix Version/s: backlog

> Incorrect free in url_close
> ---
>
> Key: HAWQ-968
> URL: https://issues.apache.org/jira/browse/HAWQ-968
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables
>Affects Versions: backlog
>Reporter: hongwu
>Assignee: Lei Chang
> Fix For: backlog
>
>
> There is potential double free risk in url_close of url.c



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq issue #806: HAWQ-938. Remove ivy.xml in gpopt and read orca v...

[hsyuan]

Github user hsyuan commented on the issue:

https://github.com/apache/incubator-hawq/pull/806
  
pushed to master:

https://github.com/apache/incubator-hawq/commit/78c2766578102501e67c133df942fc6355d56c8e


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #800: HAWQ-936. Add GUC for array expansion in O...

[hsyuan]

Github user hsyuan closed the pull request at:

https://github.com/apache/incubator-hawq/pull/800


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #806: HAWQ-938. Remove ivy.xml in gpopt and read...

[hsyuan]

Github user hsyuan closed the pull request at:

https://github.com/apache/incubator-hawq/pull/806


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #800: HAWQ-936. Add GUC for array expansion in ORCA opt...

[hsyuan]

Github user hsyuan commented on the issue:

https://github.com/apache/incubator-hawq/pull/800
  
pushed to master:

https://github.com/apache/incubator-hawq/commit/950065bb0db627992be6b96afb4671b3c5ab1c90


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #799: HAWQ-934. Populate canSetTag of PlannedStm...

[hsyuan]

Github user hsyuan closed the pull request at:

https://github.com/apache/incubator-hawq/pull/799


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #799: HAWQ-934. Populate canSetTag of PlannedStmt from ...

[hsyuan]

Github user hsyuan commented on the issue:

https://github.com/apache/incubator-hawq/pull/799
  
Pushed to master:

https://github.com/apache/incubator-hawq/commit/eae0585f6109534c7ead553604c44b1dd483d8a3


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #795: HAWQ-860. Fix ORCA wrong plan when correlated sub...

[hsyuan]

Github user hsyuan commented on the issue:

https://github.com/apache/incubator-hawq/pull/795
  
Pushed to master:

https://github.com/apache/incubator-hawq/commit/e10f9a4e17f3012bb5afb98c53172dedefc8854e

https://github.com/apache/incubator-hawq/commit/4e08f10fc87301682721d6bffcb74a188ac7b595


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #795: HAWQ-860. Fix ORCA wrong plan when correla...

[hsyuan]

Github user hsyuan closed the pull request at:

https://github.com/apache/incubator-hawq/pull/795


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #808: HAWQ-944. Implement new pg_ltoa function as per p...

[paul-guo-]

Github user paul-guo- commented on the issue:

https://github.com/apache/incubator-hawq/pull/808
  
It looks like pg_ltoa() does not work for the corner case: integer 0?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (HAWQ-970) Provide More Accurate Information When LibYARN Meets an Exception

[Lin Wen (JIRA)]

Lin Wen created HAWQ-970:


 Summary: Provide More Accurate Information When LibYARN Meets an 
Exception
 Key: HAWQ-970
 URL: https://issues.apache.org/jira/browse/HAWQ-970
 Project: Apache HAWQ
  Issue Type: Bug
  Components: libyarn
Reporter: Lin Wen
Assignee: Lei Chang


Sometimes when an exception happens in libyarn, the log information is not 
accurate enough. For example, below is an exception related to kerberos ticket 
expiration, but we can't know from this log. 

2016-07-06 01:47:51.945902 
BST,,,p182375,th1403270400,con4,,seg-1,"WARNING","01000","YARN mode 
resource broker failed to get container report. 
LibYarnClient::getContainerReports, Catch the Exception:YarnIOException: 
Unexpected exception: when calling ApplicationCl
ientProtocol::getContainers in 
/data1/pulse2-agent/agents/agent1/work/LIBYARN-main-opt/rhel5_x86_64/src/libyarnserver/ApplicationClientProtocol.cpp:
 195",,,0,,"resourcebroker_LIBYARN_proc.c",1748,



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HAWQ-969) Add getting configuration from HDFS and YARN

[Chunling Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chunling Wang reassigned HAWQ-969:
--

Assignee: Chunling Wang  (was: Jiali Yao)

> Add getting configuration from HDFS and YARN
> 
>
> Key: HAWQ-969
> URL: https://issues.apache.org/jira/browse/HAWQ-969
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Tests
>Reporter: Chunling Wang
>Assignee: Chunling Wang
> Fix For: 2.0.1.0-incubating
>
>
> Add getting configuration from HDFS and YARN and writing xml file in 
> xml_parser.cpp.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq pull request #828: HAWQ-969. Add getting configuration from H...

[xunzhang]

Github user xunzhang commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/828#discussion_r72937658
  
--- Diff: src/test/feature/lib/xml_parser.h ---
@@ -17,10 +17,27 @@ class XmlConfig {
  public:
   explicit XmlConfig(std::string);
   
+  // read an XML file into a tree
+  bool open();
+
+  // only free the XML document pointer
+  void closeNotSave();
+
+  // save the updated document to disk and free the XML document pointer
+  void closeAndSave();
+
   // parse the configuration file
   void parse();
 
   // @param key The key of the configuration item
+  // @param value The updated value
+  // @param save whether save the updated document to disk, if save is 
false, open() and closeAndSave() should be called additionally
+  // @ return The value of configuration item
+  bool setString(const std::string , const std::string , const 
bool );
--- End diff --

With POD types, there is no case using const reference. For example, here 
we should define as `bool setString(const std::string , const std::string 
, bool save);`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #828: HAWQ-969. Add getting configuration from H...

[xunzhang]

Github user xunzhang commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/828#discussion_r72936411
  
--- Diff: src/test/feature/lib/hdfs_config.h ---
@@ -0,0 +1,172 @@
+#ifndef HAWQ_SRC_TEST_FEATURE_LIB_HDFS_CONFIG_H_
+#define HAWQ_SRC_TEST_FEATURE_LIB_HDFS_CONFIG_H_
+
+#include 
+#include 
+
+#include "psql.h"
+#include "sql_util.h"
+#include "xml_parser.h"
+
+namespace hawq {
+namespace test {
+
+/**
+ * HdfsConfig common libray. Get detailed information about HDFS
+ * including checking state of namenodes and datanodes, get parameter value
+ * @author Chunling Wang
+ */
+class HdfsConfig {
+  public:
+/**
+ * HdfsConfig constructor
+ */
+HdfsConfig(): psql(HAWQ_DB, HAWQ_HOST, HAWQ_PORT, HAWQ_USER, 
HAWQ_PASSWORD) {}
+
+/**
+ * HdfsConfig desstructor
+ */
+~HdfsConfig() {}
+
+/**
+ * whether HDFS is in HA mode
+ * @return true if HDFS is HA
+ */
+bool isHA();
+
+/**
+ * whether HDFS is kerbos
+ * @return true if HDFS is kerbos
+ */
+bool isKerbos();
+
+/**
+ * whether HDFS supports truncate operation
+ * @return true if HDFS supports truncate operation
+ */
+bool isTruncate();
+
+/**
+ * get HADOOP working directory
+ * @return HADOOP working directory
+ */
+std::string getHadoopHome();
+
+/**
+ * get HDFS active namenode's hostname and port information
+ * @param activenamenode, active namenode hostname reference which 
will be set
+ * @param port, active namenode port reference which will be set
+ * @return true if getActiveNamenode succeeded
+ */
+bool getActiveNamenode(std::string ,
+   int );
+
+/**
+ * get HDFS standby namenode's hostname and port information
+ * @param standbynamenode, standby namenode hostname reference which 
will be set
+ * @param port, standby namenode port reference which will be set
+ * @return true if getStandbyNamenode succeeded
+ */
+bool getStandbyNamenode(std::string ,
+int );
+
+/**
+ * get HDFS namenode(s) information
+ * @param namenodes, namenodes' hostnames reference which will be set
+ * @param port, namenodes' ports reference which will be set
+ */
+void getNamenodes(std::vector ,
+  std::vector );
+
+/**
+ * get HDFS datanodes information
+ * @param datanodelist, datanodes' hostnames reference which will be 
set
+ * @param port, datanodes' ports reference which will be set
+ */
+void getDatanodelist(std::vector ,
+ std::vector );
+
+/**
+ * get HDFS active datanodes information
+ * @param activedatanodes, active datanodes' hostnames reference which 
will be set
+ * @param port, active datanodes' ports reference which will be set
+ */
+void getActiveDatanodes(std::vector ,
+std::vector );
+
+/**
+ * whether HDFS is in safe mode
+ * @return true if HDFS is in safe node
+ */
+bool isSafemode();
+
+/**
+ * get parameter value in ./etc/hdfs-client.xml or 
./etc/hadoop/hdfs-site.xml according to parameter name
+ * @param parameterName, used to get parameter value
+ * @param conftype, get parameter value, 'hdfs' or 'HDFS' from 
./etc/hdfs-client.xml, others from ./etc/hadoop/hdfs-site.xml
+ * @return parameter value
+ */
+std::string getParameterValue(const std::string , const 
std::string );
+
+/**
+ * get parameter value in ./etc/hadoop/hdfs-site.xml according to 
parameter name
+ * @param parameterName, used to get parameter value
+ * @return parameter value
+ */
+std::string getParameterValue(const std::string );
+
+/**
+ * set parameter value in ./etc/hdfs-client.xml or 
./etc/hadoop/hdfs-site.xml according to parameter name
+ * @param parameterName, parameter name which used to set parameter 
value
+ * @param parameterValue, parameter value which to be set
+ * @param conftype, get parameter value, 'hdfs' or 'HDFS' from 
./etc/hdfs-client.xml, others from ./etc/hadoop/hdfs-site.xml
+ * @return true if succeeded
+ */
+bool setParameterValue(const std::string , const 
std::string , const std::string );
+
+/**
+ * set parameter value in ./etc/hadoop/hdfs-site.xml according to 
parameter name
+ * @param parameterName, parameter name which 

[GitHub] incubator-hawq pull request #828: HAWQ-969. Add getting configuration from H...

[xunzhang]

Github user xunzhang commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/828#discussion_r72936024
  
--- Diff: src/test/feature/lib/hdfs_config.h ---
@@ -0,0 +1,172 @@
+#ifndef HAWQ_SRC_TEST_FEATURE_LIB_HDFS_CONFIG_H_
+#define HAWQ_SRC_TEST_FEATURE_LIB_HDFS_CONFIG_H_
+
+#include 
+#include 
+
+#include "psql.h"
+#include "sql_util.h"
+#include "xml_parser.h"
+
+namespace hawq {
+namespace test {
+
+/**
+ * HdfsConfig common libray. Get detailed information about HDFS
+ * including checking state of namenodes and datanodes, get parameter value
+ * @author Chunling Wang
+ */
+class HdfsConfig {
+  public:
+/**
+ * HdfsConfig constructor
+ */
+HdfsConfig(): psql(HAWQ_DB, HAWQ_HOST, HAWQ_PORT, HAWQ_USER, 
HAWQ_PASSWORD) {}
+
+/**
+ * HdfsConfig desstructor
+ */
+~HdfsConfig() {}
+
+/**
+ * whether HDFS is in HA mode
+ * @return true if HDFS is HA
+ */
+bool isHA();
+
+/**
+ * whether HDFS is kerbos
+ * @return true if HDFS is kerbos
+ */
+bool isKerbos();
--- End diff --

I am not sure kerbos is a short name of kerberos. I have googled that, I 
think you use a incorrect library name.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #828: HAWQ-969. Add getting configuration from H...

[xunzhang]

Github user xunzhang commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/828#discussion_r72935754
  
--- Diff: src/test/feature/lib/hdfs_config.cpp ---
@@ -0,0 +1,284 @@
+#include 
+#include 
+#include 
+#include 
+
+#include "hdfs_config.h"
+#include "command.h"
+#include "psql.h"
+#include "xml_parser.h"
+#include "string_util.h"
+
+using std::string;
+
+namespace hawq {
+namespace test {
+
+string HdfsConfig::getHdfsUser() {
+  string cmd = "ps aux|grep hdfs.server|grep -v grep";
+  Command c(cmd);
+  string result = c.run().getResultOutput();
+  auto lines = hawq::test::split(result, '\n');
+  if (lines.size() >= 1) {
+return hawq::test::trim(hawq::test::split(lines[lines.size()-1], ' 
')[0]);
+  } 
+  return "hdfs";
+}
+
+bool HdfsConfig::LoadFromHawqConfigFile() {
+  const char *env = getenv("GPHOME");
+  string confPath = env ? env : "";
+  if (!confPath.empty()) {
+confPath.append("/etc/hdfs-client.xml");
+  } else {
+return false;
+  }
+
+  hawqxmlconf.reset(new XmlConfig(confPath));
+  hawqxmlconf->parse();
+  return true;
+}
+
+bool HdfsConfig::LoadFromHdfsConfigFile() {
+  string confPath=getHadoopHome();
+  if (confPath == "")
+return false;
+  confPath.append("/etc/hadoop/hdfs-site.xml");
+  hdfsxmlconf.reset(new XmlConfig(confPath));
+  hdfsxmlconf->parse();
+  return true;
+}
+
+bool HdfsConfig::isHA() {
+  bool ret = LoadFromHawqConfigFile();
+  if (!ret) {
+return false;
+  }
+  string nameservice = hawqxmlconf->getString("dfs.nameservices");
+  if (nameservice.length() > 0) {
+return true;
+  } else {
+return false;
+  }
+}
+
+bool HdfsConfig::isKerbos() {
+  bool ret = LoadFromHawqConfigFile();
+  if (!ret) {
+return false;
--- End diff --

What is the semantics of the interface returning `false`? If it means that 
"without Kerbos", the implementation is incorrect. You should throw exception 
or load default configuration instead.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #828: HAWQ-969. Add getting configuration from H...

[xunzhang]

Github user xunzhang commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/828#discussion_r72935273
  
--- Diff: src/test/feature/lib/hdfs_config.cpp ---
@@ -0,0 +1,284 @@
+#include 
+#include 
+#include 
+#include 
+
+#include "hdfs_config.h"
+#include "command.h"
+#include "psql.h"
+#include "xml_parser.h"
+#include "string_util.h"
+
+using std::string;
+
+namespace hawq {
+namespace test {
+
+string HdfsConfig::getHdfsUser() {
+  string cmd = "ps aux|grep hdfs.server|grep -v grep";
+  Command c(cmd);
+  string result = c.run().getResultOutput();
+  auto lines = hawq::test::split(result, '\n');
+  if (lines.size() >= 1) {
+return hawq::test::trim(hawq::test::split(lines[lines.size()-1], ' 
')[0]);
+  } 
+  return "hdfs";
+}
+
+bool HdfsConfig::LoadFromHawqConfigFile() {
+  const char *env = getenv("GPHOME");
+  string confPath = env ? env : "";
+  if (!confPath.empty()) {
+confPath.append("/etc/hdfs-client.xml");
+  } else {
+return false;
+  }
+
+  hawqxmlconf.reset(new XmlConfig(confPath));
+  hawqxmlconf->parse();
+  return true;
+}
+
+bool HdfsConfig::LoadFromHdfsConfigFile() {
+  string confPath=getHadoopHome();
+  if (confPath == "")
+return false;
+  confPath.append("/etc/hadoop/hdfs-site.xml");
+  hdfsxmlconf.reset(new XmlConfig(confPath));
+  hdfsxmlconf->parse();
+  return true;
+}
+
+bool HdfsConfig::isHA() {
+  bool ret = LoadFromHawqConfigFile();
+  if (!ret) {
+return false;
+  }
+  string nameservice = hawqxmlconf->getString("dfs.nameservices");
+  if (nameservice.length() > 0) {
+return true;
+  } else {
+return false;
+  }
+}
+
+bool HdfsConfig::isKerbos() {
+  bool ret = LoadFromHawqConfigFile();
+  if (!ret) {
+return false;
+  }
+  string authentication = 
hawqxmlconf->getString("hadoop.security.authentication");
+  if (authentication == "kerberos") {
+return true;
+  } else {
+return false;
+  }
+}
+
+bool HdfsConfig::isTruncate() {
+  string cmd = "hadoop fs -truncate";
+  Command c(cmd);
+  string result = c.run().getResultOutput();
+  auto lines = hawq::test::split(result, '\n');
+  if (lines.size() >= 1) {
+string valueLine = lines[0];
+int find = valueLine.find("-truncate: Unknown command");
+if (find < 0)
--- End diff --

Unify your coding style with parentheses


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #828: HAWQ-969. Add getting configuration from H...

[xunzhang]

Github user xunzhang commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/828#discussion_r72934755
  
--- Diff: src/test/feature/testlib/test_lib.cpp ---
@@ -56,6 +58,69 @@ TEST_F(TestCommonLib, TestHawqConfig) {
   return;
 }
 
+TEST_F(TestCommonLib, TestHdfsConfig) {
+  hawq::test::HdfsConfig hc;
+  hc.isHA();
--- End diff --

As I mentioned in **[REVIEW-LINK-1]**, if you implemented with loading from 
a configuration file, you should not just invoke the interface. You should load 
from a test config file and use `EXPECT_EQ`-like macro function to write your 
tests.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #828: HAWQ-969. Add getting configuration from H...

[xunzhang]

Github user xunzhang commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/828#discussion_r72934303
  
--- Diff: src/test/feature/lib/hdfs_config.cpp ---
@@ -0,0 +1,284 @@
+#include 
+#include 
+#include 
+#include 
+
+#include "hdfs_config.h"
+#include "command.h"
+#include "psql.h"
+#include "xml_parser.h"
+#include "string_util.h"
+
+using std::string;
+
+namespace hawq {
+namespace test {
+
+string HdfsConfig::getHdfsUser() {
+  string cmd = "ps aux|grep hdfs.server|grep -v grep";
+  Command c(cmd);
+  string result = c.run().getResultOutput();
+  auto lines = hawq::test::split(result, '\n');
+  if (lines.size() >= 1) {
+return hawq::test::trim(hawq::test::split(lines[lines.size()-1], ' 
')[0]);
+  } 
+  return "hdfs";
+}
+
+bool HdfsConfig::LoadFromHawqConfigFile() {
+  const char *env = getenv("GPHOME");
+  string confPath = env ? env : "";
+  if (!confPath.empty()) {
+confPath.append("/etc/hdfs-client.xml");
+  } else {
+return false;
+  }
+
+  hawqxmlconf.reset(new XmlConfig(confPath));
+  hawqxmlconf->parse();
+  return true;
+}
+
+bool HdfsConfig::LoadFromHdfsConfigFile() {
+  string confPath=getHadoopHome();
+  if (confPath == "")
+return false;
+  confPath.append("/etc/hadoop/hdfs-site.xml");
+  hdfsxmlconf.reset(new XmlConfig(confPath));
+  hdfsxmlconf->parse();
+  return true;
+}
+
+bool HdfsConfig::isHA() {
--- End diff --

[REVIEW-LINK-1] I think the implementation of these functions are 
problematic. Say if user modified these configuration files without restart 
HAWQ, the result could be confused. A better way is to get these kinds of 
status from running HAWQ, is that possible?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #828: HAWQ-969. Add getting configuration from H...

[xunzhang]

Github user xunzhang commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/828#discussion_r72933008
  
--- Diff: src/test/feature/lib/hdfs_config.h ---
@@ -0,0 +1,172 @@
+#ifndef HAWQ_SRC_TEST_FEATURE_LIB_HDFS_CONFIG_H_
+#define HAWQ_SRC_TEST_FEATURE_LIB_HDFS_CONFIG_H_
+
+#include 
+#include 
+
+#include "psql.h"
+#include "sql_util.h"
+#include "xml_parser.h"
+
+namespace hawq {
+namespace test {
+
+/**
+ * HdfsConfig common libray. Get detailed information about HDFS
+ * including checking state of namenodes and datanodes, get parameter value
+ * @author Chunling Wang
+ */
+class HdfsConfig {
+  public:
+/**
+ * HdfsConfig constructor
+ */
+HdfsConfig(): psql(HAWQ_DB, HAWQ_HOST, HAWQ_PORT, HAWQ_USER, 
HAWQ_PASSWORD) {}
+
+/**
+ * HdfsConfig desstructor
--- End diff --

...typo


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #828: HAWQ-969. Add getting configuration from H...

[xunzhang]

Github user xunzhang commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/828#discussion_r72932215
  
--- Diff: src/test/feature/lib/hdfs_config.h ---
@@ -0,0 +1,172 @@
+#ifndef HAWQ_SRC_TEST_FEATURE_LIB_HDFS_CONFIG_H_
+#define HAWQ_SRC_TEST_FEATURE_LIB_HDFS_CONFIG_H_
+
+#include 
+#include 
+
+#include "psql.h"
+#include "sql_util.h"
+#include "xml_parser.h"
+
+namespace hawq {
+namespace test {
+
+/**
+ * HdfsConfig common libray. Get detailed information about HDFS
+ * including checking state of namenodes and datanodes, get parameter value
+ * @author Chunling Wang
+ */
+class HdfsConfig {
+  public:
+/**
+ * HdfsConfig constructor
+ */
+HdfsConfig(): psql(HAWQ_DB, HAWQ_HOST, HAWQ_PORT, HAWQ_USER, 
HAWQ_PASSWORD) {}
+
+/**
+ * HdfsConfig desstructor
+ */
+~HdfsConfig() {}
+
+/**
+ * whether HDFS is in HA mode
+ * @return true if HDFS is HA
+ */
+bool isHA();
+
+/**
+ * whether HDFS is kerbos
+ * @return true if HDFS is kerbos
+ */
+bool isKerbos();
+
+/**
+ * whether HDFS supports truncate operation
+ * @return true if HDFS supports truncate operation
+ */
+bool isTruncate();
+
+/**
+ * get HADOOP working directory
+ * @return HADOOP working directory
+ */
+std::string getHadoopHome();
+
+/**
+ * get HDFS active namenode's hostname and port information
+ * @param activenamenode, active namenode hostname reference which 
will be set
+ * @param port, active namenode port reference which will be set
+ * @return true if getActiveNamenode succeeded
+ */
+bool getActiveNamenode(std::string ,
+   int );
+
+/**
+ * get HDFS standby namenode's hostname and port information
+ * @param standbynamenode, standby namenode hostname reference which 
will be set
+ * @param port, standby namenode port reference which will be set
+ * @return true if getStandbyNamenode succeeded
+ */
+bool getStandbyNamenode(std::string ,
+int );
+
+/**
+ * get HDFS namenode(s) information
+ * @param namenodes, namenodes' hostnames reference which will be set
+ * @param port, namenodes' ports reference which will be set
+ */
+void getNamenodes(std::vector ,
+  std::vector );
+
+/**
+ * get HDFS datanodes information
+ * @param datanodelist, datanodes' hostnames reference which will be 
set
+ * @param port, datanodes' ports reference which will be set
+ */
+void getDatanodelist(std::vector ,
+ std::vector );
+
+/**
+ * get HDFS active datanodes information
+ * @param activedatanodes, active datanodes' hostnames reference which 
will be set
+ * @param port, active datanodes' ports reference which will be set
+ */
+void getActiveDatanodes(std::vector ,
+std::vector );
+
+/**
+ * whether HDFS is in safe mode
+ * @return true if HDFS is in safe node
+ */
+bool isSafemode();
+
+/**
+ * get parameter value in ./etc/hdfs-client.xml or 
./etc/hadoop/hdfs-site.xml according to parameter name
+ * @param parameterName, used to get parameter value
+ * @param conftype, get parameter value, 'hdfs' or 'HDFS' from 
./etc/hdfs-client.xml, others from ./etc/hadoop/hdfs-site.xml
+ * @return parameter value
+ */
+std::string getParameterValue(const std::string , const 
std::string );
+
+/**
+ * get parameter value in ./etc/hadoop/hdfs-site.xml according to 
parameter name
+ * @param parameterName, used to get parameter value
+ * @return parameter value
+ */
+std::string getParameterValue(const std::string );
+
+/**
+ * set parameter value in ./etc/hdfs-client.xml or 
./etc/hadoop/hdfs-site.xml according to parameter name
+ * @param parameterName, parameter name which used to set parameter 
value
+ * @param parameterValue, parameter value which to be set
+ * @param conftype, get parameter value, 'hdfs' or 'HDFS' from 
./etc/hdfs-client.xml, others from ./etc/hadoop/hdfs-site.xml
+ * @return true if succeeded
+ */
+bool setParameterValue(const std::string , const 
std::string , const std::string );
+
+/**
+ * set parameter value in ./etc/hadoop/hdfs-site.xml according to 
parameter name
+ * @param parameterName, parameter name which 

[GitHub] incubator-hawq pull request #828: HAWQ-969. Add getting configuration from H...

[wcl14]

GitHub user wcl14 opened a pull request:

https://github.com/apache/incubator-hawq/pull/828

HAWQ-969. Add getting configuration from HDFS and YARN

Add getting configuration from HDFS and YARN, and writing xml file in 
xml_parser.cpp.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wcl14/incubator-hawq HAWQ-969

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/828.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #828


commit f892618a5c3e69f59d8b683352141ed67b478d89
Author: Chunling Wang 
Date:   2016-08-01T06:59:17Z

HAWQ-969. Add getting configuration from HDFS and YARN




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (HAWQ-969) Add getting configuration from HDFS and YARN

[Chunling Wang (JIRA)]

Chunling Wang created HAWQ-969:
--

 Summary: Add getting configuration from HDFS and YARN
 Key: HAWQ-969
 URL: https://issues.apache.org/jira/browse/HAWQ-969
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: Tests
Reporter: Chunling Wang
Assignee: Jiali Yao


Add getting configuration from HDFS and YARN and writing xml file in 
xml_parser.cpp.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)