[jira] [Commented] (HAWQ-567) Add HAWQ commands option conflict checks

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207923#comment-15207923
 ] 

ASF GitHub Bot commented on HAWQ-567:
-

Github user radarwave closed the pull request at:

https://github.com/apache/incubator-hawq/pull/492


> Add HAWQ commands option conflict checks
> 
>
> Key: HAWQ-567
> URL: https://issues.apache.org/jira/browse/HAWQ-567
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Command Line Tools
>Reporter: Radar Lei
>Assignee: Radar Lei
>
> We should checks option conflicts when running hawq command line tools.
> For example:
> --quiet can't be run with --verbose



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-445) Support large strings (up to a GB) in text_to_array()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207878#comment-15207878
 ] 

ASF GitHub Bot commented on HAWQ-445:
-

Github user wengyanqing commented on the pull request:

https://github.com/apache/incubator-hawq/pull/378#issuecomment-200185372
  
I had merged this PR, @armenatzoglou would you please close it ?


> Support large strings (up to a GB) in text_to_array()
> -
>
> Key: HAWQ-445
> URL: https://issues.apache.org/jira/browse/HAWQ-445
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Query Execution
>Reporter: Nikos Armenatzoglou
>Assignee: George Caragea
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-578) Avoid validating whether total allocated resource is more than maximum YARN resource queue capacity

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207804#comment-15207804
 ] 

ASF GitHub Bot commented on HAWQ-578:
-

Github user linwen commented on the pull request:

https://github.com/apache/incubator-hawq/pull/493#issuecomment-200145132
  
+1 


> Avoid validating whether total allocated resource is more than maximum YARN 
> resource queue capacity
> ---
>
> Key: HAWQ-578
> URL: https://issues.apache.org/jira/browse/HAWQ-578
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Resource Manager
>Reporter: Yi Jin
>Assignee: Yi Jin
>
> YARN may provides more than maximum possible count of containers. And 
> Resource manager should not accept more containers more than the target 
> segment's resource limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-573) Resource leak when cancel copy

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207801#comment-15207801
 ] 

ASF GitHub Bot commented on HAWQ-573:
-

Github user jiny2 commented on the pull request:

https://github.com/apache/incubator-hawq/pull/491#issuecomment-200144812
  
+1 LGTM


> Resource leak when cancel copy
> --
>
> Key: HAWQ-573
> URL: https://issues.apache.org/jira/browse/HAWQ-573
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Resource Manager
>Reporter: Dong Li
>Assignee: Lei Chang
>
> Using select pg_cancel_backend() to cancel a "copy test5_pt2 (c1, c2, c3, c4, 
> c5, c6) FROM stdin;"
> query make the resource leak.
> You can see the log, when cancel the query, it didn't return resource to 
> resource manager.
> p25926 is the QD process. The qd process didn't think it has acquired 
> resource, but the RM has allocated resource.
> When qd is running in the function processAllCommFileDescs (called by 
> acquireResourceFromRM,  which is a RPC to allocate resource from QD to RM), 
> it received a "query cancel pending'" interrupt, and it handles the interrupt 
> and error out.
> {code}
> 2016-03-16 11:55:13.576705 
> CST,"intern","resourceleak",p25926,th-1593601580,"[local]",,2016-03-16 
> 11:55:12 CST,6494,con2476,cmd3,seg-1,,,x6494,sx1,"LOG","0","AsyncComm 
> framework receives message 2310 from FD 12",  ,"copy test5_pt2 (c1, 
> c2, c3, c4, c5, c6) FROM stdin;",0,,"rmcomm_Message.c",100,
> 13457 2016-03-16 11:55:13.576779 
> CST,"intern","resourceleak",p25926,th-1593601580,"[local]",,2016-03-16 
> 11:55:12 CST,6494,con2476,cmd3,seg-1,,,x6494,sx1,"LOG","0","ConnID 
> 427. Registered in HAWQ resource manager (By OID)",  ,"copy test5_pt2 
> (c1, c2, c3, c4, c5, c6) FROM stdin;",0,,"rmcomm_QD2RM.c",601,
> 13458 2016-03-16 11:55:13.576860 
> CST,"intern","resourceleak",p25926,th-1593601580,"[local]",,2016-03-16 
> 11:55:12 CST,6494,con2476,cmd3,seg-1,,,x6494,sx1,"LOG","0","ConnID: 
> 427. Acquire resource request for index 0. Max vse  g size 6 Min vseg 
> size 6 Estimated slice size 1 estimated IO bytes size 1 Preferred node count 
> 0.",,"copy test5_pt2 (c1, c2, c3, c4, c5, c6) FROM 
> stdin;",0,,"rmcomm_QD2RM.c",693,
> 13459 2016-03-16 11:55:13.577266 
> CST,,,p92142,th-15936015800,con18,,seg-1,"LOG","0","AsyncComm 
> framework receives message 259 from FD 5",,,0,,"rmcomm_Message.c",100,
> 13460 2016-03-16 11:55:13.577311 
> CST,,,p92142,th-15936015800,con18,,seg-1,"LOG","0","ConnID 
> 427. Expect query resource for session 
> 2476",,,0,,"resqueuemanager.c",2225,
> 13461 2016-03-16 11:55:13.577347 
> CST,,,p92142,th-15936015800,con18,,seg-1,"LOG","0","ConnID 
> 427. Expect query resource (256 MB, 0.062500 CORE) x 256 (MIN 6) after 
> checking queue capacity.",,,0,,"resqueuemanager  .c",3760,
> 13462 2016-03-16 11:55:13.577391 
> CST,,,p92142,th-15936015800,con18,,seg-1,"LOG","0","ConnID 
> 427. Expect query resource (256 MB, 0.062500 CORE) x 6 (MIN 6) after checking 
> query expectation 6 (MIN 6).",,,0,,"resq  ueuemanager.c",3790,
> 13463 2016-03-16 11:55:13.577440 
> CST,,,p92142,th-15936015800,con18,,seg-1,"LOG","0","ConnID 
> 427. Expect query resource (256 MB, 0.062500 CORE) x 6 ( MIN 6 ) resource 
> after adjusting based on queue NVSEG limits.",,,  
> 0,,"resqueuemanager.c",2247,
> 13464 2016-03-16 11:55:13.577490 
> CST,,,p92142,th-15936015800,con18,,seg-1,"LOG","0","Latency 
> of getting resource allocated is 193us",,,0,,"resqueuemanager.c",4745,
> 13465 2016-03-16 11:55:13.588859 
> CST,"intern","resourceleak",p25958,th-1593601580,"[local]",,2016-03-16 
> 11:55:13 
> CST,0,con2492,,seg-1,"LOG","0","getLocalTmpDirFromMasterConfig 
> session_id:2492 tmpdir:/tmp",,,0,,"pos  tinit.c",461,
> 13466 2016-03-16 11:55:13.628191 
> CST,"intern","resourceleak",p25960,th-1593601580,"[local]",,2016-03-16 
> 11:55:13 
> CST,0,con2493,,seg-1,"LOG","0","getLocalTmpDirFromMasterConfig 
> session_id:2493 tmpdir:/tmp",,,0,,"pos  tinit.c",461,
> 13467 2016-03-16 11:55:13.671378 
> CST,"intern","resourceleak",p25963,th-1593601580,"[local]",,2016-03-16 
> 11:55:13 
> CST,0,con2494,,seg-1,"LOG","0","getLocalTmpDirFromMasterConfig 
> session_id:2494 tmpdir:/tmp",,,0,,"pos  tinit.c",461,
> 13468 2016-03-16 11:55:13.675873 
> CST,"intern","resourceleak",p25926,th-1593601580,"[local]",,2016-03-16 
> 11:55:12 CST,6494,con2476,cmd3,seg-1,,,x6494,sx1,"LOG","0","Process 
> interrupt for 'query cancel pending'.",,"copy   test5_pt2 (c1, c2, 
> c3, c4, c5, c6) FROM stdin;",0,,"postgres.c",3517,
> 13469 2016-03-16 11:55:13.675949 
> 

[jira] [Commented] (HAWQ-567) Add HAWQ commands option conflict checks

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207796#comment-15207796
 ] 

ASF GitHub Bot commented on HAWQ-567:
-

Github user yaoj2 commented on the pull request:

https://github.com/apache/incubator-hawq/pull/492#issuecomment-200143766
  
+1


> Add HAWQ commands option conflict checks
> 
>
> Key: HAWQ-567
> URL: https://issues.apache.org/jira/browse/HAWQ-567
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Command Line Tools
>Reporter: Radar Lei
>Assignee: Radar Lei
>
> We should checks option conflicts when running hawq command line tools.
> For example:
> --quiet can't be run with --verbose



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-578) Avoid validating whether total allocated resource is more than maximum YARN resource queue capacity

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207792#comment-15207792
 ] 

ASF GitHub Bot commented on HAWQ-578:
-

GitHub user jiny2 opened a pull request:

https://github.com/apache/incubator-hawq/pull/493

HAWQ-578. Avoid validating whether total allocated resource is more than 
maximum YARN resource queue capacity



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jiny2/incubator-hawq HAWQ-578

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/493.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #493


commit 24aef5aaca81ecc519e5a0f6b3457822b33665bc
Author: YI JIN 
Date:   2016-03-23T02:48:57Z

HAWQ-578. Avoid validating whether total allocated resource is more than 
maximum YARN resource queue capacity




> Avoid validating whether total allocated resource is more than maximum YARN 
> resource queue capacity
> ---
>
> Key: HAWQ-578
> URL: https://issues.apache.org/jira/browse/HAWQ-578
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Resource Manager
>Reporter: Yi Jin
>Assignee: Yi Jin
>
> YARN may provides more than maximum possible count of containers. And 
> Resource manager should not accept more containers more than the target 
> segment's resource limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-567) Add HAWQ commands option conflict checks

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207794#comment-15207794
 ] 

ASF GitHub Bot commented on HAWQ-567:
-

Github user huor commented on the pull request:

https://github.com/apache/incubator-hawq/pull/492#issuecomment-200143626
  
+1


> Add HAWQ commands option conflict checks
> 
>
> Key: HAWQ-567
> URL: https://issues.apache.org/jira/browse/HAWQ-567
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Command Line Tools
>Reporter: Radar Lei
>Assignee: Radar Lei
>
> We should checks option conflicts when running hawq command line tools.
> For example:
> --quiet can't be run with --verbose



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-567) Add HAWQ commands option conflict checks

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207789#comment-15207789
 ] 

ASF GitHub Bot commented on HAWQ-567:
-

GitHub user radarwave opened a pull request:

https://github.com/apache/incubator-hawq/pull/492

HAWQ-567. Fix -U --specia-mode option conflict check



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/radarwave/incubator-hawq HAWQ-567

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/492.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #492


commit bcd47c4b66d08563958c5b31a7f1483b6211c91e
Author: rlei 
Date:   2016-03-23T02:49:53Z

HAWQ-567. Fix -U --specia-mode option conflict check




> Add HAWQ commands option conflict checks
> 
>
> Key: HAWQ-567
> URL: https://issues.apache.org/jira/browse/HAWQ-567
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Command Line Tools
>Reporter: Radar Lei
>Assignee: Radar Lei
>
> We should checks option conflicts when running hawq command line tools.
> For example:
> --quiet can't be run with --verbose



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HAWQ-575) QE core dumped when report "Resume interrupt holdoff count is bad (0) (xact.c:2907)"

[Ming LI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ming LI resolved HAWQ-575.
--
Resolution: Fixed

> QE core dumped when report "Resume interrupt holdoff count is bad (0) 
> (xact.c:2907)"
> 
>
> Key: HAWQ-575
> URL: https://issues.apache.org/jira/browse/HAWQ-575
> Project: Apache HAWQ
>  Issue Type: Bug
>Reporter: Ming LI
>Assignee: Ming LI
>
> Core was generated by `postgres: port  5532, gpadmin tpch_row_2... 
> 172.28.8.250(18226) con92825 seg97'.
> Program terminated with signal 6, Aborted.
> #0  0x0033d4032925 in raise () from /lib64/libc.so.6
> Missing separate debuginfos, use: debuginfo-install 
> hawq-2.0.0.0_beta-20925.x86_64
> (gdb) bt
> #0  0x0033d4032925 in raise () from /lib64/libc.so.6
> #1  0x0033d4034105 in abort () from /lib64/libc.so.6
> #2  0x00871c6e in errfinish (dummy=) at 
> elog.c:682
> #3  0x008727bb in elog_finish (elevel=, 
> fmt=) at elog.c:1459
> #4  0x004e171b in AbortTransaction () at xact.c:2907
> #5  0x004e2c45 in AbortCurrentTransaction () at xact.c:3377
> #6  0x007b01ea in PostgresMain (argc=37474312, argv=0x0, 
> username=) at postgres.c:4507
> #7  0x00763c03 in BackendRun (port=0x2373210) at postmaster.c:5889
> #8  BackendStartup (port=0x2373210) at postmaster.c:5484
> #9  0x0076435d in ServerLoop () at postmaster.c:2163
> #10 0x0076618e in PostmasterMain (argc=9, argv=0x236a5b0) at 
> postmaster.c:1454
> #11 0x006c028a in main (argc=9, argv=0x236a570) at main.c:226
> (gdb) f 3
> #3  0x008727bb in elog_finish (elevel=, 
> fmt=) at elog.c:1459
> (gdb) p *edata
> $1 = {elevel = 22, output_to_server = 1 '\001', output_to_client = 1 '\001', 
> show_funcname = 0 '\000', omit_location = 0 '\000', fatal_return = 0 '\000',
>   hide_stmt = 0 '\000', send_alert = 1 '\001', filename = 0x9cc38e "xact.c", 
> lineno = 2907, funcname = 0x9c66c0 "AbortTransaction",
>   domain = 0xafb668 "postgres-8.2", sqlerrcode = 2600, message = 0x236da50 
> "Resume interrupt holdoff count is bad (0) (xact.c:2907)", detail = 0x0,
>   detail_log = 0x0, hint = 0x0, context = 0x0, cursorpos = 0, internalpos = 
> 0, internalquery = 0x0, saved_errno = 5, stacktracearray = {0x871f7f, 
> 0x872659,
> 0x4e171b, 0x4e2c45, 0x7b01ea, 0x763c03, 0x76435d, 0x76618e, 0x6c028a, 
> 0x33d401ed1d, 0x4a17e9, 0x0 }, stacktracesize = 11,
>   printstack = 0 '\000'}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-465) Implement stored procedure to return fields metainfo from PXF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207769#comment-15207769
 ] 

ASF GitHub Bot commented on HAWQ-465:
-

Github user hornn commented on the pull request:

https://github.com/apache/incubator-hawq/pull/479#issuecomment-200136727
  
@sansanichfb Thank you for the link to the design doc - it made things much 
more clear.
I think the examples in the JIRA need to be updated, though - the pattern 
and path are combined.
Great job, it looks really good!

@shivzone Good point regarding the API changes. We didn't consider take 
releases into account until now when changing the protocol version.


> Implement stored procedure to return fields metainfo from PXF
> -
>
> Key: HAWQ-465
> URL: https://issues.apache.org/jira/browse/HAWQ-465
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Hcatalog, PXF
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
> Fix For: 2.0.0
>
>
> User should be able to call built-in function:
> {code}
> select pxf_get_object_fields('source_name', 'container_name', 'object_name');
>  pxf_get_object_fields 
> {code}
> to retrieve all metadata for given source, container, object.
> Input parameters:
> ||Name||Type||Mode||
> |profile|text|IN|
> |pattern|text|IN|
> |path|text|OUT|
> |objectname|text|OUT|
> |fieldname|text|OUT|
> |fieldtype|text|OUT|
> |fieldmodifiers|text|OUT|
> Example:
> {noformat}
> # select * from pxf_get_object_fields('Hive', 'default/tab*');
>   path  | objectname  | fieldname   |fieldtype   |  fieldmodifiers |
> -+-+-- 
> ++---+
> default |  table1 |col1 |int4   | not 
> null   |
> default |  table1 |col2 |text   | 
>|
> default |  table2 |col1 |int4   | not 
> null   |
> default |  table2 |col2 |text   | 
>|
> (4 rows)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-576) Remove useless code for identity.c

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207759#comment-15207759
 ] 

ASF GitHub Bot commented on HAWQ-576:
-

Github user yaoj2 commented on the pull request:

https://github.com/apache/incubator-hawq/pull/487#issuecomment-200134960
  
＋1


> Remove useless code for identity.c
> --
>
> Key: HAWQ-576
> URL: https://issues.apache.org/jira/browse/HAWQ-576
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Catalog
>Reporter: Dong Li
>Assignee: Lei Chang
>
> Remove useless code for identity.c to improve code coverage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-465) Implement stored procedure to return fields metainfo from PXF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207756#comment-15207756
 ] 

ASF GitHub Bot commented on HAWQ-465:
-

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/479#discussion_r57101396
  
--- Diff: src/include/catalog/namespace.h ---
@@ -95,4 +95,6 @@ extern char *namespace_search_path;
 
 extern List *fetch_search_path(bool includeImplicit);
 
+#define HiveProfileName "Hive"
--- End diff --

then I think it should be part of namespace.c - the header file is used by 
other files, so if it's not useful for them it shouldn't be there.


> Implement stored procedure to return fields metainfo from PXF
> -
>
> Key: HAWQ-465
> URL: https://issues.apache.org/jira/browse/HAWQ-465
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Hcatalog, PXF
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
> Fix For: 2.0.0
>
>
> User should be able to call built-in function:
> {code}
> select pxf_get_object_fields('source_name', 'container_name', 'object_name');
>  pxf_get_object_fields 
> {code}
> to retrieve all metadata for given source, container, object.
> Input parameters:
> ||Name||Type||Mode||
> |profile|text|IN|
> |pattern|text|IN|
> |path|text|OUT|
> |objectname|text|OUT|
> |fieldname|text|OUT|
> |fieldtype|text|OUT|
> |fieldmodifiers|text|OUT|
> Example:
> {noformat}
> # select * from pxf_get_object_fields('Hive', 'default/tab*');
>   path  | objectname  | fieldname   |fieldtype   |  fieldmodifiers |
> -+-+-- 
> ++---+
> default |  table1 |col1 |int4   | not 
> null   |
> default |  table1 |col2 |text   | 
>|
> default |  table2 |col1 |int4   | not 
> null   |
> default |  table2 |col2 |text   | 
>|
> (4 rows)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-573) Resource leak when cancel copy

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207750#comment-15207750
 ] 

ASF GitHub Bot commented on HAWQ-573:
-

Github user realdawn commented on the pull request:

https://github.com/apache/incubator-hawq/pull/486#issuecomment-200134508
  
I will change the code and pull a new request.


> Resource leak when cancel copy
> --
>
> Key: HAWQ-573
> URL: https://issues.apache.org/jira/browse/HAWQ-573
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Resource Manager
>Reporter: Dong Li
>Assignee: Lei Chang
>
> Using select pg_cancel_backend() to cancel a "copy test5_pt2 (c1, c2, c3, c4, 
> c5, c6) FROM stdin;"
> query make the resource leak.
> You can see the log, when cancel the query, it didn't return resource to 
> resource manager.
> p25926 is the QD process. The qd process didn't think it has acquired 
> resource, but the RM has allocated resource.
> When qd is running in the function processAllCommFileDescs (called by 
> acquireResourceFromRM,  which is a RPC to allocate resource from QD to RM), 
> it received a "query cancel pending'" interrupt, and it handles the interrupt 
> and error out.
> {code}
> 2016-03-16 11:55:13.576705 
> CST,"intern","resourceleak",p25926,th-1593601580,"[local]",,2016-03-16 
> 11:55:12 CST,6494,con2476,cmd3,seg-1,,,x6494,sx1,"LOG","0","AsyncComm 
> framework receives message 2310 from FD 12",  ,"copy test5_pt2 (c1, 
> c2, c3, c4, c5, c6) FROM stdin;",0,,"rmcomm_Message.c",100,
> 13457 2016-03-16 11:55:13.576779 
> CST,"intern","resourceleak",p25926,th-1593601580,"[local]",,2016-03-16 
> 11:55:12 CST,6494,con2476,cmd3,seg-1,,,x6494,sx1,"LOG","0","ConnID 
> 427. Registered in HAWQ resource manager (By OID)",  ,"copy test5_pt2 
> (c1, c2, c3, c4, c5, c6) FROM stdin;",0,,"rmcomm_QD2RM.c",601,
> 13458 2016-03-16 11:55:13.576860 
> CST,"intern","resourceleak",p25926,th-1593601580,"[local]",,2016-03-16 
> 11:55:12 CST,6494,con2476,cmd3,seg-1,,,x6494,sx1,"LOG","0","ConnID: 
> 427. Acquire resource request for index 0. Max vse  g size 6 Min vseg 
> size 6 Estimated slice size 1 estimated IO bytes size 1 Preferred node count 
> 0.",,"copy test5_pt2 (c1, c2, c3, c4, c5, c6) FROM 
> stdin;",0,,"rmcomm_QD2RM.c",693,
> 13459 2016-03-16 11:55:13.577266 
> CST,,,p92142,th-15936015800,con18,,seg-1,"LOG","0","AsyncComm 
> framework receives message 259 from FD 5",,,0,,"rmcomm_Message.c",100,
> 13460 2016-03-16 11:55:13.577311 
> CST,,,p92142,th-15936015800,con18,,seg-1,"LOG","0","ConnID 
> 427. Expect query resource for session 
> 2476",,,0,,"resqueuemanager.c",2225,
> 13461 2016-03-16 11:55:13.577347 
> CST,,,p92142,th-15936015800,con18,,seg-1,"LOG","0","ConnID 
> 427. Expect query resource (256 MB, 0.062500 CORE) x 256 (MIN 6) after 
> checking queue capacity.",,,0,,"resqueuemanager  .c",3760,
> 13462 2016-03-16 11:55:13.577391 
> CST,,,p92142,th-15936015800,con18,,seg-1,"LOG","0","ConnID 
> 427. Expect query resource (256 MB, 0.062500 CORE) x 6 (MIN 6) after checking 
> query expectation 6 (MIN 6).",,,0,,"resq  ueuemanager.c",3790,
> 13463 2016-03-16 11:55:13.577440 
> CST,,,p92142,th-15936015800,con18,,seg-1,"LOG","0","ConnID 
> 427. Expect query resource (256 MB, 0.062500 CORE) x 6 ( MIN 6 ) resource 
> after adjusting based on queue NVSEG limits.",,,  
> 0,,"resqueuemanager.c",2247,
> 13464 2016-03-16 11:55:13.577490 
> CST,,,p92142,th-15936015800,con18,,seg-1,"LOG","0","Latency 
> of getting resource allocated is 193us",,,0,,"resqueuemanager.c",4745,
> 13465 2016-03-16 11:55:13.588859 
> CST,"intern","resourceleak",p25958,th-1593601580,"[local]",,2016-03-16 
> 11:55:13 
> CST,0,con2492,,seg-1,"LOG","0","getLocalTmpDirFromMasterConfig 
> session_id:2492 tmpdir:/tmp",,,0,,"pos  tinit.c",461,
> 13466 2016-03-16 11:55:13.628191 
> CST,"intern","resourceleak",p25960,th-1593601580,"[local]",,2016-03-16 
> 11:55:13 
> CST,0,con2493,,seg-1,"LOG","0","getLocalTmpDirFromMasterConfig 
> session_id:2493 tmpdir:/tmp",,,0,,"pos  tinit.c",461,
> 13467 2016-03-16 11:55:13.671378 
> CST,"intern","resourceleak",p25963,th-1593601580,"[local]",,2016-03-16 
> 11:55:13 
> CST,0,con2494,,seg-1,"LOG","0","getLocalTmpDirFromMasterConfig 
> session_id:2494 tmpdir:/tmp",,,0,,"pos  tinit.c",461,
> 13468 2016-03-16 11:55:13.675873 
> CST,"intern","resourceleak",p25926,th-1593601580,"[local]",,2016-03-16 
> 11:55:12 CST,6494,con2476,cmd3,seg-1,,,x6494,sx1,"LOG","0","Process 
> interrupt for 'query cancel pending'.",,"copy   test5_pt2 (c1, c2, 
> c3, c4, c5, c6) FROM stdin;",0,,"postgres.c",3517,
> 13469 2016-03-16 

[jira] [Commented] (HAWQ-507) Remove dead code for cdbpersistentrelfile.c

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207752#comment-15207752
 ] 

ASF GitHub Bot commented on HAWQ-507:
-

Github user liming01 commented on the pull request:

https://github.com/apache/incubator-hawq/pull/427#issuecomment-200134601
  
+1.


> Remove dead code for cdbpersistentrelfile.c
> ---
>
> Key: HAWQ-507
> URL: https://issues.apache.org/jira/browse/HAWQ-507
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Transaction
>Reporter: Dong Li
>Assignee: Lei Chang
>
> As the code coverage of cdbpersistentrelfile.c is only 59% due to dead code, 
> so we should remove the dead code in that file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-570) YARN queue report should be a relative percentage value instead of an absolute value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207738#comment-15207738
 ] 

ASF GitHub Bot commented on HAWQ-570:
-

Github user yaoj2 commented on the pull request:

https://github.com/apache/incubator-hawq/pull/488#issuecomment-200133020
  
+1


> YARN queue report should be a relative percentage value instead of an 
> absolute value
> 
>
> Key: HAWQ-570
> URL: https://issues.apache.org/jira/browse/HAWQ-570
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Resource Manager
>Reporter: Yi Jin
>Assignee: Yi Jin
>
> Resource manager measure queue capacity using absolute value causes 
> validation error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-570) YARN queue report should be a relative percentage value instead of an absolute value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207739#comment-15207739
 ] 

ASF GitHub Bot commented on HAWQ-570:
-

Github user linwen commented on the pull request:

https://github.com/apache/incubator-hawq/pull/488#issuecomment-200133031
  
+1 


> YARN queue report should be a relative percentage value instead of an 
> absolute value
> 
>
> Key: HAWQ-570
> URL: https://issues.apache.org/jira/browse/HAWQ-570
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Resource Manager
>Reporter: Yi Jin
>Assignee: Yi Jin
>
> Resource manager measure queue capacity using absolute value causes 
> validation error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HAWQ-578) Avoid validating whether total allocated resource is more than maximum YARN resource queue capacity

[Yi Jin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yi Jin reassigned HAWQ-578:
---

Assignee: Yi Jin  (was: Lei Chang)

> Avoid validating whether total allocated resource is more than maximum YARN 
> resource queue capacity
> ---
>
> Key: HAWQ-578
> URL: https://issues.apache.org/jira/browse/HAWQ-578
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Resource Manager
>Reporter: Yi Jin
>Assignee: Yi Jin
>
> YARN may provides more than maximum possible count of containers. And 
> Resource manager should not accept more containers more than the target 
> segment's resource limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-578) Avoid validating whether total allocated resource is more than maximum YARN resource queue capacity

[Yi Jin (JIRA)]

Yi Jin created HAWQ-578:
---

 Summary: Avoid validating whether total allocated resource is more 
than maximum YARN resource queue capacity
 Key: HAWQ-578
 URL: https://issues.apache.org/jira/browse/HAWQ-578
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Resource Manager
Reporter: Yi Jin
Assignee: Lei Chang


YARN may provides more than maximum possible count of containers. And Resource 
manager should not accept more containers more than the target segment's 
resource limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-559) QD hangs when QE is killed after connected to QD

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207736#comment-15207736
 ] 

ASF GitHub Bot commented on HAWQ-559:
-

Github user ictmalili closed the pull request at:

https://github.com/apache/incubator-hawq/pull/472


> QD hangs when QE is killed after connected to QD
> 
>
> Key: HAWQ-559
> URL: https://issues.apache.org/jira/browse/HAWQ-559
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Dispatcher
>Affects Versions: 2.0.0
> Environment: mac os X 10.10
>Reporter: Chunling Wang
>Assignee: Lili Ma
> Fix For: 2.0.0
>
>
> When the first query finishes, the QE is still alive. Then we run the second 
> query. After the thread of QD is created and bind to QE but not send data to 
> QE, we kill this QE and find QD hangs.
> Here is the backtrace when QD hangs:
> {code}
> * thread #1: tid = 0x1c4afd, 0x7fff890355be libsystem_kernel.dylib`poll + 
> 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
>   * frame #0: 0x7fff890355be libsystem_kernel.dylib`poll + 10
> frame #1: 0x00010745692c postgres`receiveChunksUDP [inlined] 
> udpSignalPoll + 42 at ic_udp.c:2882
> frame #2: 0x000107456902 postgres`receiveChunksUDP + 26 at 
> ic_udp.c:2715
> frame #3: 0x0001074568e8 postgres`receiveChunksUDP [inlined] 
> waitOnCondition(timeout_us=25) + 82 at ic_udp.c:1599
> frame #4: 0x000107456896 
> postgres`receiveChunksUDP(pTransportStates=0x7ff2a381ae48, 
> pEntry=0x7ff2a18f2230, motNodeID=, 
> srcRoute=0x7fff58c0ce96, conn=, inTeardown='\0') + 726 at 
> ic_udp.c:4039
> frame #5: 0x000107452a86 postgres`RecvTupleChunkFromAnyUDP [inlined] 
> RecvTupleChunkFromAnyUDP_Internal + 498 at ic_udp.c:4146
> frame #6: 0x000107452894 
> postgres`RecvTupleChunkFromAnyUDP(mlStates=, 
> transportStates=, motNodeID=1, srcRoute=0x7fff58c0ce96) + 
> 100 at ic_udp.c:4167
> frame #7: 0x000107442254 postgres`RecvTupleFrom [inlined] 
> processIncomingChunks(mlStates=0x7ff2a3812a30, 
> transportStates=0x7ff2a381ae48, motNodeID=1, srcRoute=) + 34 
> at cdbmotion.c:684
> frame #8: 0x000107442232 
> postgres`RecvTupleFrom(mlStates=0x7ff2a3812a30, 
> transportStates=, motNodeID=1, tup_i=0x7fff58c0cf00, 
> srcRoute=-100) + 370 at cdbmotion.c:610
> frame #9: 0x0001071c8778 postgres`ExecMotion [inlined] 
> execMotionUnsortedReceiver(node=) + 57 at nodeMotion.c:466
> frame #10: 0x0001071c873f postgres`ExecMotion(node=) + 
> 1071 at nodeMotion.c:298
> frame #11: 0x0001071a4835 
> postgres`ExecProcNode(node=0x7ff2a38164b8) + 613 at execProcnode.c:999
> frame #12: 0x0001071b9f82 postgres`ExecAgg + 104 at nodeAgg.c:1163
> frame #13: 0x0001071b9f1a postgres`ExecAgg + 316 at nodeAgg.c:1693
> frame #14: 0x0001071b9dde postgres`ExecAgg(node=0x7ff2a3815348) + 
> 126 at nodeAgg.c:1138
> frame #15: 0x0001071a4803 
> postgres`ExecProcNode(node=0x7ff2a3815348) + 563 at execProcnode.c:979
> frame #16: 0x00010719ecfd 
> postgres`ExecutePlan(estate=0x7ff2a3814e30, planstate=0x7ff2a3815348, 
> operation=CMD_SELECT, numberTuples=0, direction=, 
> dest=0x7ff2a28db178) + 1181 at execMain.c:3218
> frame #17: 0x00010719e619 
> postgres`ExecutorRun(queryDesc=0x7ff2a3811f00, 
> direction=ForwardScanDirection, count=0) + 569 at execMain.c:1213
> frame #18: 0x0001072e7fc2 postgres`PortalRun + 14 at pquery.c:1649
> frame #19: 0x0001072e7fb4 
> postgres`PortalRun(portal=0x7ff2a1893e30, count=, 
> isTopLevel='\x01', dest=, altdest=0x7ff2a28db178, 
> completionTag=0x7fff58c0d530) + 1124 at pquery.c:1471
> frame #20: 0x0001072e4a8e 
> postgres`exec_simple_query(query_string=0x7ff2a380fe30, 
> seqServerHost=0x, seqServerPort=-1) + 2078 at postgres.c:1745
> frame #21: 0x0001072e0c4c postgres`PostgresMain(argc=, 
> argv=, username=0x7ff2a201bcf0) + 9404 at postgres.c:4754
> frame #22: 0x00010729a002 postgres`ServerLoop [inlined] BackendRun + 
> 105 at postmaster.c:5889
> frame #23: 0x000107299f99 postgres`ServerLoop at postmaster.c:5484
> frame #24: 0x000107299f99 postgres`ServerLoop + 9593 at 
> postmaster.c:2163
> frame #25: 0x000107296f3b postgres`PostmasterMain(argc=, 
> argv=) + 5019 at postmaster.c:1454
> frame #26: 0x000107200ca9 postgres`main(argc=9, 
> argv=0x7ff2a141eef0) + 1433 at main.c:209
> frame #27: 0x7fff95e8c5c9 libdyld.dylib`start + 1
>   thread #2: tid = 0x1c4afe, 0x7fff890355be libsystem_kernel.dylib`poll + 
> 10
> frame #0: 0x7fff890355be libsystem_kernel.dylib`poll + 10
> frame #1: 0x00010744d8e3 postgres`rxThreadFunc(arg=) + 
> 

[jira] [Commented] (HAWQ-574) Memory Leak in dispatcher

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207735#comment-15207735
 ] 

ASF GitHub Bot commented on HAWQ-574:
-

Github user ictmalili closed the pull request at:

https://github.com/apache/incubator-hawq/pull/483


> Memory Leak in dispatcher
> -
>
> Key: HAWQ-574
> URL: https://issues.apache.org/jira/browse/HAWQ-574
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Dispatcher
>Reporter: Lili Ma
>Assignee: Lili Ma
> Fix For: 2.0.0
>
>
> SIZE:2,352 TYPE:direct_lost REPEAT:6
> STACK:
> ('calloc', 'vg_replace_malloc.c:711')
> ('cdbdisp_returnResults', 'cdbdisp.c:356')
> ('dispatch_statement', 'dispatcher.c:1490')
> ('dispatch_statement_node', 'dispatcher.c:1547')
> ('PerformSharedStorageOpTasks', 'cdbsharedstorageop.c:161')
> ('CreateAppendOnlyParquetSegFileForRelationOnMaster', 
> 'execMain.c:2664')
> ('CreateAppendOnlyParquetSegFileOnMaster', 'execMain.c:2469')
> ('InitializeResultRelations', 'execMain.c:1774')
> ('InitPlan', 'execMain.c:1959')
> ('ExecutorStart', 'execMain.c:697')
> ('ProcessQuery', 'pquery.c:316')
> ('PortalRunMulti', 'pquery.c:1969')
> ('PortalRun', 'pquery.c:1514')
> ('exec_simple_query', 'postgres.c:1745')
> SIZE:16 TYPE:direct_lost REPEAT:2
> STACK:
> ('malloc', 'vg_replace_malloc.c:299')
> ('pqParseInput3', 'fe-protocol3.c:596')
> ('parseInput', 'fe-exec.c:1812')
> ('PQisBusy', 'fe-exec.c:1827')
> ('executormgr_consume', 'executormgr.c:481')
> ('dispmgt_thread_func_run', 'dispatcher_mgt.c:387')
> ('workermgr_thread_func', 'workermgr.c:191')
> ('start_thread', '/lib64/libpthread-2.12.so')
> ('clone', '/lib64/libc-2.12.so')



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HAWQ-574) Memory Leak in dispatcher

[Lili Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lili Ma resolved HAWQ-574.
--
   Resolution: Fixed
Fix Version/s: 2.0.0

> Memory Leak in dispatcher
> -
>
> Key: HAWQ-574
> URL: https://issues.apache.org/jira/browse/HAWQ-574
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Dispatcher
>Reporter: Lili Ma
>Assignee: Lili Ma
> Fix For: 2.0.0
>
>
> SIZE:2,352 TYPE:direct_lost REPEAT:6
> STACK:
> ('calloc', 'vg_replace_malloc.c:711')
> ('cdbdisp_returnResults', 'cdbdisp.c:356')
> ('dispatch_statement', 'dispatcher.c:1490')
> ('dispatch_statement_node', 'dispatcher.c:1547')
> ('PerformSharedStorageOpTasks', 'cdbsharedstorageop.c:161')
> ('CreateAppendOnlyParquetSegFileForRelationOnMaster', 
> 'execMain.c:2664')
> ('CreateAppendOnlyParquetSegFileOnMaster', 'execMain.c:2469')
> ('InitializeResultRelations', 'execMain.c:1774')
> ('InitPlan', 'execMain.c:1959')
> ('ExecutorStart', 'execMain.c:697')
> ('ProcessQuery', 'pquery.c:316')
> ('PortalRunMulti', 'pquery.c:1969')
> ('PortalRun', 'pquery.c:1514')
> ('exec_simple_query', 'postgres.c:1745')
> SIZE:16 TYPE:direct_lost REPEAT:2
> STACK:
> ('malloc', 'vg_replace_malloc.c:299')
> ('pqParseInput3', 'fe-protocol3.c:596')
> ('parseInput', 'fe-exec.c:1812')
> ('PQisBusy', 'fe-exec.c:1827')
> ('executormgr_consume', 'executormgr.c:481')
> ('dispmgt_thread_func_run', 'dispatcher_mgt.c:387')
> ('workermgr_thread_func', 'workermgr.c:191')
> ('start_thread', '/lib64/libpthread-2.12.so')
> ('clone', '/lib64/libc-2.12.so')



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-572) Improve code coverage for dispatcher: fail_qe_after_connection & fail_qe_when_do_query & fail_qe_when_begin_parquet_scan

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207708#comment-15207708
 ] 

ASF GitHub Bot commented on HAWQ-572:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-hawq/pull/484


> Improve code coverage for dispatcher: fail_qe_after_connection & 
> fail_qe_when_do_query & fail_qe_when_begin_parquet_scan
> 
>
> Key: HAWQ-572
> URL: https://issues.apache.org/jira/browse/HAWQ-572
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Dispatcher
>Reporter: Chunling Wang
>Assignee: Lei Chang
>
> Add those fault injections:
> 1. fail_qe_after_connection 
> 2. fail_qe_when_do_query 
> 3. fail_qe_when_begin_parquet_scan
> And add test cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-543) src backend makefile fails when you try to run unittest's target

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207594#comment-15207594
 ] 

ASF GitHub Bot commented on HAWQ-543:
-

GitHub user cjcjameson opened a pull request:

https://github.com/apache/incubator-hawq/pull/490

HAWQ-543. Remove references to bootstrap_tokens.h

`bootstrap_tokens.h` isn't a file that exists anymore; was causing `make 
distprep` to fail

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apache/incubator-hawq HAWQ-543

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/490.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #490


commit a464aa34a4fc2b70c3cc4fc2a8bb08a9fcceae73
Author: C.J. Jameson and Devadass Santhosh Sampath 
Date:   2016-03-23T00:04:05Z

HAWQ-543. Remove references to bootstrap_tokens.h

Isn't a file that exists anymore; was causing `make distprep` to fail




> src backend makefile fails when you try to run unittest's target
> 
>
> Key: HAWQ-543
> URL: https://issues.apache.org/jira/browse/HAWQ-543
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Build
>Reporter: C.J. Jameson
>Assignee: Lei Chang
>
> When we try to package the software, we see a failure



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-465) Implement stored procedure to return fields metainfo from PXF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207577#comment-15207577
 ] 

ASF GitHub Bot commented on HAWQ-465:
-

Github user sansanichfb commented on the pull request:

https://github.com/apache/incubator-hawq/pull/479#issuecomment-200087621
  
@hornn we are not caching metadata in this stored procedure, we just use 
the same code path as for select from Hive tables, also we made it more 
generic. Please refer to JIRA https://issues.apache.org/jira/browse/HAWQ-393 
and to design doc 
https://docs.google.com/document/d/1-P9eEfcS9SZGDMDRNc3xt8cluDL-BghICRhYz9F-MLU/edit.


> Implement stored procedure to return fields metainfo from PXF
> -
>
> Key: HAWQ-465
> URL: https://issues.apache.org/jira/browse/HAWQ-465
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Hcatalog, PXF
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
> Fix For: 2.0.0
>
>
> User should be able to call built-in function:
> {code}
> select pxf_get_object_fields('source_name', 'container_name', 'object_name');
>  pxf_get_object_fields 
> {code}
> to retrieve all metadata for given source, container, object.
> Input parameters:
> ||Name||Type||Mode||
> |profile|text|IN|
> |pattern|text|IN|
> |path|text|OUT|
> |objectname|text|OUT|
> |fieldname|text|OUT|
> |fieldtype|text|OUT|
> |fieldmodifiers|text|OUT|
> Example:
> {noformat}
> # select * from pxf_get_object_fields('Hive', 'default/tab*');
>   path  | objectname  | fieldname   |fieldtype   |  fieldmodifiers |
> -+-+-- 
> ++---+
> default |  table1 |col1 |int4   | not 
> null   |
> default |  table1 |col2 |text   | 
>|
> default |  table2 |col1 |int4   | not 
> null   |
> default |  table2 |col2 |text   | 
>|
> (4 rows)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-465) Implement stored procedure to return fields metainfo from PXF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207564#comment-15207564
 ] 

ASF GitHub Bot commented on HAWQ-465:
-

Github user sansanichfb commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/479#discussion_r57090315
  
--- Diff: src/backend/utils/adt/pxf_functions.c ---
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include "catalog/external/externalmd.h"
+#include "postgres.h"
+#include "fmgr.h"
+#include "funcapi.h"
+#include "utils/builtins.h"
+
+
+typedef struct ObjectContext
+{
+   ListCell *current_object;
+   ListCell *current_field;
+} ObjectContext;
+
+ListCell* pxf_object_fields_enum_start(text *profile, text *pattern);
+ObjectContext* pxf_object_fields_enum_next(ObjectContext *object_context);
+void pxf_object_fields_enum_end(void);
+
+ListCell*
+pxf_object_fields_enum_start(text *profile, text *pattern)
+{
+   List *objects = NIL;
+
+   char *profile_cstr = text_to_cstring(profile);
+   char *pattern_cstr = text_to_cstring(pattern);
+
+   objects = get_pxf_object_metadata(profile_cstr, pattern_cstr, NULL);
+
+   return list_head(objects);
+}
+
+ObjectContext*
+pxf_object_fields_enum_next(ObjectContext *object_context)
+{
+
+   //first time call
+   if (object_context->current_object && !object_context->current_field)
+   object_context->current_field = list_head(((PxfItem *) 
lfirst(object_context->current_object))->fields);
+
+   //next field for the same object
+   else if lnext(object_context->current_field)
+   object_context->current_field = 
lnext(object_context->current_field);
+   //next table
+   else if lnext(object_context->current_object)
+   {
+   object_context->current_object = 
lnext(object_context->current_object);
+   object_context->current_field = list_head(((PxfItem *) 
lfirst(object_context->current_object))->fields);
+
+   //no objects, no fields left
+   } else
+   object_context = NULL;
+
+   return object_context;
+}
+
+void pxf_object_fields_enum_end(void)
+{
+   //cleanup
+}
+
+Datum pxf_get_object_fields(PG_FUNCTION_ARGS)
+{
+   MemoryContext oldcontext;
+   FuncCallContext *funcctx;
+   HeapTuple tuple;
+   Datum result;
+   Datum values[4];
+   bool nulls[4];
+
+   ObjectContext *object_context;
+
+   text *profile = PG_GETARG_TEXT_P(0);
+   text *pattern = PG_GETARG_TEXT_P(1);
+
+   /* stuff done only on the first call of the function */
+   if (SRF_IS_FIRSTCALL())
+   {
+   TupleDesc tupdesc;
+
+   /* create a function context for cross-call persistence */
+   funcctx = SRF_FIRSTCALL_INIT();
+
+   /*
+* switch to memory context appropriate for multiple function 
calls
+*/
+   oldcontext = 
MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+   /* initialize object fileds metadata scanning code */
+   object_context = (ObjectContext *) 
palloc0(sizeof(ObjectContext));
+   object_context->current_object = 
pxf_object_fields_enum_start(profile, pattern);
+   funcctx->user_fctx = (void *) object_context;
+
+   /*
+* build tupdesc for result tuples. This must match this 
function's
+* pg_proc entry!
+*/
+   tupdesc = CreateTemplateTupleDesc(4, false);
+   TupleDescInitEntry(tupdesc, (AttrNumber) 1, "path",
+   TEXTOID, -1, 0);
+   TupleDescInitEntry(tupdesc, (AttrNumber) 2, "objectname",
+   TEXTOID, -1, 0);
+   TupleDescInitEntry(tupdesc, (AttrNumber) 3, "columnname",
+   TEXTOID, -1, 0);
+

[jira] [Commented] (HAWQ-465) Implement stored procedure to return fields metainfo from PXF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207559#comment-15207559
 ] 

ASF GitHub Bot commented on HAWQ-465:
-

Github user sansanichfb commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/479#discussion_r57090072
  
--- Diff: src/include/catalog/namespace.h ---
@@ -95,4 +95,6 @@ extern char *namespace_search_path;
 
 extern List *fetch_search_path(bool includeImplicit);
 
+#define HiveProfileName "Hive"
--- End diff --

Because only namespace.c uses it.


> Implement stored procedure to return fields metainfo from PXF
> -
>
> Key: HAWQ-465
> URL: https://issues.apache.org/jira/browse/HAWQ-465
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Hcatalog, PXF
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
> Fix For: 2.0.0
>
>
> User should be able to call built-in function:
> {code}
> select pxf_get_object_fields('source_name', 'container_name', 'object_name');
>  pxf_get_object_fields 
> {code}
> to retrieve all metadata for given source, container, object.
> Input parameters:
> ||Name||Type||Mode||
> |profile|text|IN|
> |pattern|text|IN|
> |path|text|OUT|
> |objectname|text|OUT|
> |fieldname|text|OUT|
> |fieldtype|text|OUT|
> |fieldmodifiers|text|OUT|
> Example:
> {noformat}
> # select * from pxf_get_object_fields('Hive', 'default/tab*');
>   path  | objectname  | fieldname   |fieldtype   |  fieldmodifiers |
> -+-+-- 
> ++---+
> default |  table1 |col1 |int4   | not 
> null   |
> default |  table1 |col2 |text   | 
>|
> default |  table2 |col1 |int4   | not 
> null   |
> default |  table2 |col2 |text   | 
>|
> (4 rows)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-465) Implement stored procedure to return fields metainfo from PXF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207544#comment-15207544
 ] 

ASF GitHub Bot commented on HAWQ-465:
-

Github user sansanichfb commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/479#discussion_r57089182
  
--- Diff: src/backend/catalog/external/externalmd.c ---
@@ -45,22 +46,25 @@
 #include "utils/numeric.h"
 #include "utils/guc.h"
 
-static HCatalogTable *ParseHCatalogTable(struct json_object *hcatalogMD);
-static void LoadHCatalogEntry(HCatalogTable *hcatalogTable);
-static Oid LoadHCatalogNamespace(const char *namespaceName);
-static void LoadHCatalogTable(Oid namespaceOid, HCatalogTable 
*hcatalogTable);
-static void LoadHCatalogType(Oid relid, Oid reltypeoid, NameData relname, 
Oid relnamespaceoid);
-static void LoadHCatalogDistributionPolicy(Oid relid, HCatalogTable 
*hcatalogTable);
-static void LoadHCatalogExtTable(Oid relid, HCatalogTable *hcatalogTable);
-static void LoadHCatalogColumns(Oid relid, List *columns);
+
+List *ParsePxfEntries(StringInfo json, char *profile, Oid dboid);
--- End diff --

Sure, made them static.


> Implement stored procedure to return fields metainfo from PXF
> -
>
> Key: HAWQ-465
> URL: https://issues.apache.org/jira/browse/HAWQ-465
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Hcatalog, PXF
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
> Fix For: 2.0.0
>
>
> User should be able to call built-in function:
> {code}
> select pxf_get_object_fields('source_name', 'container_name', 'object_name');
>  pxf_get_object_fields 
> {code}
> to retrieve all metadata for given source, container, object.
> Input parameters:
> ||Name||Type||Mode||
> |profile|text|IN|
> |pattern|text|IN|
> |path|text|OUT|
> |objectname|text|OUT|
> |fieldname|text|OUT|
> |fieldtype|text|OUT|
> |fieldmodifiers|text|OUT|
> Example:
> {noformat}
> # select * from pxf_get_object_fields('Hive', 'default/tab*');
>   path  | objectname  | fieldname   |fieldtype   |  fieldmodifiers |
> -+-+-- 
> ++---+
> default |  table1 |col1 |int4   | not 
> null   |
> default |  table1 |col2 |text   | 
>|
> default |  table2 |col1 |int4   | not 
> null   |
> default |  table2 |col2 |text   | 
>|
> (4 rows)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-543) src backend makefile fails when you try to run unittest's target

[C.J. Jameson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207518#comment-15207518
 ] 

C.J. Jameson commented on HAWQ-543:
---

from the directory `src/backend/`, try to run `make distprep`

Observe the error:
`make[1]: *** No rule to make target `bootstrap_tokens.h'.  Stop.`

The bootstrap_tokens.h file doesn't exist.

> src backend makefile fails when you try to run unittest's target
> 
>
> Key: HAWQ-543
> URL: https://issues.apache.org/jira/browse/HAWQ-543
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Build
>Reporter: C.J. Jameson
>Assignee: Lei Chang
>
> When we try to package the software, we see a failure



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-563) Restore guc hawq_rm_nvseg_variance_amon_seg_respool_limit and fix its bug when resource manager resets resource

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207402#comment-15207402
 ] 

ASF GitHub Bot commented on HAWQ-563:
-

Github user jiny2 closed the pull request at:

https://github.com/apache/incubator-hawq/pull/474


> Restore guc hawq_rm_nvseg_variance_amon_seg_respool_limit and fix its bug 
> when resource manager resets resource
> ---
>
> Key: HAWQ-563
> URL: https://issues.apache.org/jira/browse/HAWQ-563
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Resource Manager
>Reporter: Yi Jin
>Assignee: Yi Jin
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-570) YARN queue report should be a relative percentage value instead of an absolute value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207404#comment-15207404
 ] 

ASF GitHub Bot commented on HAWQ-570:
-

GitHub user jiny2 opened a pull request:

https://github.com/apache/incubator-hawq/pull/488

HAWQ-570. YARN queue report should be a relative percentage value instead 
of an absolute value



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jiny2/incubator-hawq HAWQ-570

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/488.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #488


commit bb4a69042854019044568cb64990f2c9baf09616
Author: YI JIN 
Date:   2016-03-22T22:02:11Z

HAWQ-570. YARN queue report should be a relative percentage value instead 
of an absolute value




> YARN queue report should be a relative percentage value instead of an 
> absolute value
> 
>
> Key: HAWQ-570
> URL: https://issues.apache.org/jira/browse/HAWQ-570
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Resource Manager
>Reporter: Yi Jin
>Assignee: Yi Jin
>
> Resource manager measure queue capacity using absolute value causes 
> validation error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-561) Wrongly check variance of number of virtual segments

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207403#comment-15207403
 ] 

ASF GitHub Bot commented on HAWQ-561:
-

Github user jiny2 closed the pull request at:

https://github.com/apache/incubator-hawq/pull/471


> Wrongly check variance of number of virtual segments
> 
>
> Key: HAWQ-561
> URL: https://issues.apache.org/jira/browse/HAWQ-561
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Resource Manager
>Reporter: Yi Jin
>Assignee: Yi Jin
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-465) Implement stored procedure to return fields metainfo from PXF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207229#comment-15207229
 ] 

ASF GitHub Bot commented on HAWQ-465:
-

Github user sansanichfb commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/479#discussion_r57063009
  
--- Diff: src/backend/catalog/external/externalmd.c ---
@@ -23,10 +23,12 @@
  *  Author: antova
  *
  *
- * Utilities for loading external hcatalog metadata
+ * Utilities for loading external PXF metadata
  *
  */
 
+#include "catalog/external/externalmd.h"
--- End diff --

Sure


> Implement stored procedure to return fields metainfo from PXF
> -
>
> Key: HAWQ-465
> URL: https://issues.apache.org/jira/browse/HAWQ-465
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Hcatalog, PXF
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
> Fix For: 2.0.0
>
>
> User should be able to call built-in function:
> {code}
> select pxf_get_object_fields('source_name', 'container_name', 'object_name');
>  pxf_get_object_fields 
> {code}
> to retrieve all metadata for given source, container, object.
> Input parameters:
> ||Name||Type||Mode||
> |profile|text|IN|
> |pattern|text|IN|
> |path|text|OUT|
> |objectname|text|OUT|
> |fieldname|text|OUT|
> |fieldtype|text|OUT|
> |fieldmodifiers|text|OUT|
> Example:
> {noformat}
> # select * from pxf_get_object_fields('Hive', 'default/tab*');
>   path  | objectname  | fieldname   |fieldtype   |  fieldmodifiers |
> -+-+-- 
> ++---+
> default |  table1 |col1 |int4   | not 
> null   |
> default |  table1 |col2 |text   | 
>|
> default |  table2 |col1 |int4   | not 
> null   |
> default |  table2 |col2 |text   | 
>|
> (4 rows)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-465) Implement stored procedure to return fields metainfo from PXF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207218#comment-15207218
 ] 

ASF GitHub Bot commented on HAWQ-465:
-

Github user sansanichfb commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/479#discussion_r57062325
  
--- Diff: src/backend/catalog/external/externalmd.c ---
@@ -438,18 +444,18 @@ void LoadHCatalogColumns(Oid relid, List *columns)
AttrNumber attno = 1;
foreach(lc, columns)
{
-   HCatalogColumn *hcatCol = lfirst(lc);
+   PxfField *hcatCol = lfirst(lc);
--- End diff --

Sure, updated.


> Implement stored procedure to return fields metainfo from PXF
> -
>
> Key: HAWQ-465
> URL: https://issues.apache.org/jira/browse/HAWQ-465
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Hcatalog, PXF
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
> Fix For: 2.0.0
>
>
> User should be able to call built-in function:
> {code}
> select pxf_get_object_fields('source_name', 'container_name', 'object_name');
>  pxf_get_object_fields 
> {code}
> to retrieve all metadata for given source, container, object.
> Input parameters:
> ||Name||Type||Mode||
> |profile|text|IN|
> |pattern|text|IN|
> |path|text|OUT|
> |objectname|text|OUT|
> |fieldname|text|OUT|
> |fieldtype|text|OUT|
> |fieldmodifiers|text|OUT|
> Example:
> {noformat}
> # select * from pxf_get_object_fields('Hive', 'default/tab*');
>   path  | objectname  | fieldname   |fieldtype   |  fieldmodifiers |
> -+-+-- 
> ++---+
> default |  table1 |col1 |int4   | not 
> null   |
> default |  table1 |col2 |text   | 
>|
> default |  table2 |col1 |int4   | not 
> null   |
> default |  table2 |col2 |text   | 
>|
> (4 rows)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-577) Stream PXF metadata response

[Shivram Mani (JIRA)]

Shivram Mani created HAWQ-577:
-

 Summary: Stream PXF metadata response 
 Key: HAWQ-577
 URL: https://issues.apache.org/jira/browse/HAWQ-577
 Project: Apache HAWQ
  Issue Type: Bug
  Components: PXF
Reporter: Shivram Mani
Assignee: Goden Yao


getMetadata api returns the metadata corresponding to the user specified 
pattern. There is no limit to the #of tables the pattern can correspond do and 
the current approach of building in memory the json object might not scale.

We needed to serialize them inside a streaming object similar to the approach 
used for streaming the FragmentsResponse

The same applies also for the debug function that prints metadata of all the 
items - if there are too many of them the StringBuilder will run out of memory. 
The solution in the fragments case was to print a log of one fragment at a time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HAWQ-577) Stream PXF metadata response

[Shivram Mani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shivram Mani reassigned HAWQ-577:
-

Assignee: Shivram Mani  (was: Goden Yao)

> Stream PXF metadata response 
> -
>
> Key: HAWQ-577
> URL: https://issues.apache.org/jira/browse/HAWQ-577
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: PXF
>Reporter: Shivram Mani
>Assignee: Shivram Mani
>
> getMetadata api returns the metadata corresponding to the user specified 
> pattern. There is no limit to the #of tables the pattern can correspond do 
> and the current approach of building in memory the json object might not 
> scale.
> We needed to serialize them inside a streaming object similar to the approach 
> used for streaming the FragmentsResponse
> The same applies also for the debug function that prints metadata of all the 
> items - if there are too many of them the StringBuilder will run out of 
> memory. The solution in the fragments case was to print a log of one fragment 
> at a time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-465) Implement stored procedure to return fields metainfo from PXF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207036#comment-15207036
 ] 

ASF GitHub Bot commented on HAWQ-465:
-

Github user shivzone commented on the pull request:

https://github.com/apache/incubator-hawq/pull/479#issuecomment-199961634
  
@hornn the reason we didn't want to bump up the pxf api version was we 
haven't quite made a release on hawq git based on v14 and wanted to stick to 
this version until then


> Implement stored procedure to return fields metainfo from PXF
> -
>
> Key: HAWQ-465
> URL: https://issues.apache.org/jira/browse/HAWQ-465
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Hcatalog, PXF
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
> Fix For: 2.0.0
>
>
> User should be able to call built-in function:
> {code}
> select pxf_get_object_fields('source_name', 'container_name', 'object_name');
>  pxf_get_object_fields 
> {code}
> to retrieve all metadata for given source, container, object.
> Input parameters:
> ||Name||Type||Mode||
> |profile|text|IN|
> |pattern|text|IN|
> |path|text|OUT|
> |objectname|text|OUT|
> |fieldname|text|OUT|
> |fieldtype|text|OUT|
> |fieldmodifiers|text|OUT|
> Example:
> {noformat}
> # select * from pxf_get_object_fields('Hive', 'default/tab*');
>   path  | objectname  | fieldname   |fieldtype   |  fieldmodifiers |
> -+-+-- 
> ++---+
> default |  table1 |col1 |int4   | not 
> null   |
> default |  table1 |col2 |text   | 
>|
> default |  table2 |col1 |int4   | not 
> null   |
> default |  table2 |col2 |text   | 
>|
> (4 rows)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-465) Implement stored procedure to return fields metainfo from PXF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207007#comment-15207007
 ] 

ASF GitHub Bot commented on HAWQ-465:
-

Github user hornn commented on the pull request:

https://github.com/apache/incubator-hawq/pull/479#issuecomment-199957300
  
Not sure I understood the purpose of caching the external data, but nice 
implementation.
Could you please add an explanation here or in the code as to when it's is 
going to be used?

To continue the conversation in Shivram's PR of the Java changes - since 
the API between HAWQ and PXF changed, I think the protocol version should be 
bumped up.


> Implement stored procedure to return fields metainfo from PXF
> -
>
> Key: HAWQ-465
> URL: https://issues.apache.org/jira/browse/HAWQ-465
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Hcatalog, PXF
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
> Fix For: 2.0.0
>
>
> User should be able to call built-in function:
> {code}
> select pxf_get_object_fields('source_name', 'container_name', 'object_name');
>  pxf_get_object_fields 
> {code}
> to retrieve all metadata for given source, container, object.
> Input parameters:
> ||Name||Type||Mode||
> |profile|text|IN|
> |pattern|text|IN|
> |path|text|OUT|
> |objectname|text|OUT|
> |fieldname|text|OUT|
> |fieldtype|text|OUT|
> |fieldmodifiers|text|OUT|
> Example:
> {noformat}
> # select * from pxf_get_object_fields('Hive', 'default/tab*');
>   path  | objectname  | fieldname   |fieldtype   |  fieldmodifiers |
> -+-+-- 
> ++---+
> default |  table1 |col1 |int4   | not 
> null   |
> default |  table1 |col2 |text   | 
>|
> default |  table2 |col1 |int4   | not 
> null   |
> default |  table2 |col2 |text   | 
>|
> (4 rows)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-465) Implement stored procedure to return fields metainfo from PXF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15206999#comment-15206999
 ] 

ASF GitHub Bot commented on HAWQ-465:
-

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/479#discussion_r57044714
  
--- Diff: src/test/regress/output/json_load.source ---
@@ -184,7 +184,7 @@ END TRANSACTION;
 -- negative test: duplicated tables
 BEGIN TRANSACTION;
 SELECT 
load_json_data('@abs_builddir@/data/hcatalog/multi_table_duplicates.json');
-ERROR:  relation "hcatalog.db.t" already exists
+ERROR:  relation "db.t" already exists in namespace with oid=4284481535
--- End diff --

is this oid fixed?


> Implement stored procedure to return fields metainfo from PXF
> -
>
> Key: HAWQ-465
> URL: https://issues.apache.org/jira/browse/HAWQ-465
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Hcatalog, PXF
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
> Fix For: 2.0.0
>
>
> User should be able to call built-in function:
> {code}
> select pxf_get_object_fields('source_name', 'container_name', 'object_name');
>  pxf_get_object_fields 
> {code}
> to retrieve all metadata for given source, container, object.
> Input parameters:
> ||Name||Type||Mode||
> |profile|text|IN|
> |pattern|text|IN|
> |path|text|OUT|
> |objectname|text|OUT|
> |fieldname|text|OUT|
> |fieldtype|text|OUT|
> |fieldmodifiers|text|OUT|
> Example:
> {noformat}
> # select * from pxf_get_object_fields('Hive', 'default/tab*');
>   path  | objectname  | fieldname   |fieldtype   |  fieldmodifiers |
> -+-+-- 
> ++---+
> default |  table1 |col1 |int4   | not 
> null   |
> default |  table1 |col2 |text   | 
>|
> default |  table2 |col1 |int4   | not 
> null   |
> default |  table2 |col2 |text   | 
>|
> (4 rows)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-465) Implement stored procedure to return fields metainfo from PXF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207001#comment-15207001
 ] 

ASF GitHub Bot commented on HAWQ-465:
-

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/479#discussion_r57044806
  
--- Diff: src/test/unit/mock/backend/utils/adt/pxf_functions_mock.c ---
@@ -0,0 +1,58 @@
+/*
--- End diff --

I don't auto-generated files should be committed.


> Implement stored procedure to return fields metainfo from PXF
> -
>
> Key: HAWQ-465
> URL: https://issues.apache.org/jira/browse/HAWQ-465
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Hcatalog, PXF
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
> Fix For: 2.0.0
>
>
> User should be able to call built-in function:
> {code}
> select pxf_get_object_fields('source_name', 'container_name', 'object_name');
>  pxf_get_object_fields 
> {code}
> to retrieve all metadata for given source, container, object.
> Input parameters:
> ||Name||Type||Mode||
> |profile|text|IN|
> |pattern|text|IN|
> |path|text|OUT|
> |objectname|text|OUT|
> |fieldname|text|OUT|
> |fieldtype|text|OUT|
> |fieldmodifiers|text|OUT|
> Example:
> {noformat}
> # select * from pxf_get_object_fields('Hive', 'default/tab*');
>   path  | objectname  | fieldname   |fieldtype   |  fieldmodifiers |
> -+-+-- 
> ++---+
> default |  table1 |col1 |int4   | not 
> null   |
> default |  table1 |col2 |text   | 
>|
> default |  table2 |col1 |int4   | not 
> null   |
> default |  table2 |col2 |text   | 
>|
> (4 rows)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-465) Implement stored procedure to return fields metainfo from PXF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15206996#comment-15206996
 ] 

ASF GitHub Bot commented on HAWQ-465:
-

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/479#discussion_r57044521
  
--- Diff: src/backend/utils/adt/pxf_functions.c ---
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include "catalog/external/externalmd.h"
+#include "postgres.h"
+#include "fmgr.h"
+#include "funcapi.h"
+#include "utils/builtins.h"
+
+
+typedef struct ObjectContext
+{
+   ListCell *current_object;
+   ListCell *current_field;
+} ObjectContext;
+
+ListCell* pxf_object_fields_enum_start(text *profile, text *pattern);
+ObjectContext* pxf_object_fields_enum_next(ObjectContext *object_context);
+void pxf_object_fields_enum_end(void);
+
+ListCell*
+pxf_object_fields_enum_start(text *profile, text *pattern)
+{
+   List *objects = NIL;
+
+   char *profile_cstr = text_to_cstring(profile);
+   char *pattern_cstr = text_to_cstring(pattern);
+
+   objects = get_pxf_object_metadata(profile_cstr, pattern_cstr, NULL);
+
+   return list_head(objects);
+}
+
+ObjectContext*
+pxf_object_fields_enum_next(ObjectContext *object_context)
+{
+
+   //first time call
+   if (object_context->current_object && !object_context->current_field)
+   object_context->current_field = list_head(((PxfItem *) 
lfirst(object_context->current_object))->fields);
+
+   //next field for the same object
+   else if lnext(object_context->current_field)
+   object_context->current_field = 
lnext(object_context->current_field);
+   //next table
+   else if lnext(object_context->current_object)
+   {
+   object_context->current_object = 
lnext(object_context->current_object);
+   object_context->current_field = list_head(((PxfItem *) 
lfirst(object_context->current_object))->fields);
+
+   //no objects, no fields left
+   } else
+   object_context = NULL;
+
+   return object_context;
+}
+
+void pxf_object_fields_enum_end(void)
+{
+   //cleanup
+}
+
+Datum pxf_get_object_fields(PG_FUNCTION_ARGS)
+{
+   MemoryContext oldcontext;
+   FuncCallContext *funcctx;
+   HeapTuple tuple;
+   Datum result;
+   Datum values[4];
+   bool nulls[4];
+
+   ObjectContext *object_context;
+
+   text *profile = PG_GETARG_TEXT_P(0);
+   text *pattern = PG_GETARG_TEXT_P(1);
+
+   /* stuff done only on the first call of the function */
+   if (SRF_IS_FIRSTCALL())
+   {
+   TupleDesc tupdesc;
+
+   /* create a function context for cross-call persistence */
+   funcctx = SRF_FIRSTCALL_INIT();
+
+   /*
+* switch to memory context appropriate for multiple function 
calls
+*/
+   oldcontext = 
MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+   /* initialize object fileds metadata scanning code */
+   object_context = (ObjectContext *) 
palloc0(sizeof(ObjectContext));
+   object_context->current_object = 
pxf_object_fields_enum_start(profile, pattern);
+   funcctx->user_fctx = (void *) object_context;
+
+   /*
+* build tupdesc for result tuples. This must match this 
function's
+* pg_proc entry!
+*/
+   tupdesc = CreateTemplateTupleDesc(4, false);
+   TupleDescInitEntry(tupdesc, (AttrNumber) 1, "path",
+   TEXTOID, -1, 0);
+   TupleDescInitEntry(tupdesc, (AttrNumber) 2, "objectname",
+   TEXTOID, -1, 0);
+   TupleDescInitEntry(tupdesc, (AttrNumber) 3, "columnname",
+   TEXTOID, -1, 0);
+  

[jira] [Commented] (HAWQ-465) Implement stored procedure to return fields metainfo from PXF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15206984#comment-15206984
 ] 

ASF GitHub Bot commented on HAWQ-465:
-

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/479#discussion_r57043174
  
--- Diff: src/include/catalog/namespace.h ---
@@ -95,4 +95,6 @@ extern char *namespace_search_path;
 
 extern List *fetch_search_path(bool includeImplicit);
 
+#define HiveProfileName "Hive"
--- End diff --

why is it defined here? shouldn't it be in a less general location, as it's 
only used by pxf/hcatalog?


> Implement stored procedure to return fields metainfo from PXF
> -
>
> Key: HAWQ-465
> URL: https://issues.apache.org/jira/browse/HAWQ-465
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Hcatalog, PXF
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
> Fix For: 2.0.0
>
>
> User should be able to call built-in function:
> {code}
> select pxf_get_object_fields('source_name', 'container_name', 'object_name');
>  pxf_get_object_fields 
> {code}
> to retrieve all metadata for given source, container, object.
> Input parameters:
> ||Name||Type||Mode||
> |profile|text|IN|
> |pattern|text|IN|
> |path|text|OUT|
> |objectname|text|OUT|
> |fieldname|text|OUT|
> |fieldtype|text|OUT|
> |fieldmodifiers|text|OUT|
> Example:
> {noformat}
> # select * from pxf_get_object_fields('Hive', 'default/tab*');
>   path  | objectname  | fieldname   |fieldtype   |  fieldmodifiers |
> -+-+-- 
> ++---+
> default |  table1 |col1 |int4   | not 
> null   |
> default |  table1 |col2 |text   | 
>|
> default |  table2 |col1 |int4   | not 
> null   |
> default |  table2 |col2 |text   | 
>|
> (4 rows)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HAWQ-459) Enhance Metadata Rest API to support table(s)

[Shivram Mani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shivram Mani resolved HAWQ-459.
---
Resolution: Fixed

> Enhance Metadata Rest API to support table(s)
> -
>
> Key: HAWQ-459
> URL: https://issues.apache.org/jira/browse/HAWQ-459
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Hcatalog, PXF
>Reporter: Shivram Mani
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> Enhance the metadata endpoint using /Metdata/getMetadata to support 
> retrieving metadata for generic datasources, i.e. Hive, HBase, HDFS, etc and 
> support retrieving metadata for multiple objects based on wildcard pattern
> h1.Method supported
> ||*name*||
> |GET|
> h1.Path mapped
> pxf//Metadata/getMetadata
> h1.Input parameters:
> ||Name||Description||Is mandatory?||Could be a wildcard?||Valid input values||
> |profile|Type of source to fetch data from, should support Hive.|Yes|No|hive|
> |pattern|File or table name or pattern. Patterns are supported using wildcard 
> *. |Yes|Yes|db.table or file path|
> In case of Hive, patterns corresponds to schema.table
> In case of HBase, pattern=schema.table
> In case of HDFS, pattern=absolute path to file
> h1. HTTP return codes:
> ||*Code*||*Meaning*||
> |200|found some result set|
> |400|not all required parameters passed|
> |404|didn't find any results for given parameters, empty result set|
> |405|called method other than GET|
> |500|unable to access underlying metastore(hive, hbase, hdfs etc), datasource 
> not supported, some other exceptions|
> h1. Return data format for 200 code:
> {code}
> {  
>"PXFMetadata":[  
>   {  
>  "item":{  
> "path":"default",
> "name":"abc"
>  },
>  "fields":[  
> {  
>"name":"a",
>"type":"int"
> },
> {  
>"name":"b",
>"type":"float"
> }
>  ]
>   },
>   {  
>  "item":{  
> "path":"default",
> "name":"abcdef"
>  },
>  "fields":[  
> {  
>"name":"a",
>"type":"int"
> },
> {  
>"name":"b",
>"type":"float"
> }
>  ]
>   },
>   {  
>  "item":{  
> "path":"default",
> "name":"abcdefghk"
>  },
>  "fields":[  
> {  
>"name":"a",
>"type":"int"
> },
> {  
>"name":"b",
>"type":"float"
> }
>  ]
>   }
>]
> }
> {code}
> h1. Return data format for 404 code:
> {code}
> {  
>"PXFMetadata":[]
> }
> {code}
> h1. Return data format for 500 code:
> {code}
> {  
>"PXFError":{"code" : "XXX1", "message" : "Unable to reach Hive metastore"}
> }
> {code}
> h1. Examples
> Request:
> {code}
> curl -i 
> "http://localhost:51200/pxf/v14/Metadata/getMetadata?profile=hive=default.abc*;
>  
> {code}
> Response:
> {code}
> HTTP/1.1 200 OK
> Server: Apache-Coyote/1.1
> Content-Type: application/json
> Transfer-Encoding: chunked
> Date: Fri, 26 Feb 2016 03:33:53 GMT
> {  
>"PXFMetadata":[  
>   {  
>  "item":{ 
> "path":"default",
> "name":"abc"
>  },
>  "fields":[  
> {  
>"name":"a",
>"type":"int"
> },
> {  
>"name":"b",
>"type":"float"
> }
>  ]
>   },
>   {  
>  "item":{  
> "path":"default",
> "name":"abcdef"
>  },
>  "fields":[  
> {  
>"name":"a",
>"type":"int"
> },
> {  
>"name":"b",
>"type":"float"
> }
>  ]
>   },
>   {  
>  "item":{  
> "path":"default",
> "item":"abcdefghk"
>  },
>  "fields":[  
> {  
>"name":"a",
>"type":"int"
> },
> {  
>"name":"b",
>"type":"float"
> }
>  ]
>   }
>]
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-465) Implement stored procedure to return fields metainfo from PXF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15206976#comment-15206976
 ] 

ASF GitHub Bot commented on HAWQ-465:
-

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/479#discussion_r57042640
  
--- Diff: src/backend/utils/adt/pxf_functions.c ---
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include "catalog/external/externalmd.h"
+#include "postgres.h"
+#include "fmgr.h"
+#include "funcapi.h"
+#include "utils/builtins.h"
+
+
+typedef struct ObjectContext
+{
+   ListCell *current_object;
+   ListCell *current_field;
+} ObjectContext;
+
+ListCell* pxf_object_fields_enum_start(text *profile, text *pattern);
+ObjectContext* pxf_object_fields_enum_next(ObjectContext *object_context);
+void pxf_object_fields_enum_end(void);
+
+ListCell*
+pxf_object_fields_enum_start(text *profile, text *pattern)
+{
+   List *objects = NIL;
+
+   char *profile_cstr = text_to_cstring(profile);
+   char *pattern_cstr = text_to_cstring(pattern);
+
+   objects = get_pxf_object_metadata(profile_cstr, pattern_cstr, NULL);
+
+   return list_head(objects);
+}
+
+ObjectContext*
+pxf_object_fields_enum_next(ObjectContext *object_context)
+{
+
+   //first time call
+   if (object_context->current_object && !object_context->current_field)
+   object_context->current_field = list_head(((PxfItem *) 
lfirst(object_context->current_object))->fields);
+
+   //next field for the same object
+   else if lnext(object_context->current_field)
+   object_context->current_field = 
lnext(object_context->current_field);
--- End diff --

fix indentation


> Implement stored procedure to return fields metainfo from PXF
> -
>
> Key: HAWQ-465
> URL: https://issues.apache.org/jira/browse/HAWQ-465
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Hcatalog, PXF
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
> Fix For: 2.0.0
>
>
> User should be able to call built-in function:
> {code}
> select pxf_get_object_fields('source_name', 'container_name', 'object_name');
>  pxf_get_object_fields 
> {code}
> to retrieve all metadata for given source, container, object.
> Input parameters:
> ||Name||Type||Mode||
> |profile|text|IN|
> |pattern|text|IN|
> |path|text|OUT|
> |objectname|text|OUT|
> |fieldname|text|OUT|
> |fieldtype|text|OUT|
> |fieldmodifiers|text|OUT|
> Example:
> {noformat}
> # select * from pxf_get_object_fields('Hive', 'default/tab*');
>   path  | objectname  | fieldname   |fieldtype   |  fieldmodifiers |
> -+-+-- 
> ++---+
> default |  table1 |col1 |int4   | not 
> null   |
> default |  table1 |col2 |text   | 
>|
> default |  table2 |col1 |int4   | not 
> null   |
> default |  table2 |col2 |text   | 
>|
> (4 rows)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-465) Implement stored procedure to return fields metainfo from PXF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15206974#comment-15206974
 ] 

ASF GitHub Bot commented on HAWQ-465:
-

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/479#discussion_r57042532
  
--- Diff: src/backend/utils/adt/pxf_functions.c ---
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include "catalog/external/externalmd.h"
+#include "postgres.h"
+#include "fmgr.h"
+#include "funcapi.h"
+#include "utils/builtins.h"
+
+
+typedef struct ObjectContext
+{
+   ListCell *current_object;
+   ListCell *current_field;
+} ObjectContext;
+
+ListCell* pxf_object_fields_enum_start(text *profile, text *pattern);
+ObjectContext* pxf_object_fields_enum_next(ObjectContext *object_context);
+void pxf_object_fields_enum_end(void);
+
+ListCell*
+pxf_object_fields_enum_start(text *profile, text *pattern)
+{
+   List *objects = NIL;
+
+   char *profile_cstr = text_to_cstring(profile);
+   char *pattern_cstr = text_to_cstring(pattern);
+
+   objects = get_pxf_object_metadata(profile_cstr, pattern_cstr, NULL);
+
+   return list_head(objects);
+}
+
+ObjectContext*
+pxf_object_fields_enum_next(ObjectContext *object_context)
+{
+
+   //first time call
--- End diff --

please change all comments to c convention `/* ... */`


> Implement stored procedure to return fields metainfo from PXF
> -
>
> Key: HAWQ-465
> URL: https://issues.apache.org/jira/browse/HAWQ-465
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Hcatalog, PXF
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
> Fix For: 2.0.0
>
>
> User should be able to call built-in function:
> {code}
> select pxf_get_object_fields('source_name', 'container_name', 'object_name');
>  pxf_get_object_fields 
> {code}
> to retrieve all metadata for given source, container, object.
> Input parameters:
> ||Name||Type||Mode||
> |profile|text|IN|
> |pattern|text|IN|
> |path|text|OUT|
> |objectname|text|OUT|
> |fieldname|text|OUT|
> |fieldtype|text|OUT|
> |fieldmodifiers|text|OUT|
> Example:
> {noformat}
> # select * from pxf_get_object_fields('Hive', 'default/tab*');
>   path  | objectname  | fieldname   |fieldtype   |  fieldmodifiers |
> -+-+-- 
> ++---+
> default |  table1 |col1 |int4   | not 
> null   |
> default |  table1 |col2 |text   | 
>|
> default |  table2 |col1 |int4   | not 
> null   |
> default |  table2 |col2 |text   | 
>|
> (4 rows)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-434) Add Cassandra Plugin for PXF

[Goden Yao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15206964#comment-15206964
 ] 

Goden Yao commented on HAWQ-434:


Are you referring to single node installation ?

If you only want to test cassandra - I think 8GB should be fine (you don't 
need to run Hive and Hbase at the same time.)
Use a linux machine or VM. 

To try other plugins, you can shutdown cassandra then run Hbase or Hive.


> Add Cassandra Plugin for PXF
> 
>
> Key: HAWQ-434
> URL: https://issues.apache.org/jira/browse/HAWQ-434
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: PXF
>Reporter: Goden Yao
>Assignee: Goden Yao
>  Labels: gsoc2016
>
> (Cassandra | http://cassandra.apache.org/) has been proved a popular 
> key/value storage for open source community. We had some contribution from 
> early PXF users, it'd be good to integrate the code back to HAWQ code base 
> and follow build , test policy.
> Original Contribution:
> http://pivotal-field-engineering.github.io/pxf-field/cassandra.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-465) Implement stored procedure to return fields metainfo from PXF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15206962#comment-15206962
 ] 

ASF GitHub Bot commented on HAWQ-465:
-

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/479#discussion_r57041688
  
--- Diff: src/backend/utils/adt/pxf_functions.c ---
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include "catalog/external/externalmd.h"
--- End diff --

please re-org the includes in lexicographic order (but I think postgres.h 
should be in the beginning)


> Implement stored procedure to return fields metainfo from PXF
> -
>
> Key: HAWQ-465
> URL: https://issues.apache.org/jira/browse/HAWQ-465
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Hcatalog, PXF
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
> Fix For: 2.0.0
>
>
> User should be able to call built-in function:
> {code}
> select pxf_get_object_fields('source_name', 'container_name', 'object_name');
>  pxf_get_object_fields 
> {code}
> to retrieve all metadata for given source, container, object.
> Input parameters:
> ||Name||Type||Mode||
> |profile|text|IN|
> |pattern|text|IN|
> |path|text|OUT|
> |objectname|text|OUT|
> |fieldname|text|OUT|
> |fieldtype|text|OUT|
> |fieldmodifiers|text|OUT|
> Example:
> {noformat}
> # select * from pxf_get_object_fields('Hive', 'default/tab*');
>   path  | objectname  | fieldname   |fieldtype   |  fieldmodifiers |
> -+-+-- 
> ++---+
> default |  table1 |col1 |int4   | not 
> null   |
> default |  table1 |col2 |text   | 
>|
> default |  table2 |col1 |int4   | not 
> null   |
> default |  table2 |col2 |text   | 
>|
> (4 rows)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-465) Implement stored procedure to return fields metainfo from PXF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15206958#comment-15206958
 ] 

ASF GitHub Bot commented on HAWQ-465:
-

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/479#discussion_r57041307
  
--- Diff: src/backend/catalog/external/externalmd.c ---
@@ -129,45 +135,45 @@ HCatalogTable *ParseHCatalogTable(struct json_object 
*hcatalogMD)
const int numModifiers = 
json_object_array_length(jsonModifiers);
Assert(2 >= numModifiers);

-   hcatalogCol->nTypeModifiers = numModifiers;
+   pxfField->nTypeModifiers = numModifiers;
for (int j = 0; j < numModifiers; j++)
{
struct json_object *jsonMod = 
json_object_array_get_idx(jsonModifiers, j);
-   hcatalogCol->typeModifiers[j] = 
json_object_get_int(jsonMod);
+   pxfField->typeModifiers[j] = 
json_object_get_int(jsonMod);

-   elog(DEBUG1, "modifier[%d]: %d", j, 
hcatalogCol->typeModifiers[j]);
+   elog(DEBUG1, "modifier[%d]: %d", j, 
pxfField->typeModifiers[j]);
}
}
-   hcatalogTable->columns = lappend(hcatalogTable->columns, 
hcatalogCol);
+   pxfItem->fields = lappend(pxfItem->fields, pxfField);
}
 
-   return hcatalogTable;
+   return pxfItem;
 }
 
 /*
- * LoadHcatalogTable
- * Load the given hcatalog table into in-memory heap tables
+ * LoadPxfItem
+ * Load the given PXF item into in-memory heap tables
  */
-void LoadHCatalogEntry(HCatalogTable *hcatalogTable)
+void LoadPxfItem(PxfItem *pxfItem, Oid dboid)
 {
-   Oid namespaceOid = LookupNamespaceId(hcatalogTable->dbName, 
HcatalogDbOid);
+   Oid namespaceOid = LookupNamespaceId(pxfItem->path, dboid);
 
if (!OidIsValid(namespaceOid))
{
-   /* hcatalog database name has not been mapped to a namespace 
yet: create it */
-   namespaceOid = LoadHCatalogNamespace(hcatalogTable->dbName);
-   elog(DEBUG1, "No namespace found: %s. Generated new namespace 
oid: %u", hcatalogTable->dbName, namespaceOid);
+   /* external database name has not been mapped to a namespace 
yet: create it */
+   namespaceOid = LoadNamespace(pxfItem->path, dboid);
+   elog(DEBUG1, "No namespace found: %s. Generated new namespace 
oid: %u", pxfItem->path, namespaceOid);
}

-   LoadHCatalogTable(namespaceOid, hcatalogTable);
+   LoadTable(namespaceOid, pxfItem);
 }
 
 /*
- * CreateHCatalogNamespace
- * Create an entry for the given HCatalog namespace in the 
in-memory heap tables and
+ * LoadNamespace
+ * Create an entry for the given PXF namespace in the 
in-memory heap tables and
  * return the reserved namespace oid
  */
-Oid LoadHCatalogNamespace(const char *namespaceName)
+Oid LoadNamespace(const char *namespaceName, Oid dboid)
 {
--- End diff --

add assert that dboid is not invalid oid?


> Implement stored procedure to return fields metainfo from PXF
> -
>
> Key: HAWQ-465
> URL: https://issues.apache.org/jira/browse/HAWQ-465
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Hcatalog, PXF
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
> Fix For: 2.0.0
>
>
> User should be able to call built-in function:
> {code}
> select pxf_get_object_fields('source_name', 'container_name', 'object_name');
>  pxf_get_object_fields 
> {code}
> to retrieve all metadata for given source, container, object.
> Input parameters:
> ||Name||Type||Mode||
> |profile|text|IN|
> |pattern|text|IN|
> |path|text|OUT|
> |objectname|text|OUT|
> |fieldname|text|OUT|
> |fieldtype|text|OUT|
> |fieldmodifiers|text|OUT|
> Example:
> {noformat}
> # select * from pxf_get_object_fields('Hive', 'default/tab*');
>   path  | objectname  | fieldname   |fieldtype   |  fieldmodifiers |
> -+-+-- 
> ++---+
> default |  table1 |col1 |int4   | not 
> null   |
> default |  table1 |col2 |text   | 
>|
> default |  table2 |col1 |int4   | not 
> null   |
> default |  table2 |

[jira] [Commented] (HAWQ-465) Implement stored procedure to return fields metainfo from PXF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15206955#comment-15206955
 ] 

ASF GitHub Bot commented on HAWQ-465:
-

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/479#discussion_r57040862
  
--- Diff: src/backend/catalog/external/externalmd.c ---
@@ -45,22 +46,25 @@
 #include "utils/numeric.h"
 #include "utils/guc.h"
 
-static HCatalogTable *ParseHCatalogTable(struct json_object *hcatalogMD);
-static void LoadHCatalogEntry(HCatalogTable *hcatalogTable);
-static Oid LoadHCatalogNamespace(const char *namespaceName);
-static void LoadHCatalogTable(Oid namespaceOid, HCatalogTable 
*hcatalogTable);
-static void LoadHCatalogType(Oid relid, Oid reltypeoid, NameData relname, 
Oid relnamespaceoid);
-static void LoadHCatalogDistributionPolicy(Oid relid, HCatalogTable 
*hcatalogTable);
-static void LoadHCatalogExtTable(Oid relid, HCatalogTable *hcatalogTable);
-static void LoadHCatalogColumns(Oid relid, List *columns);
+
+List *ParsePxfEntries(StringInfo json, char *profile, Oid dboid);
--- End diff --

are all of these functions static? if so, please add `static`, to make the 
code more clear.


> Implement stored procedure to return fields metainfo from PXF
> -
>
> Key: HAWQ-465
> URL: https://issues.apache.org/jira/browse/HAWQ-465
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Hcatalog, PXF
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
> Fix For: 2.0.0
>
>
> User should be able to call built-in function:
> {code}
> select pxf_get_object_fields('source_name', 'container_name', 'object_name');
>  pxf_get_object_fields 
> {code}
> to retrieve all metadata for given source, container, object.
> Input parameters:
> ||Name||Type||Mode||
> |profile|text|IN|
> |pattern|text|IN|
> |path|text|OUT|
> |objectname|text|OUT|
> |fieldname|text|OUT|
> |fieldtype|text|OUT|
> |fieldmodifiers|text|OUT|
> Example:
> {noformat}
> # select * from pxf_get_object_fields('Hive', 'default/tab*');
>   path  | objectname  | fieldname   |fieldtype   |  fieldmodifiers |
> -+-+-- 
> ++---+
> default |  table1 |col1 |int4   | not 
> null   |
> default |  table1 |col2 |text   | 
>|
> default |  table2 |col1 |int4   | not 
> null   |
> default |  table2 |col2 |text   | 
>|
> (4 rows)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-465) Implement stored procedure to return fields metainfo from PXF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15206950#comment-15206950
 ] 

ASF GitHub Bot commented on HAWQ-465:
-

Github user hornn commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/479#discussion_r57040549
  
--- Diff: src/backend/catalog/external/externalmd.c ---
@@ -23,10 +23,12 @@
  *  Author: antova
  *
  *
- * Utilities for loading external hcatalog metadata
+ * Utilities for loading external PXF metadata
  *
  */
 
+#include "catalog/external/externalmd.h"
--- End diff --

please move to its right place in the include list (after 
catalog/catquery.h)


> Implement stored procedure to return fields metainfo from PXF
> -
>
> Key: HAWQ-465
> URL: https://issues.apache.org/jira/browse/HAWQ-465
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Hcatalog, PXF
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
> Fix For: 2.0.0
>
>
> User should be able to call built-in function:
> {code}
> select pxf_get_object_fields('source_name', 'container_name', 'object_name');
>  pxf_get_object_fields 
> {code}
> to retrieve all metadata for given source, container, object.
> Input parameters:
> ||Name||Type||Mode||
> |profile|text|IN|
> |pattern|text|IN|
> |path|text|OUT|
> |objectname|text|OUT|
> |fieldname|text|OUT|
> |fieldtype|text|OUT|
> |fieldmodifiers|text|OUT|
> Example:
> {noformat}
> # select * from pxf_get_object_fields('Hive', 'default/tab*');
>   path  | objectname  | fieldname   |fieldtype   |  fieldmodifiers |
> -+-+-- 
> ++---+
> default |  table1 |col1 |int4   | not 
> null   |
> default |  table1 |col2 |text   | 
>|
> default |  table2 |col1 |int4   | not 
> null   |
> default |  table2 |col2 |text   | 
>|
> (4 rows)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-465) Implement stored procedure to return fields metainfo from PXF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15206903#comment-15206903
 ] 

ASF GitHub Bot commented on HAWQ-465:
-

Github user shivzone commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/479#discussion_r57036285
  
--- Diff: src/backend/catalog/external/externalmd.c ---
@@ -438,18 +444,18 @@ void LoadHCatalogColumns(Oid relid, List *columns)
AttrNumber attno = 1;
foreach(lc, columns)
{
-   HCatalogColumn *hcatCol = lfirst(lc);
+   PxfField *hcatCol = lfirst(lc);
--- End diff --

change from hcatCol to fieldCol ?


> Implement stored procedure to return fields metainfo from PXF
> -
>
> Key: HAWQ-465
> URL: https://issues.apache.org/jira/browse/HAWQ-465
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Hcatalog, PXF
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
> Fix For: 2.0.0
>
>
> User should be able to call built-in function:
> {code}
> select pxf_get_object_fields('source_name', 'container_name', 'object_name');
>  pxf_get_object_fields 
> {code}
> to retrieve all metadata for given source, container, object.
> Input parameters:
> ||Name||Type||Mode||
> |profile|text|IN|
> |pattern|text|IN|
> |path|text|OUT|
> |objectname|text|OUT|
> |fieldname|text|OUT|
> |fieldtype|text|OUT|
> |fieldmodifiers|text|OUT|
> Example:
> {noformat}
> # select * from pxf_get_object_fields('Hive', 'default/tab*');
>   path  | objectname  | fieldname   |fieldtype   |  fieldmodifiers |
> -+-+-- 
> ++---+
> default |  table1 |col1 |int4   | not 
> null   |
> default |  table1 |col2 |text   | 
>|
> default |  table2 |col1 |int4   | not 
> null   |
> default |  table2 |col2 |text   | 
>|
> (4 rows)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-465) Implement stored procedure to return fields metainfo from PXF

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15206871#comment-15206871
 ] 

ASF GitHub Bot commented on HAWQ-465:
-

Github user shivzone commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/479#discussion_r57034300
  
--- Diff: src/backend/access/external/pxfmasterapi.c ---
@@ -145,7 +145,7 @@ void free_datanode_rest_server(PxfServer* srv)
 PxfFragmentStatsElem *get_fragments_statistics(GPHDUri* hadoop_uri,

   ClientContext *client_context)
 {
-   char *restMsg = 
concat("http://%s:%s/%s/%s/Fragmenter/getFragmentsStats?path=;, 
hadoop_uri->data);
+   char *restMsg = concat(2, 
"http://%s:%s/%s/%s/Fragmenter/getFragmentsStats?path=;, hadoop_uri->data);
--- End diff --

whats the purpose of 2 ?


> Implement stored procedure to return fields metainfo from PXF
> -
>
> Key: HAWQ-465
> URL: https://issues.apache.org/jira/browse/HAWQ-465
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Hcatalog, PXF
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
> Fix For: 2.0.0
>
>
> User should be able to call built-in function:
> {code}
> select pxf_get_object_fields('source_name', 'container_name', 'object_name');
>  pxf_get_object_fields 
> {code}
> to retrieve all metadata for given source, container, object.
> Input parameters:
> ||Name||Type||Mode||
> |profile|text|IN|
> |pattern|text|IN|
> |path|text|OUT|
> |objectname|text|OUT|
> |fieldname|text|OUT|
> |fieldtype|text|OUT|
> |fieldmodifiers|text|OUT|
> Example:
> {noformat}
> # select * from pxf_get_object_fields('Hive', 'default/tab*');
>   path  | objectname  | fieldname   |fieldtype   |  fieldmodifiers |
> -+-+-- 
> ++---+
> default |  table1 |col1 |int4   | not 
> null   |
> default |  table1 |col2 |text   | 
>|
> default |  table2 |col1 |int4   | not 
> null   |
> default |  table2 |col2 |text   | 
>|
> (4 rows)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-492) PXF Query Push-Down Feature enhancement to enable further push down to Datasource

[Michael Andre Pearce (IG) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15206843#comment-15206843
 ] 

Michael Andre Pearce (IG) commented on HAWQ-492:


Should I look there for to re-open this ticket? Or submit a new?

> PXF Query Push-Down Feature enhancement to enable further push down to 
> Datasource
> -
>
> Key: HAWQ-492
> URL: https://issues.apache.org/jira/browse/HAWQ-492
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: PXF
>Reporter: Michael Andre Pearce (IG)
>Assignee: Lei Chang
>
> PXF currently supports push down of filters to the PXF plugin. So that PXF 
> itself only needs to return the fields and rows that match the filters.
> Unfortunately it seems (and it could be simply miss-understanding) is done in 
> a way where by the PXF plugin must retrieve the complete row/entry from the 
> source then the filters are applied. With Query Filter Push-Down feature.
> With a JDBC PXF (https://github.com/kojec/pxf-field/tree/master/jdbc-pxf-ext) 
> this seems to be the case where by the SELECT * FROM table is done. 
> It would be much more efficient if the filter and fields are push down in a 
> way so that the plugin can if in its use case can actually delegate that 
> further down into datasource it is using, and retrieving and applying the 
> filters needed to reduce the data it needs to be pulled back.
> As such the PXF accessor would be able to construct SELECT fieldA fieldB FROM 
> table WHERE fieldA > 6
> It maybe that this is indeed possible, but looking at 
> https://github.com/kojec/pxf-field/tree/master/jdbc-pxf-ext and the API's and 
> documents currently available we cannot see a way to improve this jdbc pxf 
> plugin to achieve this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-492) PXF Query Push-Down Feature enhancement to enable further push down to Datasource

[Goden Yao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15206831#comment-15206831
 ] 

Goden Yao commented on HAWQ-492:


I see. I think what you need is to return Partial content of a record - in 
other words, return the columns in the SELECT query only. 
This is only achievable with columnar storage format (e.g. parquet or ORC) not 
for text, sequence , etc.

So this implementation depends on specific plugin implementation. (e.g. if we 
have a parquet plugin for PXF, this plugin can implement this logic.)

We don't have these JIRAs filed yet but we certainly welcome ideas like this 
and hope you and others may help to contribute.

> PXF Query Push-Down Feature enhancement to enable further push down to 
> Datasource
> -
>
> Key: HAWQ-492
> URL: https://issues.apache.org/jira/browse/HAWQ-492
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: PXF
>Reporter: Michael Andre Pearce (IG)
>Assignee: Lei Chang
>
> PXF currently supports push down of filters to the PXF plugin. So that PXF 
> itself only needs to return the fields and rows that match the filters.
> Unfortunately it seems (and it could be simply miss-understanding) is done in 
> a way where by the PXF plugin must retrieve the complete row/entry from the 
> source then the filters are applied. With Query Filter Push-Down feature.
> With a JDBC PXF (https://github.com/kojec/pxf-field/tree/master/jdbc-pxf-ext) 
> this seems to be the case where by the SELECT * FROM table is done. 
> It would be much more efficient if the filter and fields are push down in a 
> way so that the plugin can if in its use case can actually delegate that 
> further down into datasource it is using, and retrieving and applying the 
> filters needed to reduce the data it needs to be pulled back.
> As such the PXF accessor would be able to construct SELECT fieldA fieldB FROM 
> table WHERE fieldA > 6
> It maybe that this is indeed possible, but looking at 
> https://github.com/kojec/pxf-field/tree/master/jdbc-pxf-ext and the API's and 
> documents currently available we cannot see a way to improve this jdbc pxf 
> plugin to achieve this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HAWQ-434) Add Cassandra Plugin for PXF

[Pubudu Dodangoda (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15206175#comment-15206175
 ] 

Pubudu Dodangoda edited comment on HAWQ-434 at 3/22/16 12:21 PM:
-

Hello [~GodenYao]

I referred the existing classes of the reference cassandra plugin and also the 
actual implementations of hbase plugin. I also referred the pxf wiki page and 
got somewhat familiar. However, I just noticed that to install HAWQ, the system 
requirements are too high (16GB RAM, 8CORE CPUs). I don't have that much of 
resources. Is there a way to test the pxf plugins without having to go for such 
an amount of resources?

Thanks


was (Author: pupudu):
Hello Goden Yao,

I referred the existing classes of the reference cassandra plugin and also the 
actual implementations of hbase plugin. I also referred the pxf wiki page and 
got somewhat familiar. However, I just noticed that to install HAWQ, the system 
requirements are too high (16GB RAM, 8CORE CPUs). I don't have that much of 
resources. Is there a way to test the pxf plugins without having to go for such 
an amount of resources?

Thanks

> Add Cassandra Plugin for PXF
> 
>
> Key: HAWQ-434
> URL: https://issues.apache.org/jira/browse/HAWQ-434
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: PXF
>Reporter: Goden Yao
>Assignee: Goden Yao
>  Labels: gsoc2016
>
> (Cassandra | http://cassandra.apache.org/) has been proved a popular 
> key/value storage for open source community. We had some contribution from 
> early PXF users, it'd be good to integrate the code back to HAWQ code base 
> and follow build , test policy.
> Original Contribution:
> http://pivotal-field-engineering.github.io/pxf-field/cassandra.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-434) Add Cassandra Plugin for PXF

[Pubudu Dodangoda (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15206175#comment-15206175
 ] 

Pubudu Dodangoda commented on HAWQ-434:
---

Hello Goden Yao,

I referred the existing classes of the reference cassandra plugin and also the 
actual implementations of hbase plugin. I also referred the pxf wiki page and 
got somewhat familiar. However, I just noticed that to install HAWQ, the system 
requirements are too high (16GB RAM, 8CORE CPUs). I don't have that much of 
resources. Is there a way to test the pxf plugins without having to go for such 
an amount of resources?

Thanks

> Add Cassandra Plugin for PXF
> 
>
> Key: HAWQ-434
> URL: https://issues.apache.org/jira/browse/HAWQ-434
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: PXF
>Reporter: Goden Yao
>Assignee: Goden Yao
>  Labels: gsoc2016
>
> (Cassandra | http://cassandra.apache.org/) has been proved a popular 
> key/value storage for open source community. We had some contribution from 
> early PXF users, it'd be good to integrate the code back to HAWQ code base 
> and follow build , test policy.
> Original Contribution:
> http://pivotal-field-engineering.github.io/pxf-field/cassandra.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-564) QD hangs when connecting to resource manager

[Chunling Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15206095#comment-15206095
 ] 

Chunling Wang commented on HAWQ-564:


And 'kill -6' can cause same result.

> QD hangs when connecting to resource manager
> 
>
> Key: HAWQ-564
> URL: https://issues.apache.org/jira/browse/HAWQ-564
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Resource Manager
>Affects Versions: 2.0.0
>Reporter: Chunling Wang
>Assignee: Lei Chang
>
> When first inject panic in QE process, we run a query and segment is down. 
> After the segment is up, we run another query and get correct answer. Then we 
> inject the same panic second time. After the segment is down and then up 
> again, we run a query and find QD process hangs when connecting to resource 
> manager. Here is the backtrace when QD hangs:
> {code}
> * thread #1: tid = 0x21d8be, 0x7fff890355be libsystem_kernel.dylib`poll + 
> 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
>   * frame #0: 0x7fff890355be libsystem_kernel.dylib`poll + 10
> frame #1: 0x000101daeafe postgres`processAllCommFileDescs + 158 at 
> rmcomm_AsyncComm.c:156
> frame #2: 0x000101db85f5 
> postgres`callSyncRPCRemote(hostname=0x7f9c19e00cd0, port=5437, 
> sendbuff=0x7f9c1b918f50, sendbuffsize=80, sendmsgid=259, 
> exprecvmsgid=2307, recvsmb=, errorbuf=0x00010230c1a0, 
> errorbufsize=) + 645 at rmcomm_SyncComm.c:122
> frame #3: 0x000101db2d85 postgres`acquireResourceFromRM [inlined] 
> callSyncRPCToRM(sendbuff=0x7f9c1b918f50, sendbuffsize=, 
> sendmsgid=259, exprecvmsgid=2307, recvsmb=0x7f9c1b918e70, 
> errorbuf=, errorbufsize=1024) + 73 at rmcomm_QD2RM.c:2780
> frame #4: 0x000101db2d3c 
> postgres`acquireResourceFromRM(index=, sessionid=12, 
> slice_size=462524016, iobytes=134217728, preferred_nodes=0x7f9c1a02d398, 
> preferred_nodes_size=, max_seg_count_fix=, 
> min_seg_count_fix=, errorbuf=, 
> errorbufsize=) + 572 at rmcomm_QD2RM.c:742
> frame #5: 0x000101c979e7 postgres`AllocateResource(life=QRL_ONCE, 
> slice_size=5, iobytes=134217728, max_target_segment_num=1, 
> min_target_segment_num=1, vol_info=0x7f9c1a02d398, vol_info_size=1) + 631 
> at pquery.c:796
> frame #6: 0x000101e8c60f 
> postgres`calculate_planner_segment_num(query=, 
> resourceLife=QRL_ONCE, fullRangeTable=, 
> intoPolicy=, sliceNum=5) + 14287 at cdbdatalocality.c:4207
> frame #7: 0x000101c0f671 postgres`planner + 106 at planner.c:496
> frame #8: 0x000101c0f607 postgres`planner(parse=0x7f9c1a02a140, 
> cursorOptions=, boundParams=0x, 
> resourceLife=QRL_ONCE) + 311 at planner.c:310
> frame #9: 0x000101c8eb33 
> postgres`pg_plan_query(querytree=0x7f9c1a02a140, 
> boundParams=0x, resource_life=QRL_ONCE) + 99 at postgres.c:837
> frame #10: 0x000101c956ae postgres`exec_simple_query + 21 at 
> postgres.c:911
> frame #11: 0x000101c95699 
> postgres`exec_simple_query(query_string=0x7f9c1a028a30, 
> seqServerHost=0x, seqServerPort=-1) + 1577 at postgres.c:1671
> frame #12: 0x000101c91a4c postgres`PostgresMain(argc=, 
> argv=, username=0x7f9c1b808cf0) + 9404 at postgres.c:4754
> frame #13: 0x000101c4ae02 postgres`ServerLoop [inlined] BackendRun + 
> 105 at postmaster.c:5889
> frame #14: 0x000101c4ad99 postgres`ServerLoop at postmaster.c:5484
> frame #15: 0x000101c4ad99 postgres`ServerLoop + 9593 at 
> postmaster.c:2163
> frame #16: 0x000101c47d3b postgres`PostmasterMain(argc=, 
> argv=) + 5019 at postmaster.c:1454
> frame #17: 0x000101bb1aa9 postgres`main(argc=9, 
> argv=0x7f9c19c1eef0) + 1433 at main.c:209
> frame #18: 0x7fff95e8c5c9 libdyld.dylib`start + 1
>   thread #2: tid = 0x21d8bf, 0x7fff890355be libsystem_kernel.dylib`poll + 
> 10
> frame #0: 0x7fff890355be libsystem_kernel.dylib`poll + 10
> frame #1: 0x000101dfe723 postgres`rxThreadFunc(arg=) + 
> 2163 at ic_udp.c:6251
> frame #2: 0x7fff95e822fc libsystem_pthread.dylib`_pthread_body + 131
> frame #3: 0x7fff95e82279 libsystem_pthread.dylib`_pthread_start + 176
> frame #4: 0x7fff95e804b1 libsystem_pthread.dylib`thread_start + 13
>   thread #3: tid = 0x21d9c2, 0x7fff890343f6 
> libsystem_kernel.dylib`__select + 10
> frame #0: 0x7fff890343f6 libsystem_kernel.dylib`__select + 10
> frame #1: 0x000101e9d42e postgres`pg_usleep(microsec=) + 
> 78 at pgsleep.c:43
> frame #2: 0x000101db1a66 
> postgres`generateResourceRefreshHeartBeat(arg=0x7f9c19f02480) + 166 at 
> rmcomm_QD2RM.c:1519
> frame #3: 0x7fff95e822fc libsystem_pthread.dylib`_pthread_body + 131
> frame #4: 0x7fff95e82279 

[jira] [Assigned] (HAWQ-575) QE core dumped when report "Resume interrupt holdoff count is bad (0) (xact.c:2907)"

[Ming LI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ming LI reassigned HAWQ-575:


Assignee: Ming LI  (was: Lei Chang)

> QE core dumped when report "Resume interrupt holdoff count is bad (0) 
> (xact.c:2907)"
> 
>
> Key: HAWQ-575
> URL: https://issues.apache.org/jira/browse/HAWQ-575
> Project: Apache HAWQ
>  Issue Type: Bug
>Reporter: Ming LI
>Assignee: Ming LI
>
> Core was generated by `postgres: port  5532, gpadmin tpch_row_2... 
> 172.28.8.250(18226) con92825 seg97'.
> Program terminated with signal 6, Aborted.
> #0  0x0033d4032925 in raise () from /lib64/libc.so.6
> Missing separate debuginfos, use: debuginfo-install 
> hawq-2.0.0.0_beta-20925.x86_64
> (gdb) bt
> #0  0x0033d4032925 in raise () from /lib64/libc.so.6
> #1  0x0033d4034105 in abort () from /lib64/libc.so.6
> #2  0x00871c6e in errfinish (dummy=) at 
> elog.c:682
> #3  0x008727bb in elog_finish (elevel=, 
> fmt=) at elog.c:1459
> #4  0x004e171b in AbortTransaction () at xact.c:2907
> #5  0x004e2c45 in AbortCurrentTransaction () at xact.c:3377
> #6  0x007b01ea in PostgresMain (argc=37474312, argv=0x0, 
> username=) at postgres.c:4507
> #7  0x00763c03 in BackendRun (port=0x2373210) at postmaster.c:5889
> #8  BackendStartup (port=0x2373210) at postmaster.c:5484
> #9  0x0076435d in ServerLoop () at postmaster.c:2163
> #10 0x0076618e in PostmasterMain (argc=9, argv=0x236a5b0) at 
> postmaster.c:1454
> #11 0x006c028a in main (argc=9, argv=0x236a570) at main.c:226
> (gdb) f 3
> #3  0x008727bb in elog_finish (elevel=, 
> fmt=) at elog.c:1459
> (gdb) p *edata
> $1 = {elevel = 22, output_to_server = 1 '\001', output_to_client = 1 '\001', 
> show_funcname = 0 '\000', omit_location = 0 '\000', fatal_return = 0 '\000',
>   hide_stmt = 0 '\000', send_alert = 1 '\001', filename = 0x9cc38e "xact.c", 
> lineno = 2907, funcname = 0x9c66c0 "AbortTransaction",
>   domain = 0xafb668 "postgres-8.2", sqlerrcode = 2600, message = 0x236da50 
> "Resume interrupt holdoff count is bad (0) (xact.c:2907)", detail = 0x0,
>   detail_log = 0x0, hint = 0x0, context = 0x0, cursorpos = 0, internalpos = 
> 0, internalquery = 0x0, saved_errno = 5, stacktracearray = {0x871f7f, 
> 0x872659,
> 0x4e171b, 0x4e2c45, 0x7b01ea, 0x763c03, 0x76435d, 0x76618e, 0x6c028a, 
> 0x33d401ed1d, 0x4a17e9, 0x0 }, stacktracesize = 11,
>   printstack = 0 '\000'}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-575) QE core dumped when report "Resume interrupt holdoff count is bad (0) (xact.c:2907)"

[Ming LI (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15206036#comment-15206036
 ] 

Ming LI commented on HAWQ-575:
--

The log is below:

2016-03-19 23:44:54.621653 
PDT,"gpadmin","tpch_row_200gpn_quicklz1_part_random_gpadmin",p1372,th-1217652544,"172.28.8.250","15627",2016-03-19
 22:52:05 
PDT,1172392,con92730,cmd2,seg97,slice5,,x1172392,sx1,"ERROR","58030","could not 
read from temporary file: Input/output error",,"select ...

2016-03-19 23:44:54.649381 
PDT,"gpadmin","tpch_row_200gpn_quicklz1_part_random_gpadmin",p797594,th-1217652544,"172.28.8.250","14688",2016-03-19
 22:48:14 
PDT,1172286,con92501,cmd2,seg101,slice7,,x1172286,sx1,"FATAL","08006","connection
 to client lost",,,0,,"postgres.c",3512,
2016-03-19 23:44:54.675656 
PDT,"gpadmin","tpch_row_200gpn_quicklz1_part_random_gpadmin",p2466,th-1217652544,"172.28.8.250","18226",2016-03-19
 22:53:40 
PDT,1172688,con92825,cmd2,seg97,slice5,,x1172688,sx1,"ERROR","58030","could not 
read from temporary file: Input/output error",,"select
 nation,
 o_year,
 sum(amount) as sum_profit
from
 (
 select
 n_name as nation,
 extract(year from o_orderdate) as o_year,
 l_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity as amount
 from
 part,
 supplier,
 lineitem,
 partsupp,
 orders,
 nation
 where
 s_suppkey = l_suppkey
 and ps_suppkey = l_suppkey
 and ps_partkey = l_partkey
 and p_partkey = l_partkey
 and o_orderkey = l_orderkey
 and s_nationkey = n_nationkey
 and p_name like '%aquamarine%'
 ) as profit
group by
 nation,
 o_year
order by
 nation,
 o_year desc;",0,,"compress_nothing.c",61,
2016-03-19 23:44:54.683709 
PDT,"gpadmin","tpch_row_200gpn_quicklz1_part_random_gpadmin",p2466,th-1217652544,"172.28.8.250","18226",2016-03-19
 22:53:40 
PDT,1172688,con92825,cmd2,seg97,slice5,,x1172688,sx1,"ERROR","58030","could not 
close temporary file 
/data21/tmp/pgsql_tmp/workfile_set_HashJoin_Slice5.SzweO6/spillfile_f95: 
Input/output error",,,0,,"bfz.c",466,
2016-03-19 23:44:54.689898 
PDT,"gpadmin","tpch_row_200gpn_quicklz1_part_random_gpadmin",p2466,th-1217652544,"172.28.8.250","18226",2016-03-19
 22:53:40 
PDT,1172688,con92825,cmd2,seg97,slice5,,x1172688,sx1,"WARNING","58030","could 
not close temporary file 
/data21/tmp/pgsql_tmp/workfile_set_HashJoin_Slice5.SzweO6/spillfile_f123: 
Input/output error",,,0,,"bfz.c",466,

2016-03-19 23:45:08.582441 
PDT,"gpadmin","tpch_row_200gpn_quicklz1_part_random_gpadmin",p2466,th-1217652544,"172.28.8.250","18226",2016-03-19
 22:53:40 
PDT,1172688,con92825,cmd2,seg97,slice5,,x1172688,sx1,"PANIC","XX000","Resume 
interrupt holdoff count is bad (0) (xact.c:2907)",,,0,,"xact.c",2907,"Stack 
trace:
10x871f7f postgres  + 0x871f7f
20x872659 postgres elog_finish + 0xa9
30x4e171b postgres AbortTransaction + 0x7cb
40x4e2c45 postgres AbortCurrentTransaction + 0x25
50x7b01ea postgres PostgresMain + 0xaba
60x763c03 postgres  + 0x763c03
70x76435d postgres  + 0x76435d
80x76618e postgres PostmasterMain + 0xc7e
90x6c028a postgres main + 0x48a
10   0x33d401ed1d libc.so.6 __libc_start_main + 0xfd
11   0x4a17e9 postgres  + 0x4a17e9


>From the log above, the root cause is:
1) con92730,cmd2,seg97,slice5 reported: could not read from temporary file: 
Input/output error
2) So the transaction will be aborted. Master node will send SIGQUIT to all 
processes on segment and quit
3) con92501,cmd2,seg101,slice7: before processing SIGQUIT, it first detect that 
connection to QD error, so report FATAL.
4) con92825,cmd2,seg97,slice5: why 2 occurrence of report error here? Maybe the 
second error is called in the AbortTransaction() which will set 
InterruptHoldoffCount to 0.

> QE core dumped when report "Resume interrupt holdoff count is bad (0) 
> (xact.c:2907)"
> 
>
> Key: HAWQ-575
> URL: https://issues.apache.org/jira/browse/HAWQ-575
> Project: Apache HAWQ
>  Issue Type: Bug
>Reporter: Ming LI
>Assignee: Lei Chang
>
> Core was generated by `postgres: port  5532, gpadmin tpch_row_2... 
> 172.28.8.250(18226) con92825 seg97'.
> Program terminated with signal 6, Aborted.
> #0  0x0033d4032925 in raise () from /lib64/libc.so.6
> Missing separate debuginfos, use: debuginfo-install 
> hawq-2.0.0.0_beta-20925.x86_64
> (gdb) bt
> #0  0x0033d4032925 in raise () from /lib64/libc.so.6
> #1  0x0033d4034105 in abort () from /lib64/libc.so.6
> #2  0x00871c6e in errfinish (dummy=) at 
> elog.c:682
> #3  0x008727bb in elog_finish (elevel=, 
> fmt=) at elog.c:1459
> #4  0x004e171b in AbortTransaction () at xact.c:2907
> #5  0x004e2c45 in AbortCurrentTransaction () at xact.c:3377
> #6  0x007b01ea in PostgresMain (argc=37474312, argv=0x0, 
> username=) at postgres.c:4507
> #7  0x00763c03 

[jira] [Created] (HAWQ-575) QE core dumped when report "Resume interrupt holdoff count is bad (0) (xact.c:2907)"

[Ming LI (JIRA)]

Ming LI created HAWQ-575:


 Summary: QE core dumped when report "Resume interrupt holdoff 
count is bad (0) (xact.c:2907)"
 Key: HAWQ-575
 URL: https://issues.apache.org/jira/browse/HAWQ-575
 Project: Apache HAWQ
  Issue Type: Bug
Reporter: Ming LI
Assignee: Lei Chang


Core was generated by `postgres: port  5532, gpadmin tpch_row_2... 
172.28.8.250(18226) con92825 seg97'.
Program terminated with signal 6, Aborted.
#0  0x0033d4032925 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install 
hawq-2.0.0.0_beta-20925.x86_64
(gdb) bt
#0  0x0033d4032925 in raise () from /lib64/libc.so.6
#1  0x0033d4034105 in abort () from /lib64/libc.so.6
#2  0x00871c6e in errfinish (dummy=) at elog.c:682
#3  0x008727bb in elog_finish (elevel=, fmt=) at elog.c:1459
#4  0x004e171b in AbortTransaction () at xact.c:2907
#5  0x004e2c45 in AbortCurrentTransaction () at xact.c:3377
#6  0x007b01ea in PostgresMain (argc=37474312, argv=0x0, 
username=) at postgres.c:4507
#7  0x00763c03 in BackendRun (port=0x2373210) at postmaster.c:5889
#8  BackendStartup (port=0x2373210) at postmaster.c:5484
#9  0x0076435d in ServerLoop () at postmaster.c:2163
#10 0x0076618e in PostmasterMain (argc=9, argv=0x236a5b0) at 
postmaster.c:1454
#11 0x006c028a in main (argc=9, argv=0x236a570) at main.c:226
(gdb) f 3
#3  0x008727bb in elog_finish (elevel=, fmt=) at elog.c:1459
(gdb) p *edata
$1 = {elevel = 22, output_to_server = 1 '\001', output_to_client = 1 '\001', 
show_funcname = 0 '\000', omit_location = 0 '\000', fatal_return = 0 '\000',
  hide_stmt = 0 '\000', send_alert = 1 '\001', filename = 0x9cc38e "xact.c", 
lineno = 2907, funcname = 0x9c66c0 "AbortTransaction",
  domain = 0xafb668 "postgres-8.2", sqlerrcode = 2600, message = 0x236da50 
"Resume interrupt holdoff count is bad (0) (xact.c:2907)", detail = 0x0,
  detail_log = 0x0, hint = 0x0, context = 0x0, cursorpos = 0, internalpos = 0, 
internalquery = 0x0, saved_errno = 5, stacktracearray = {0x871f7f, 0x872659,
0x4e171b, 0x4e2c45, 0x7b01ea, 0x763c03, 0x76435d, 0x76618e, 0x6c028a, 
0x33d401ed1d, 0x4a17e9, 0x0 }, stacktracesize = 11,
  printstack = 0 '\000'}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-572) Improve code coverage for dispatcher: fail_qe_after_connection & fail_qe_when_do_query & fail_qe_when_begin_parquet_scan

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15206018#comment-15206018
 ] 

ASF GitHub Bot commented on HAWQ-572:
-

GitHub user wcl14 opened a pull request:

https://github.com/apache/incubator-hawq/pull/484

HAWQ-572. Add fault injection test cases for QE side

Add fault injection test cases for QE side.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wcl14/incubator-hawq HAWQ-572

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/484.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #484


commit 31fdef71c1d77070b1db84d37c460127487456cd
Author: Chunling Wang 
Date:   2016-03-22T08:49:07Z

HAWQ-572. Add fault injection test cases for QE side




> Improve code coverage for dispatcher: fail_qe_after_connection & 
> fail_qe_when_do_query & fail_qe_when_begin_parquet_scan
> 
>
> Key: HAWQ-572
> URL: https://issues.apache.org/jira/browse/HAWQ-572
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Dispatcher
>Reporter: Chunling Wang
>Assignee: Lei Chang
>
> Add those fault injections:
> 1. fail_qe_after_connection 
> 2. fail_qe_when_do_query 
> 3. fail_qe_when_begin_parquet_scan
> And add test cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-574) Memory Leak in dispatcher

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15206014#comment-15206014
 ] 

ASF GitHub Bot commented on HAWQ-574:
-

Github user wengyanqing commented on the pull request:

https://github.com/apache/incubator-hawq/pull/483#issuecomment-199705562
  
LGTM


> Memory Leak in dispatcher
> -
>
> Key: HAWQ-574
> URL: https://issues.apache.org/jira/browse/HAWQ-574
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Dispatcher
>Reporter: Lili Ma
>Assignee: Lili Ma
>
> SIZE:2,352 TYPE:direct_lost REPEAT:6
> STACK:
> ('calloc', 'vg_replace_malloc.c:711')
> ('cdbdisp_returnResults', 'cdbdisp.c:356')
> ('dispatch_statement', 'dispatcher.c:1490')
> ('dispatch_statement_node', 'dispatcher.c:1547')
> ('PerformSharedStorageOpTasks', 'cdbsharedstorageop.c:161')
> ('CreateAppendOnlyParquetSegFileForRelationOnMaster', 
> 'execMain.c:2664')
> ('CreateAppendOnlyParquetSegFileOnMaster', 'execMain.c:2469')
> ('InitializeResultRelations', 'execMain.c:1774')
> ('InitPlan', 'execMain.c:1959')
> ('ExecutorStart', 'execMain.c:697')
> ('ProcessQuery', 'pquery.c:316')
> ('PortalRunMulti', 'pquery.c:1969')
> ('PortalRun', 'pquery.c:1514')
> ('exec_simple_query', 'postgres.c:1745')
> SIZE:16 TYPE:direct_lost REPEAT:2
> STACK:
> ('malloc', 'vg_replace_malloc.c:299')
> ('pqParseInput3', 'fe-protocol3.c:596')
> ('parseInput', 'fe-exec.c:1812')
> ('PQisBusy', 'fe-exec.c:1827')
> ('executormgr_consume', 'executormgr.c:481')
> ('dispmgt_thread_func_run', 'dispatcher_mgt.c:387')
> ('workermgr_thread_func', 'workermgr.c:191')
> ('start_thread', '/lib64/libpthread-2.12.so')
> ('clone', '/lib64/libc-2.12.so')



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-574) Memory Leak in dispatcher

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15206000#comment-15206000
 ] 

ASF GitHub Bot commented on HAWQ-574:
-

GitHub user ictmalili opened a pull request:

https://github.com/apache/incubator-hawq/pull/483

HAWQ-574. Fix memory leak in dispatcher



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ictmalili/incubator-hawq HAWQ-574

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/483.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #483


commit 3bcd2a1d2e045208e85331792864c7b95c456fbf
Author: Lili Ma 
Date:   2016-03-22T08:37:25Z

HAWQ-574. Fix memory leak in dispatcher




> Memory Leak in dispatcher
> -
>
> Key: HAWQ-574
> URL: https://issues.apache.org/jira/browse/HAWQ-574
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Dispatcher
>Reporter: Lili Ma
>Assignee: Lili Ma
>
> SIZE:2,352 TYPE:direct_lost REPEAT:6
> STACK:
> ('calloc', 'vg_replace_malloc.c:711')
> ('cdbdisp_returnResults', 'cdbdisp.c:356')
> ('dispatch_statement', 'dispatcher.c:1490')
> ('dispatch_statement_node', 'dispatcher.c:1547')
> ('PerformSharedStorageOpTasks', 'cdbsharedstorageop.c:161')
> ('CreateAppendOnlyParquetSegFileForRelationOnMaster', 
> 'execMain.c:2664')
> ('CreateAppendOnlyParquetSegFileOnMaster', 'execMain.c:2469')
> ('InitializeResultRelations', 'execMain.c:1774')
> ('InitPlan', 'execMain.c:1959')
> ('ExecutorStart', 'execMain.c:697')
> ('ProcessQuery', 'pquery.c:316')
> ('PortalRunMulti', 'pquery.c:1969')
> ('PortalRun', 'pquery.c:1514')
> ('exec_simple_query', 'postgres.c:1745')
> SIZE:16 TYPE:direct_lost REPEAT:2
> STACK:
> ('malloc', 'vg_replace_malloc.c:299')
> ('pqParseInput3', 'fe-protocol3.c:596')
> ('parseInput', 'fe-exec.c:1812')
> ('PQisBusy', 'fe-exec.c:1827')
> ('executormgr_consume', 'executormgr.c:481')
> ('dispmgt_thread_func_run', 'dispatcher_mgt.c:387')
> ('workermgr_thread_func', 'workermgr.c:191')
> ('start_thread', '/lib64/libpthread-2.12.so')
> ('clone', '/lib64/libc-2.12.so')



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-574) Memory Leak in dispatcher

[Lili Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lili Ma updated HAWQ-574:
-
Summary: Memory Leak in dispatcher  (was: Memory Leak in dispatcher.)

> Memory Leak in dispatcher
> -
>
> Key: HAWQ-574
> URL: https://issues.apache.org/jira/browse/HAWQ-574
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Dispatcher
>Reporter: Lili Ma
>Assignee: Lili Ma
>
> SIZE:2,352 TYPE:direct_lost REPEAT:6
> STACK:
> ('calloc', 'vg_replace_malloc.c:711')
> ('cdbdisp_returnResults', 'cdbdisp.c:356')
> ('dispatch_statement', 'dispatcher.c:1490')
> ('dispatch_statement_node', 'dispatcher.c:1547')
> ('PerformSharedStorageOpTasks', 'cdbsharedstorageop.c:161')
> ('CreateAppendOnlyParquetSegFileForRelationOnMaster', 
> 'execMain.c:2664')
> ('CreateAppendOnlyParquetSegFileOnMaster', 'execMain.c:2469')
> ('InitializeResultRelations', 'execMain.c:1774')
> ('InitPlan', 'execMain.c:1959')
> ('ExecutorStart', 'execMain.c:697')
> ('ProcessQuery', 'pquery.c:316')
> ('PortalRunMulti', 'pquery.c:1969')
> ('PortalRun', 'pquery.c:1514')
> ('exec_simple_query', 'postgres.c:1745')
> SIZE:16 TYPE:direct_lost REPEAT:2
> STACK:
> ('malloc', 'vg_replace_malloc.c:299')
> ('pqParseInput3', 'fe-protocol3.c:596')
> ('parseInput', 'fe-exec.c:1812')
> ('PQisBusy', 'fe-exec.c:1827')
> ('executormgr_consume', 'executormgr.c:481')
> ('dispmgt_thread_func_run', 'dispatcher_mgt.c:387')
> ('workermgr_thread_func', 'workermgr.c:191')
> ('start_thread', '/lib64/libpthread-2.12.so')
> ('clone', '/lib64/libc-2.12.so')



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HAWQ-574) Memory Leak in dispatcher.

[Lili Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lili Ma reassigned HAWQ-574:


Assignee: Lili Ma  (was: Lei Chang)

> Memory Leak in dispatcher.
> --
>
> Key: HAWQ-574
> URL: https://issues.apache.org/jira/browse/HAWQ-574
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Dispatcher
>Reporter: Lili Ma
>Assignee: Lili Ma
>
> SIZE:2,352 TYPE:direct_lost REPEAT:6
> STACK:
> ('calloc', 'vg_replace_malloc.c:711')
> ('cdbdisp_returnResults', 'cdbdisp.c:356')
> ('dispatch_statement', 'dispatcher.c:1490')
> ('dispatch_statement_node', 'dispatcher.c:1547')
> ('PerformSharedStorageOpTasks', 'cdbsharedstorageop.c:161')
> ('CreateAppendOnlyParquetSegFileForRelationOnMaster', 
> 'execMain.c:2664')
> ('CreateAppendOnlyParquetSegFileOnMaster', 'execMain.c:2469')
> ('InitializeResultRelations', 'execMain.c:1774')
> ('InitPlan', 'execMain.c:1959')
> ('ExecutorStart', 'execMain.c:697')
> ('ProcessQuery', 'pquery.c:316')
> ('PortalRunMulti', 'pquery.c:1969')
> ('PortalRun', 'pquery.c:1514')
> ('exec_simple_query', 'postgres.c:1745')
> SIZE:16 TYPE:direct_lost REPEAT:2
> STACK:
> ('malloc', 'vg_replace_malloc.c:299')
> ('pqParseInput3', 'fe-protocol3.c:596')
> ('parseInput', 'fe-exec.c:1812')
> ('PQisBusy', 'fe-exec.c:1827')
> ('executormgr_consume', 'executormgr.c:481')
> ('dispmgt_thread_func_run', 'dispatcher_mgt.c:387')
> ('workermgr_thread_func', 'workermgr.c:191')
> ('start_thread', '/lib64/libpthread-2.12.so')
> ('clone', '/lib64/libc-2.12.so')



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-574) Memory Leak in dispatcher.

[Lili Ma (JIRA)]

Lili Ma created HAWQ-574:


 Summary: Memory Leak in dispatcher.
 Key: HAWQ-574
 URL: https://issues.apache.org/jira/browse/HAWQ-574
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Dispatcher
Reporter: Lili Ma
Assignee: Lei Chang


SIZE:2,352 TYPE:direct_lost REPEAT:6
STACK:
('calloc', 'vg_replace_malloc.c:711')
('cdbdisp_returnResults', 'cdbdisp.c:356')
('dispatch_statement', 'dispatcher.c:1490')
('dispatch_statement_node', 'dispatcher.c:1547')
('PerformSharedStorageOpTasks', 'cdbsharedstorageop.c:161')
('CreateAppendOnlyParquetSegFileForRelationOnMaster', 'execMain.c:2664')
('CreateAppendOnlyParquetSegFileOnMaster', 'execMain.c:2469')
('InitializeResultRelations', 'execMain.c:1774')
('InitPlan', 'execMain.c:1959')
('ExecutorStart', 'execMain.c:697')
('ProcessQuery', 'pquery.c:316')
('PortalRunMulti', 'pquery.c:1969')
('PortalRun', 'pquery.c:1514')
('exec_simple_query', 'postgres.c:1745')




SIZE:16 TYPE:direct_lost REPEAT:2
STACK:
('malloc', 'vg_replace_malloc.c:299')
('pqParseInput3', 'fe-protocol3.c:596')
('parseInput', 'fe-exec.c:1812')
('PQisBusy', 'fe-exec.c:1827')
('executormgr_consume', 'executormgr.c:481')
('dispmgt_thread_func_run', 'dispatcher_mgt.c:387')
('workermgr_thread_func', 'workermgr.c:191')
('start_thread', '/lib64/libpthread-2.12.so')
('clone', '/lib64/libc-2.12.so')



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-182) Collect advanced statistics for HBase plugin

[Grace Kaboh (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15205984#comment-15205984
 ] 

Grace Kaboh commented on HAWQ-182:
--

Thanks everyone for your assistance. Time is really short to application 
dateline now however, I spent last night going through the suggested material 
and the Extension framework, I was able to write a draft proposal which I have 
submitted on the summerofcode site. please leave some comments so I can make 
any necessary corrections before submitting the final proposal. Wish me good 
luck please.

> Collect advanced statistics for HBase plugin
> 
>
> Key: HAWQ-182
> URL: https://issues.apache.org/jira/browse/HAWQ-182
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: PXF
>Reporter: Noa Horn
>Assignee: Goden Yao
>  Labels: gsoc2016
> Fix For: backlog
>
>
> Implement getFragmentsStats in HBase's fragmenter (HBaseDataFragmenter).
> As a result when running ANALYZE on PXF tables with HBase profile, advanced 
> statistics will be collected for that table.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-573) Resource leak when cancel copy

[Dong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dong Li updated HAWQ-573:
-
Description: 
Using select pg_cancel_backend() to cancel a "copy test5_pt2 (c1, c2, c3, c4, 
c5, c6) FROM stdin;"
query make the resource leak.
You can see the log, when cancel the query, it didn't return resource to 
resource manager.
p25926 is the QD process. The qd process didn't think it has acquired resource, 
but the RM has allocated resource.
When qd is running in the function processAllCommFileDescs (called by 
acquireResourceFromRM,  which is a RPC to allocate resource from QD to RM), it 
received a "query cancel pending'" interrupt, and it handles the interrupt and 
error out.
{code}
2016-03-16 11:55:13.576705 
CST,"intern","resourceleak",p25926,th-1593601580,"[local]",,2016-03-16 11:55:12 
CST,6494,con2476,cmd3,seg-1,,,x6494,sx1,"LOG","0","AsyncComm framework 
receives message 2310 from FD 12",  ,"copy test5_pt2 (c1, c2, c3, c4, 
c5, c6) FROM stdin;",0,,"rmcomm_Message.c",100,
13457 2016-03-16 11:55:13.576779 
CST,"intern","resourceleak",p25926,th-1593601580,"[local]",,2016-03-16 11:55:12 
CST,6494,con2476,cmd3,seg-1,,,x6494,sx1,"LOG","0","ConnID 427. 
Registered in HAWQ resource manager (By OID)",  ,"copy test5_pt2 (c1, 
c2, c3, c4, c5, c6) FROM stdin;",0,,"rmcomm_QD2RM.c",601,
13458 2016-03-16 11:55:13.576860 
CST,"intern","resourceleak",p25926,th-1593601580,"[local]",,2016-03-16 11:55:12 
CST,6494,con2476,cmd3,seg-1,,,x6494,sx1,"LOG","0","ConnID: 427. Acquire 
resource request for index 0. Max vse  g size 6 Min vseg size 6 Estimated 
slice size 1 estimated IO bytes size 1 Preferred node count 0.",,"copy 
test5_pt2 (c1, c2, c3, c4, c5, c6) FROM stdin;",0,,"rmcomm_QD2RM.c",693,
13459 2016-03-16 11:55:13.577266 
CST,,,p92142,th-15936015800,con18,,seg-1,"LOG","0","AsyncComm 
framework receives message 259 from FD 5",,,0,,"rmcomm_Message.c",100,
13460 2016-03-16 11:55:13.577311 
CST,,,p92142,th-15936015800,con18,,seg-1,"LOG","0","ConnID 427. 
Expect query resource for session 2476",,,0,,"resqueuemanager.c",2225,
13461 2016-03-16 11:55:13.577347 
CST,,,p92142,th-15936015800,con18,,seg-1,"LOG","0","ConnID 427. 
Expect query resource (256 MB, 0.062500 CORE) x 256 (MIN 6) after checking 
queue capacity.",,,0,,"resqueuemanager  .c",3760,
13462 2016-03-16 11:55:13.577391 
CST,,,p92142,th-15936015800,con18,,seg-1,"LOG","0","ConnID 427. 
Expect query resource (256 MB, 0.062500 CORE) x 6 (MIN 6) after checking query 
expectation 6 (MIN 6).",,,0,,"resq  ueuemanager.c",3790,
13463 2016-03-16 11:55:13.577440 
CST,,,p92142,th-15936015800,con18,,seg-1,"LOG","0","ConnID 427. 
Expect query resource (256 MB, 0.062500 CORE) x 6 ( MIN 6 ) resource after 
adjusting based on queue NVSEG limits.",,,  0,,"resqueuemanager.c",2247,
13464 2016-03-16 11:55:13.577490 
CST,,,p92142,th-15936015800,con18,,seg-1,"LOG","0","Latency of 
getting resource allocated is 193us",,,0,,"resqueuemanager.c",4745,
13465 2016-03-16 11:55:13.588859 
CST,"intern","resourceleak",p25958,th-1593601580,"[local]",,2016-03-16 11:55:13 
CST,0,con2492,,seg-1,"LOG","0","getLocalTmpDirFromMasterConfig 
session_id:2492 tmpdir:/tmp",,,0,,"pos  tinit.c",461,
13466 2016-03-16 11:55:13.628191 
CST,"intern","resourceleak",p25960,th-1593601580,"[local]",,2016-03-16 11:55:13 
CST,0,con2493,,seg-1,"LOG","0","getLocalTmpDirFromMasterConfig 
session_id:2493 tmpdir:/tmp",,,0,,"pos  tinit.c",461,
13467 2016-03-16 11:55:13.671378 
CST,"intern","resourceleak",p25963,th-1593601580,"[local]",,2016-03-16 11:55:13 
CST,0,con2494,,seg-1,"LOG","0","getLocalTmpDirFromMasterConfig 
session_id:2494 tmpdir:/tmp",,,0,,"pos  tinit.c",461,
13468 2016-03-16 11:55:13.675873 
CST,"intern","resourceleak",p25926,th-1593601580,"[local]",,2016-03-16 11:55:12 
CST,6494,con2476,cmd3,seg-1,,,x6494,sx1,"LOG","0","Process interrupt 
for 'query cancel pending'.",,"copy   test5_pt2 (c1, c2, c3, c4, c5, 
c6) FROM stdin;",0,,"postgres.c",3517,
13469 2016-03-16 11:55:13.675949 
CST,"intern","resourceleak",p25926,th-1593601580,"[local]",,2016-03-16 11:55:12 
CST,6494,con2476,cmd3,seg-1,,,x6494,sx1,"ERROR","57014","canceling 
statement due to user request",,,0,,"postg  res.c",3534,
13470 2016-03-16 11:55:13.676389 
CST,"intern","resourceleak",p25926,th-1593601580,"[local]",,2016-03-16 11:55:12 
CST,0,con2476,cmd3,seg-1,"LOG","08006","could not send data to client: 
Broken pipe",,,0,,"pqcomm.c",1292,
13471 2016-03-16 11:55:13.676436 
CST,"intern","resourceleak",p25926,th-1593601580,"[local]",,2016-03-16 11:55:12 
CST,0,con2476,cmd3,seg-1,"FATAL","08006","connection to client 
lost",,,0,,"postgres.c",3512,
13472 2016-03-16 

[jira] [Updated] (HAWQ-573) Resource leak when cancel copy

[Dong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dong Li updated HAWQ-573:
-
Description: 
Using select pg_cancel_backend() to cancel a "copy test5_pt2 (c1, c2, c3, c4, 
c5, c6) FROM stdin;"
query make the resource leak.
You can see the log, when cancel the query, it didn't return resource to 
resource manager.
p25926 is the QD process. The qd process didn't think it has acquired resource, 
but the RM has allocated resource.
When qd is running in the function processAllCommFileDescs (called by 
acquireResourceFromRM,  which is a RPC to allocate resource from QD to RM), it 
received a "query cancel pending'" interrupt, and it handles the interrupt and 
error out.


  was:
Using select pg_cancel_backend() to cancel a "copy test5_pt2 (c1, c2, c3, c4, 
c5, c6) FROM stdin;"
query make the resource leak.
You can see the log, when cancel the query, it didn't return resource to 
resource manager.
p25926 is the QD process. The qd process didn't think it has acquired resource, 
but the RM has allocated resource.
When qd is running in the function processAllCommFileDescs (called by 
acquireResourceFromRM,  which is a RPC to allocate resource from QD to RM), it 
received a "query cancel pending'" interrupt, and it handles the interrupt and 
error out.


> Resource leak when cancel copy
> --
>
> Key: HAWQ-573
> URL: https://issues.apache.org/jira/browse/HAWQ-573
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Resource Manager
>Reporter: Dong Li
>Assignee: Lei Chang
>
> Using select pg_cancel_backend() to cancel a "copy test5_pt2 (c1, c2, c3, c4, 
> c5, c6) FROM stdin;"
> query make the resource leak.
> You can see the log, when cancel the query, it didn't return resource to 
> resource manager.
> p25926 is the QD process. The qd process didn't think it has acquired 
> resource, but the RM has allocated resource.
> When qd is running in the function processAllCommFileDescs (called by 
> acquireResourceFromRM,  which is a RPC to allocate resource from QD to RM), 
> it received a "query cancel pending'" interrupt, and it handles the interrupt 
> and error out.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-573) Resource leak when cancel copy

[Dong Li (JIRA)]

Dong Li created HAWQ-573:


 Summary: Resource leak when cancel copy
 Key: HAWQ-573
 URL: https://issues.apache.org/jira/browse/HAWQ-573
 Project: Apache HAWQ
  Issue Type: Bug
  Components: Resource Manager
Reporter: Dong Li
Assignee: Lei Chang


Using select pg_cancel_backend() to cancel a "copy test5_pt2 (c1, c2, c3, c4, 
c5, c6) FROM stdin;"
query make the resource leak.
You can see the log, when cancel the query, it didn't return resource to 
resource manager.
p25926 is the QD process. The qd process didn't think it has acquired resource, 
but the RM has allocated resource.
When qd is running in the function processAllCommFileDescs (called by 
acquireResourceFromRM,  which is a RPC to allocate resource from QD to RM), it 
received a "query cancel pending'" interrupt, and it handles the interrupt and 
error out.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-572) Improve code coverage for dispatcher: fail_qe_after_connection & fail_qe_when_do_query & fail_qe_when_begin_parquet_scan

[Chunling Wang (JIRA)]

Chunling Wang created HAWQ-572:
--

 Summary: Improve code coverage for dispatcher: 
fail_qe_after_connection & fail_qe_when_do_query & 
fail_qe_when_begin_parquet_scan
 Key: HAWQ-572
 URL: https://issues.apache.org/jira/browse/HAWQ-572
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: Dispatcher
Reporter: Chunling Wang
Assignee: Lei Chang


Add those fault injections:
1. fail_qe_after_connection 
2. fail_qe_when_do_query 
3. fail_qe_when_begin_parquet_scan
And add test cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-492) PXF Query Push-Down Feature enhancement to enable further push down to Datasource

[Michael Andre Pearce (IG) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15205889#comment-15205889
 ] 

Michael Andre Pearce (IG) commented on HAWQ-492:


Thanks Goden,

Unfortunately this doesn't resolve my issue.

It covers only being able to push down the predicate WHERE logic to the 
external system. And I've been able to achieve this with being able to 
re-consititue the WHERE statement for the down stream call to the external 
table from the predicates passed of GT, LT, GE, LE, etc.

In the example it seems a full row is still returned. And indeed this is 
wanting to avoid. 

Say in HAWQ you submit SELECT a, b FROM external_mysql_table WHERE z < 3 AND x 
> 6

I can construct now SELECT * FROM mysql_table WHERE z < 3 AND x > 6 and push 
this down into the downstream db, with pushdown on and filters applied. 

Alas as you notice i am still having to return the full row, even so the HAWQ 
select only needs a and b, as such should look to only pass these columns back 
to HAWQ for effieciencies. I want to be able to construct the SELECT a, b part 
also. 

>From the documents and sample do not see how to achieve knowing i need only 
>SELECT columns a and b, only seems to cover how to create the predicate WHERE 
>clause.

Thanks.




> PXF Query Push-Down Feature enhancement to enable further push down to 
> Datasource
> -
>
> Key: HAWQ-492
> URL: https://issues.apache.org/jira/browse/HAWQ-492
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: PXF
>Reporter: Michael Andre Pearce (IG)
>Assignee: Lei Chang
>
> PXF currently supports push down of filters to the PXF plugin. So that PXF 
> itself only needs to return the fields and rows that match the filters.
> Unfortunately it seems (and it could be simply miss-understanding) is done in 
> a way where by the PXF plugin must retrieve the complete row/entry from the 
> source then the filters are applied. With Query Filter Push-Down feature.
> With a JDBC PXF (https://github.com/kojec/pxf-field/tree/master/jdbc-pxf-ext) 
> this seems to be the case where by the SELECT * FROM table is done. 
> It would be much more efficient if the filter and fields are push down in a 
> way so that the plugin can if in its use case can actually delegate that 
> further down into datasource it is using, and retrieving and applying the 
> filters needed to reduce the data it needs to be pulled back.
> As such the PXF accessor would be able to construct SELECT fieldA fieldB FROM 
> table WHERE fieldA > 6
> It maybe that this is indeed possible, but looking at 
> https://github.com/kojec/pxf-field/tree/master/jdbc-pxf-ext and the API's and 
> documents currently available we cannot see a way to improve this jdbc pxf 
> plugin to achieve this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HAWQ-571) Fix Misspellings in libyarn

[Lin Wen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lin Wen resolved HAWQ-571.
--
Resolution: Fixed

> Fix Misspellings in libyarn
> ---
>
> Key: HAWQ-571
> URL: https://issues.apache.org/jira/browse/HAWQ-571
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: libyarn
>Reporter: Lin Wen
>Assignee: Lin Wen
>
> in exception.c
> YarnBadBoolFoumat should be YarnBadBoolFormat,
> YarnBadConfigFoumat  should be YarnBadConfigFormat
> YarnBadNumFoumat should be YarnBadNumFormat
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-405) Issue with handling pg_temp schema

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15205883#comment-15205883
 ] 

ASF GitHub Bot commented on HAWQ-405:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-hawq/pull/338


> Issue with handling pg_temp schema
> --
>
> Key: HAWQ-405
> URL: https://issues.apache.org/jira/browse/HAWQ-405
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Hcatalog, PXF
>Affects Versions: 2.0.0-beta-incubating
>Reporter: Shivram Mani
>Assignee: Shivram Mani
> Fix For: 2.0.0
>
>
> We aren't handling the temp namespace the right way when LookupNamespaceId is 
> invoked.
> This change was done as part of handling hcatalog where we specifically 
> handle the pg_temp schema. 
> Steps to reproduce error
> Reproduction 1: invalid reference to FROM-clause entry for table
> {code}
> create table pg_temp.test(row integer, count integer);
> insert into pg_temp.test values (1, 10), (2, 20), (3, 30);
> select avg(pg_temp.test.count) from pg_temp.test;
> ERROR:  invalid reference to FROM-clause entry for table "test"
> LINE 1: select avg(pg_temp.test.count) from pg_temp.test;
>^
> {code}
> Reproduction 2: missing FROM-clause entry for table
> {code}
> select case when pg_temp.test.count = 30 then 30 when pg_temp.test.count = 20 
> then 20 else 10 end;
> ERROR:  missing FROM-clause entry for table "test"
> LINE 1: select case when pg_temp.test.count = 30 then 30 when pg_tem...
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-423) Fix gradle and download plugin version

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15205884#comment-15205884
 ] 

ASF GitHub Bot commented on HAWQ-423:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-hawq/pull/356


> Fix gradle and download plugin version
> --
>
> Key: HAWQ-423
> URL: https://issues.apache.org/jira/browse/HAWQ-423
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: PXF
>Reporter: Shivram Mani
>Assignee: Shivram Mani
>
> We use the plugin de.undercouch.download to download tomcat tarbal 
> .de.undercouch:gradle-download-task version 1.2 was compatible only till 
> gradle 2.5. Gradle 2.6 onwards is incompatible 
> https://github.com/michel-kraemer/gradle-download-task/issues/29
> Upgrade to "de.undercouch:gradle-download-task:2.1.0" works against all newer 
> versions including gradle 2.11



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-571) Fix Misspellings in libyarn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15205885#comment-15205885
 ] 

ASF GitHub Bot commented on HAWQ-571:
-

Github user linwen closed the pull request at:

https://github.com/apache/incubator-hawq/pull/481


> Fix Misspellings in libyarn
> ---
>
> Key: HAWQ-571
> URL: https://issues.apache.org/jira/browse/HAWQ-571
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: libyarn
>Reporter: Lin Wen
>Assignee: Lin Wen
>
> in exception.c
> YarnBadBoolFoumat should be YarnBadBoolFormat,
> YarnBadConfigFoumat  should be YarnBadConfigFormat
> YarnBadNumFoumat should be YarnBadNumFormat
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-571) Fix Misspellings in libyarn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15205859#comment-15205859
 ] 

ASF GitHub Bot commented on HAWQ-571:
-

Github user jiny2 commented on the pull request:

https://github.com/apache/incubator-hawq/pull/481#issuecomment-199650836
  
+1 LGTM


> Fix Misspellings in libyarn
> ---
>
> Key: HAWQ-571
> URL: https://issues.apache.org/jira/browse/HAWQ-571
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: libyarn
>Reporter: Lin Wen
>Assignee: Lin Wen
>
> in exception.c
> YarnBadBoolFoumat should be YarnBadBoolFormat,
> YarnBadConfigFoumat  should be YarnBadConfigFormat
> YarnBadNumFoumat should be YarnBadNumFormat
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)