[GitHub] incubator-hawq issue #1353: HAWQ-1605. Support INSERT in PXF JDBC plugin

[kongyew]

Github user kongyew commented on the issue:

https://github.com/apache/incubator-hawq/pull/1353
  
 **This JDBC profile is not officially supported** by Greenplum since it is 
not tested at all.  Use it at your own risks.  The interface maybe changed in 
the future since the current profile is not securely storing credentials.


---

[GitHub] incubator-hawq issue #1353: HAWQ-1605. Support INSERT in PXF JDBC plugin

[deem0n]

Github user deem0n commented on the issue:

https://github.com/apache/incubator-hawq/pull/1353
  
 Yep, our customers have plenty different data sources, mostly JDBC 
compliant. This feature makes GPDB a very flexible and friendly player in 
corporate IT landscape.


---

[GitHub] incubator-hawq issue #1353: HAWQ-1605. Support INSERT in PXF JDBC plugin

[mebelousov]

Github user mebelousov commented on the issue:

https://github.com/apache/incubator-hawq/pull/1353
  
This is great feature.
We really need to upload data from Greenplum to other databases.


---

[jira] [Assigned] (HAWQ-1549) Re-syncing standby fails even when stop mode is fast

[Radar Lei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei reassigned HAWQ-1549:
---

Assignee: Shubham Sharma  (was: Radar Lei)

>  Re-syncing standby fails even when stop mode is fast
> -
>
> Key: HAWQ-1549
> URL: https://issues.apache.org/jira/browse/HAWQ-1549
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Command Line Tools, Standby master
>Reporter: Shubham Sharma
>Assignee: Shubham Sharma
>Priority: Major
> Fix For: 2.3.0.0-incubating
>
>
> Recently observed a behaviour while re-syncing standby from hawq command line.
> Here are the reproduction steps -
> 1 - Open a client connection to hawq using psql
> 2 - From a different terminal run command - hawq init standby -n -v -M fast
> 3 - Standby resync fails with error
> {code}
> 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-There are other 
> connections to this instance, shutdown mode smart aborted
> 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-Either remove 
> connections, or use 'hawq stop master -M fast' or 'hawq stop master -M 
> immediate'
> 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-See hawq stop 
> --help for all options
> 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[ERROR]:-Active connections. 
> Aborting shutdown...
> 20171113:03:49:21:158143 hawq_init:hdp3:gpadmin-[ERROR]:-Stop hawq cluster 
> failed, exit
> {code}
> 4 - When -M (stop mode) is passed it should terminate existing client 
> connections. 
> The source of this issue appears to be tools/bin/hawq_ctl method 
> _resync_standby. When this is called the command formation does not include 
> stop_mode options as passed to the arguments.
> {code}
>  def _resync_standby(self):
> logger.info("Re-sync standby")
> cmd = "%s; hawq stop master -a;" % source_hawq_env
> check_return_code(local_ssh(cmd, logger), logger, "Stop hawq cluster 
> failed, exit")
> ..
> ..
> {code}
> I can start this and submit a PR when changes are done.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (HAWQ-1548) Ambiguous message while logging hawq utilization

[Radar Lei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei closed HAWQ-1548.
---
   Resolution: Not A Problem
Fix Version/s: backlog

> Ambiguous message while logging hawq utilization
> 
>
> Key: HAWQ-1548
> URL: https://issues.apache.org/jira/browse/HAWQ-1548
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: libyarn
>Reporter: Shubham Sharma
>Assignee: Lin Wen
>Priority: Major
> Fix For: backlog
>
>
> While YARN mode is enabled, resource broker logs two things - 
> - YARN cluster total resource 
> - HAWQ's total resource per node.
> Following messages are logged 
> {code}
> 2017-11-11 23:21:40.944904 
> UTC,,,p549330,th9000778560,con4,,seg-1,"LOG","0","Resource 
> manager YARN resource broker counted YARN cluster having total resource 
> (1376256 MB, 168.00 CORE).",,,0,,"resourcebroker_LIBYARN.c",776,
> 2017-11-11 23:21:40.944921 
> UTC,,,p549330,th9000778560,con4,,seg-1,"LOG","0","Resource 
> manager YARN resource broker counted HAWQ cluster now having (98304 MB, 
> 12.00 CORE) in a YARN cluster of total resource (1376256 MB, 168.00 
> CORE).",,,0,,"resourcebroker_LIBYARN.c",785,
> {code}
> The second message shown above is ambiguous, After reading the sentence below 
> it looks like that complete Hawq cluster in whole has only 98304 MB and 12 
> cores. However according to the configuration it should be 98304 MB and 12 
> cores per segment server.
> {code}
> Resource manager YARN resource broker counted HAWQ cluster now having (98304 
> MB, 12.00 CORE) in a YARN cluster of total resource (1376256 MB, 
> 168.00 CORE).
> {code}
> Either the wrong variables are printed or we can correct the message to 
> represent that the resources logged are per node. As this can confuse the 
> user into thinking that hawq cluster does not have enough resources.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HAWQ-1515) how to build and complie hawq based on suse11

[Radar Lei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei resolved HAWQ-1515.
-
   Resolution: Fixed
Fix Version/s: backlog

> how to build and complie hawq based on suse11
> -
>
> Key: HAWQ-1515
> URL: https://issues.apache.org/jira/browse/HAWQ-1515
> Project: Apache HAWQ
>  Issue Type: Task
>  Components: Build
>Reporter: FengHuang
>Assignee: Radar Lei
>Priority: Major
> Fix For: backlog
>
>
> three are a little zypper rep for all kinds of dependencies for build of hawq 
> on suse11. can you recommend some available and comprehensive zypper rep?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HAWQ-1542) PXF Demo profile should support write use case.

[Radar Lei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei resolved HAWQ-1542.
-
   Resolution: Fixed
Fix Version/s: 2.3.0.0-incubating

> PXF Demo profile should support write use case.
> ---
>
> Key: HAWQ-1542
> URL: https://issues.apache.org/jira/browse/HAWQ-1542
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: PXF
>Reporter: Alexander Denissov
>Assignee: Alexander Denissov
>Priority: Major
> Fix For: 2.3.0.0-incubating
>
>
> The Demo PXF accessors / resolvers should support a use case for defining 
> writable external table that saves data to a file on a local file system.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HAWQ-1549) Re-syncing standby fails even when stop mode is fast

[Radar Lei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei resolved HAWQ-1549.
-
   Resolution: Fixed
Fix Version/s: 2.3.0.0-incubating

>  Re-syncing standby fails even when stop mode is fast
> -
>
> Key: HAWQ-1549
> URL: https://issues.apache.org/jira/browse/HAWQ-1549
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Command Line Tools, Standby master
>Reporter: Shubham Sharma
>Assignee: Shubham Sharma
>Priority: Major
> Fix For: 2.3.0.0-incubating
>
>
> Recently observed a behaviour while re-syncing standby from hawq command line.
> Here are the reproduction steps -
> 1 - Open a client connection to hawq using psql
> 2 - From a different terminal run command - hawq init standby -n -v -M fast
> 3 - Standby resync fails with error
> {code}
> 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-There are other 
> connections to this instance, shutdown mode smart aborted
> 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-Either remove 
> connections, or use 'hawq stop master -M fast' or 'hawq stop master -M 
> immediate'
> 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-See hawq stop 
> --help for all options
> 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[ERROR]:-Active connections. 
> Aborting shutdown...
> 20171113:03:49:21:158143 hawq_init:hdp3:gpadmin-[ERROR]:-Stop hawq cluster 
> failed, exit
> {code}
> 4 - When -M (stop mode) is passed it should terminate existing client 
> connections. 
> The source of this issue appears to be tools/bin/hawq_ctl method 
> _resync_standby. When this is called the command formation does not include 
> stop_mode options as passed to the arguments.
> {code}
>  def _resync_standby(self):
> logger.info("Re-sync standby")
> cmd = "%s; hawq stop master -a;" % source_hawq_env
> check_return_code(local_ssh(cmd, logger), logger, "Stop hawq cluster 
> failed, exit")
> ..
> ..
> {code}
> I can start this and submit a PR when changes are done.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HAWQ-1605) Support INSERT in PXF JDBC plugin

[Ivan Leskin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430237#comment-16430237
 ] 

Ivan Leskin commented on HAWQ-1605:
---

All the features mentioned have already been implemented. The code is available 
at github: [https://github.com/apache/incubator-hawq/pull/1353].

> Support INSERT in PXF JDBC plugin
> -
>
> Key: HAWQ-1605
> URL: https://issues.apache.org/jira/browse/HAWQ-1605
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: PXF
>Reporter: Ivan Leskin
>Assignee: Ed Espino
>Priority: Major
>
> Add support of INSERT queries in PXF JDBC plugin:
>  * Implement `WriteAccessor` and `WriteResolver` interfaces. Both are 
> implemented in the same classes as `ReadAccessor` and `ReadResolver`;
>  * Support query batching in Accessor. The size of a batch is defined by user 
> and may be "infinite";
>  * In Accessor, use `java.sql.PreparedStatement` and built-in functions in it 
> to process queries;
>  * In `setFields()` method of Resolver, perform type conversions of the data 
> tuples received from PXF.
> Optimize and refactor the code in PXF JDBC plugin, make some fixes:
>  * Make functions for building WHERE statements static where possible to 
> reduce the number of InputData checks;
>  * Organize imports;
>  * Check the codestyle;
>  * Fix the handling of TIMESTAMP values when performing SELECT requests.
> Improve documentation:
>  * Correct or rewrite Javadoc strings for plugin functions;
>  * Rewrite README.md.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] incubator-hawq pull request #1353: HAWQ-1605. Support INSERT in PXF JDBC plu...

[leskin-in]

GitHub user leskin-in opened a pull request:

https://github.com/apache/incubator-hawq/pull/1353

HAWQ-1605. Support INSERT in PXF JDBC plugin

Add support of INSERT queries in PXF JDBC plugin:
* Implement interfaces `WriteAccessor` (by `JdbcAccessor` class) and 
`WriteResolver` (by `JdbcResolver` class);
* Support query batching in `JdbcAccessor` when processing INSERT query. 
The size of a batch is defined by user and may be "infinite";
* In `JdbcAccessor`, use `java.sql.PreparedStatement` and JDBC standard 
functions to process queries (and to support batching);
* In `setFields()` method of `JdbcResolver`, perform type conversions of 
the data tuples received from PXF;
* Support both transactional and non-transactional databases when 
performing INSERT queries.

Refactor the code in PXF JDBC plugin, make some microoptimizations and 
fixes:
* Fix the handling of TIMESTAMP when performing SELECT requests;
* Make functions for building WHERE statements static where possible to 
reduce the number of `InputData` checks. This is proposed by @hornn;
* Prettify some SQL statements generated by the PXF JDBC plugin and change 
the tests respectively;
* Organize imports;
* Expand some intricate `if ... else` constructions.

Improve documentation:
* Clarify, expand or rewrite Javadoc strings;
* Rewrite README.md.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/arenadata/incubator-hawq pxf_jdbc_writeAndFix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/1353.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1353


commit ba47942574049afc72c93033e03c7a7f1cf25f13
Author: Ivan Leskin 
Date:   2018-03-05T14:19:26Z

Fix incorrect TIMESTAMP handling

commit 3eb0daa8b5aca8c6c85cd7e6ebec8ea4b57c2966
Author: Ivan Leskin 
Date:   2018-03-07T17:24:01Z

PXF JDBC plugin update

* Add support for INSERT queries:
* The INSERT queries are processed by the same classes as the SELECT 
queries;
* INSERTs are processed by the JDBC PreparedStatement;
* INSERTs support batching (by means of JDBC);

* Minor changes in WhereSQLBuilder and JdbcPartitionFragmenter:
* Removed 'WHERE 1=1';
* The same pattern of spaces around operators everywhere ('a = b', not 
'a=b');
* JdbcPartitionFragmenter.buildFragmenterSql() made static to avoid 
extra checks of InputData (proposed by @sansanichfb);

* Refactoring and some microoptimizations;

commit 98f18c54cd3a9a63c8e9b5a13fb48f4494994051
Author: Ivan Leskin 
Date:   2018-04-02T17:57:56Z

PXF JDBC refactoring

* The README.md is completely rewritten;

* Lots of changes in comments and javadoc comments;

* Code refactoring and minor changes in codestyle




---

[jira] [Resolved] (HAWQ-1572) Travis CI build failure on master. Thrift/boost incompatibility

[Radar Lei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei resolved HAWQ-1572.
-
   Resolution: Fixed
Fix Version/s: 2.3.0.0-incubating

> Travis CI build failure on master. Thrift/boost incompatibility
> ---
>
> Key: HAWQ-1572
> URL: https://issues.apache.org/jira/browse/HAWQ-1572
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Build
>Reporter: Shubham Sharma
>Assignee: Shubham Sharma
>Priority: Major
> Fix For: 2.3.0.0-incubating
>
>
> Hi,
> The travis CI build is failing for master and new commits. The CI is erroring 
> out with
> {code}
> configure: error: thrift is required
> The command “./configure” failed and exited with 1 during .
> {code}
> I was able to reproduce this issue and looking at the config.log it looks 
> like it is failing at the line below while running a conftest.cpp -
> {code}
> /usr/local/include/thrift/stdcxx.h:32:10: fatal error: 
> 'boost/tr1/functional.hpp' file not found
> {code}
> The root cause of the problem is compatibility of thrift 0.11 with boost 
> 1.65.1 . Travis recently upgraded there xcode to 9.2 and list of default 
> packages now contains boost 1.65.1 and thrift 0.11.
> Thrift uses 
> [stdcxx.h|https://github.com/apache/thrift/blob/master/lib/cpp/src/thrift/stdcxx.h]
>  which includes boost/tr1/functional.hpp library. The support for tr1 has 
> been removed in boost 1.65, see 
> [here|http://www.boost.org/users/history/version_1_65_1.html] under topic 
> “Removed Libraries”.
> Since tr1 library is no longer present in boost 1.65, this causes thrift to 
> fail and eventually ./configure fails
> Solution
> As a solution I recommend that we uninstall boost 1.65 and install boost 
> 1.60(the last compatible build with thrift).
> I am not sure if this is a problem with thrift that they are not yet 
> compatible with boost 1.65 yet or a problem with travis ci that they have 
> included two incompatible versions. Will love to hear community's thoughts on 
> it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HAWQ-1572) Travis CI build failure on master. Thrift/boost incompatibility

[Radar Lei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei reassigned HAWQ-1572:
---

Assignee: Shubham Sharma  (was: Radar Lei)

> Travis CI build failure on master. Thrift/boost incompatibility
> ---
>
> Key: HAWQ-1572
> URL: https://issues.apache.org/jira/browse/HAWQ-1572
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Build
>Reporter: Shubham Sharma
>Assignee: Shubham Sharma
>Priority: Major
>
> Hi,
> The travis CI build is failing for master and new commits. The CI is erroring 
> out with
> {code}
> configure: error: thrift is required
> The command “./configure” failed and exited with 1 during .
> {code}
> I was able to reproduce this issue and looking at the config.log it looks 
> like it is failing at the line below while running a conftest.cpp -
> {code}
> /usr/local/include/thrift/stdcxx.h:32:10: fatal error: 
> 'boost/tr1/functional.hpp' file not found
> {code}
> The root cause of the problem is compatibility of thrift 0.11 with boost 
> 1.65.1 . Travis recently upgraded there xcode to 9.2 and list of default 
> packages now contains boost 1.65.1 and thrift 0.11.
> Thrift uses 
> [stdcxx.h|https://github.com/apache/thrift/blob/master/lib/cpp/src/thrift/stdcxx.h]
>  which includes boost/tr1/functional.hpp library. The support for tr1 has 
> been removed in boost 1.65, see 
> [here|http://www.boost.org/users/history/version_1_65_1.html] under topic 
> “Removed Libraries”.
> Since tr1 library is no longer present in boost 1.65, this causes thrift to 
> fail and eventually ./configure fails
> Solution
> As a solution I recommend that we uninstall boost 1.65 and install boost 
> 1.60(the last compatible build with thrift).
> I am not sure if this is a problem with thrift that they are not yet 
> compatible with boost 1.65 yet or a problem with travis ci that they have 
> included two incompatible versions. Will love to hear community's thoughts on 
> it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HAWQ-1574) libhdfs fails silently when hdfs extended acls are in use

[Radar Lei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei updated HAWQ-1574:

Fix Version/s: backlog

> libhdfs fails silently when hdfs extended acls are in use
> -
>
> Key: HAWQ-1574
> URL: https://issues.apache.org/jira/browse/HAWQ-1574
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: libhdfs
>Reporter: Peter Parente
>Assignee: Radar Lei
>Priority: Major
> Fix For: backlog
>
>
> {code}
> # list files in a folder
> hdfs.ls('/user/p-pparente/example')
> ['/user/p-pparente/example/1',
>  '/user/p-pparente/example/2',
>  '/user/p-pparente/example/3']
> # using the standard hdfs CLI, set some extended acls
> # hdfs dfs -setfacl -m user:analytics:rwx /user/p-pparente/example/1
> # try to list files again, nothing shows!
> hdfs.ls('/user/p-pparente/example')
> []
> # remove the extended acl using the hdfs CLI
> # hdfs dfs -setfacl -x user:analytics /user/p-pparente/example/1
> # list again, and still nothing there because the extended ACLs have been set 
> at least once
> hdfs.ls('/user/p-pparente/example')
> []
> # Remove the file from the directory entirely
> # hdfs dfs -rm /user/p-pparente/example/1
> # list again, and now everything is fine once more
> hdfs.ls('/user/p-pparente/example')
> hdfs.ls('/user/p-pparente/example')
> ['/user/p-pparente/example/1', '/user/p-pparente/example/2']
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HAWQ-1578) Regression Test (Feature->Ranger)Failed because pxfwritable_import_beginscan function was not found

[Radar Lei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei resolved HAWQ-1578.
-
   Resolution: Fixed
Fix Version/s: 2.3.0.0-incubating

> Regression Test (Feature->Ranger)Failed because pxfwritable_import_beginscan 
> function was not found 
> 
>
> Key: HAWQ-1578
> URL: https://issues.apache.org/jira/browse/HAWQ-1578
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: PXF, Tests
>Reporter: WANG Weinan
>Assignee: Chiyang Wan
>Priority: Major
> Fix For: 2.3.0.0-incubating
>
> Attachments: HAWQ-1578.patch
>
>
> The TestHawqRanger failed when do test PXFHiveTest and PXFHBaseTest, the test 
> log is shown as follow:
> Note: Google Test filter = TestHawqRanger.PXFHiveTest
> [==] Running 1 test from 1 test case.
> [--] Global test environment set-up.
> [--] 1 test from TestHawqRanger
> [ RUN  ] TestHawqRanger.PXFHiveTest
> lib/sql_util.cpp:197: Failure
> Value of: is_sql_ans_diff
>   Actual: true
> Expected: false
> lib/sql_util.cpp:203: Failure
> Value of: true
>   Actual: true
> Expected: false
> [  FAILED  ] TestHawqRanger.PXFHiveTest (89777 ms)
> [--] 1 test from TestHawqRanger (89777 ms total)
> [--] Global test environment tear-down
> [==] 1 test from 1 test case ran. (89777 ms total)
> [  PASSED  ] 0 tests.
> [  FAILED  ] 1 test, listed below:
> [  FAILED  ] TestHawqRanger.PXFHiveTest
>  1 FAILED TEST
> [125/133] TestHawqRanger.PXFHiveTest returned/aborted with exit code 1 (89787 
> ms)
> [128/133] TestHawqRanger.PXFHBaseTest (87121 ms)  
>   
> Note: Google Test filter = TestHawqRanger.PXFHBaseTest
> [==] Running 1 test from 1 test case.
> [--] Global test environment set-up.
> [--] 1 test from TestHawqRanger
> [ RUN  ] TestHawqRanger.PXFHBaseTest
> lib/sql_util.cpp:197: Failure
> Value of: is_sql_ans_diff
>   Actual: true
> Expected: false
> lib/sql_util.cpp:203: Failure
> Value of: true
>   Actual: true
> Expected: false
> [  FAILED  ] TestHawqRanger.PXFHBaseTest (87098 ms)
> [--] 1 test from TestHawqRanger (87098 ms total)
> [--] Global test environment tear-down
> [==] 1 test from 1 test case ran. (87099 ms total)
> [  PASSED  ] 0 tests.
> [  FAILED  ] 1 test, listed below:
> [  FAILED  ] TestHawqRanger.PXFHBaseTest
> We can find some suspicious log in master segment log file :
> 2018-01-03 05:21:30.170970 
> UTC,"gpadmin","hawq_feature_test_db",p109703,th-290256608,"127.0.0.1","56288",2018-01-03
>  05:21:29 
> UTC,14669,con2342,cmd4,seg-1,,,x14669,sx1,"ERROR","XX000","pxfwritable_import_beginscan
>  function was not found (nodeExternalscan.c:310)",,"select * from 
> test_hbase;",0,,"nodeExternalscan.c",310,"Stack trace:
> 10x8cf31e postgres errstart (elog.c:505)
> 20x8d11bb postgres elog_finish (elog.c:1459)
> 30x69134a postgres ExecInitExternalScan (nodeExternalscan.c:215)
> 40x670b9d postgres ExecInitNode (execProcnode.c:371)
> 50x69b7d1 postgres ExecInitMotion (nodeMotion.c:1096)
> 60x670064 postgres ExecInitNode (execProcnode.c:629)
> 70x66a407 postgres ExecutorStart (execMain.c:2048)
> 80x7f8fcd postgres PortalStart (pquery.c:1308)
> 90x7f0628 postgres  (postgres.c:1795)
> 10   0x7f1cb0 postgres PostgresMain (postgres.c:4897)
> 11   0x7a40c0 postgres  (postmaster.c:5486)
> 12   0x7a6e89 postgres PostmasterMain (postmaster.c:1459)
> 13   0x4a5a59 postgres main (main.c:226)
> 14   0x7fceea8a1d1d libc.so.6 __libc_start_main (??:0)
> 15   0x4a5ad9 postgres  (??:0)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HAWQ-1583) Add vectorized executor extension and GUC

[Radar Lei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei updated HAWQ-1583:

Fix Version/s: (was: backlog)
   2.4.0.0-incubating

> Add vectorized executor extension and GUC
> -
>
> Key: HAWQ-1583
> URL: https://issues.apache.org/jira/browse/HAWQ-1583
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Query Execution
>Reporter: Hongxu Ma
>Assignee: WANG Weinan
>Priority: Major
> Fix For: 2.4.0.0-incubating
>
>
> The vectorized executor will be implemented as a extension (located at 
> contrib directory).
> And use a GUC to enable vectorized executor, e.g:
> {code:java}
> postgres=# set vectorized_executor_enable to on;
> // run the new vectorized executor
> postgres=# set vectorized_executor_enable to off;
> // run the original HAWQ executor
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HAWQ-1591) Common tuple batch structure for vectorized execution

[Radar Lei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei updated HAWQ-1591:

Fix Version/s: (was: backlog)
   2.4.0.0-incubating

> Common tuple batch structure for vectorized execution
> -
>
> Key: HAWQ-1591
> URL: https://issues.apache.org/jira/browse/HAWQ-1591
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Query Execution
>Reporter: Hongxu Ma
>Assignee: WANG Weinan
>Priority: Major
> Fix For: 2.4.0.0-incubating
>
>
> A common tuple batch structure for vectorized execution, holds the tuples 
> which be transfered between vectorized operators.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HAWQ-1592) vectorized data types initialization and relevant function definetion

[Radar Lei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei updated HAWQ-1592:

Fix Version/s: (was: backlog)
   2.4.0.0-incubating

> vectorized data types initialization and relevant function definetion
> -
>
> Key: HAWQ-1592
> URL: https://issues.apache.org/jira/browse/HAWQ-1592
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Query Execution
>Reporter: WANG Weinan
>Assignee: zhangshujie
>Priority: Major
> Fix For: 2.4.0.0-incubating
>
>
> * vectorization data type initialization
>  * type relevant operation declaration
>  * expose these types in the catalog table
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HAWQ-1593) Vectorized execution condition check in plan tree

[Radar Lei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei updated HAWQ-1593:

Fix Version/s: (was: backlog)
   2.4.0.0-incubating

> Vectorized execution condition check in plan tree 
> --
>
> Key: HAWQ-1593
> URL: https://issues.apache.org/jira/browse/HAWQ-1593
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Query Execution
>Reporter: WANG Weinan
>Assignee: zhangshujie
>Priority: Major
> Fix For: 2.4.0.0-incubating
>
>
> check and assign "v" tag in plan tree each node.
> if a node is a leaf node and all expression can be vectorized execute, assign 
> a "v" tag 
> if a node's all child nodes are assigned "v" tag and all expression can be 
> vectorized execute, assign a "v" tag



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HAWQ-1597) Implement Runtime Filter for Hash Join

[Radar Lei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei updated HAWQ-1597:

Fix Version/s: 2.4.0.0-incubating

> Implement Runtime Filter for Hash Join
> --
>
> Key: HAWQ-1597
> URL: https://issues.apache.org/jira/browse/HAWQ-1597
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: Query Execution
>Reporter: Lin Wen
>Assignee: Lin Wen
>Priority: Major
> Fix For: 2.4.0.0-incubating
>
> Attachments: HAWQ Runtime Filter Design.pdf
>
>
> Bloom filter is a space-efficient probabilistic data structure invented in 
> 1970, which is used to test whether an element is a member of a set.
> Nowdays, bloom filter is widely used in OLAP or data-intensive applications 
> to quickly filter data. It is usually implemented in OLAP systems for hash 
> join. The basic idea is, when hash join two tables, during the build phase, 
> build a bloomfilter information for the inner table, then push down this 
> bloomfilter information to the scan of the outer table, so that, less tuples 
> from the outer table will be returned to hash join node and joined with hash 
> table. It can greatly improment the hash join performance if the selectivity 
> is high.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HAWQ-1601) Vectorized Scan qualification supported

[Radar Lei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei updated HAWQ-1601:

Fix Version/s: (was: backlog)
   2.4.0.0-incubating

> Vectorized Scan qualification supported
> ---
>
> Key: HAWQ-1601
> URL: https://issues.apache.org/jira/browse/HAWQ-1601
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Query Execution
>Reporter: WANG Weinan
>Assignee: WANG Weinan
>Priority: Major
> Fix For: 2.4.0.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HAWQ-1598) Vectorized Scan Node Framework initialization

[Radar Lei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei updated HAWQ-1598:

Fix Version/s: (was: backlog)
   2.4.0.0-incubating

> Vectorized Scan Node Framework initialization
> -
>
> Key: HAWQ-1598
> URL: https://issues.apache.org/jira/browse/HAWQ-1598
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Query Execution
>Reporter: WANG Weinan
>Assignee: WANG Weinan
>Priority: Major
> Fix For: 2.4.0.0-incubating
>
>
> Using the previous task defined hook function to init, proc and recycle 
> vectorized "TableScanState" node.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HAWQ-1600) Parquet table data vectorized scan

[Radar Lei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei updated HAWQ-1600:

Fix Version/s: (was: backlog)
   2.4.0.0-incubating

> Parquet table data vectorized scan
> --
>
> Key: HAWQ-1600
> URL: https://issues.apache.org/jira/browse/HAWQ-1600
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Query Execution
>Reporter: WANG Weinan
>Assignee: WANG Weinan
>Priority: Major
> Fix For: 2.4.0.0-incubating
>
>
> The simple query(e.g. "select col1 from tab1;") can be supported by 
> vectorized execution.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HAWQ-1602) AO table data vectorized scan

[Radar Lei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei updated HAWQ-1602:

Fix Version/s: (was: backlog)
   2.4.0.0-incubating

> AO table data vectorized scan 
> --
>
> Key: HAWQ-1602
> URL: https://issues.apache.org/jira/browse/HAWQ-1602
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Query Execution
>Reporter: WANG Weinan
>Assignee: WANG Weinan
>Priority: Major
> Fix For: 2.4.0.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HAWQ-1603) add new hook api for expressions

[Radar Lei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei updated HAWQ-1603:

Fix Version/s: (was: backlog)
   2.4.0.0-incubating

> add new hook api for expressions
> 
>
> Key: HAWQ-1603
> URL: https://issues.apache.org/jira/browse/HAWQ-1603
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Query Execution
>Reporter: zhangshujie
>Assignee: zhangshujie
>Priority: Major
> Fix For: 2.4.0.0-incubating
>
>
> 1.add new hook API for expressions
> 2.add new hook API for refactoring the plan tree



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HAWQ-1581) Separate PXF system parameters from user configurable visible parameters

[Radar Lei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei updated HAWQ-1581:

Fix Version/s: (was: 2.4.0.0-incubating)
   2.3.0.0-incubating

> Separate PXF system parameters from user configurable visible parameters
> 
>
> Key: HAWQ-1581
> URL: https://issues.apache.org/jira/browse/HAWQ-1581
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: PXF
>Reporter: Shivram Mani
>Assignee: Shivram Mani
>Priority: Major
> Fix For: 2.3.0.0-incubating
>
>
> We need to modify our system such that user configurable options are kept 
> distinct form the internal parameters. The custom parameters are configured 
> in the {{LOCATION}} section of the external table DDL, is exposed to PXF 
> server as {{X-GP-}}.
> {{X-GP-USER}} is an internal parameter used to set the user information. When 
> the DDL has a custom parameter named {{user}} it ends up updating X-GP-USER 
> to also include the user configured in the DDL Location. This causes the JDBC 
> connector to fail.
> We will instead use \{X-GP-OPTIONS-} as the prefix for all user configurable 
> parameters to keep them isolated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HAWQ-1579) x When you enable pxf DEBUG logging you might get annoying exceptions

[Radar Lei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei updated HAWQ-1579:

Fix Version/s: (was: 2.4.0.0-incubating)
   2.3.0.0-incubating

> x When you enable pxf DEBUG logging you might get annoying exceptions
> -
>
> Key: HAWQ-1579
> URL: https://issues.apache.org/jira/browse/HAWQ-1579
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables
>Reporter: Dmitriy Dorofeev
>Assignee: Shivram Mani
>Priority: Major
> Fix For: 2.3.0.0-incubating
>
>
> When you enable DEBUG logging you might get annoying exceptions:
>  
> {{SEVERE: The RuntimeException could not be mapped to a response, re-throwing 
> to the HTTP container java.lang.NullPointerException at 
> java.lang.String.(String.java:566) at 
> org.apache.hawq.pxf.service.FragmentsResponseFormatter.printList(FragmentsResponseFormatter.java:147)
>  at 
> org.apache.hawq.pxf.service.FragmentsResponseFormatter.formatResponse(FragmentsResponseFormatter.java:54)
>  at 
> org.apache.hawq.pxf.service.rest.FragmenterResource.getFragments(FragmenterResource.java:88)
>  }}
> {{ }}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HAWQ-1566) Include Pluggable Storage Format Framework in External Table Insert

[Radar Lei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei updated HAWQ-1566:

Fix Version/s: (was: 2.4.0.0-incubating)
   2.3.0.0-incubating

> Include Pluggable Storage Format Framework in External Table Insert
> ---
>
> Key: HAWQ-1566
> URL: https://issues.apache.org/jira/browse/HAWQ-1566
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: External Tables, Storage
>Reporter: Chiyang Wan
>Assignee: Chiyang Wan
>Priority: Major
> Fix For: 2.3.0.0-incubating
>
>
> There are 2 types of operation related to external table, i.e. scan, insert. 
> Including pluggable storage framework in these operations is necessary. 
> We add the external table insert and copy from(write into external table) 
> related feature here.
> In the following steps, we still need to specify some of the critical info 
> that comes from the planner and the file splits info in the pluggable 
> filesystem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HAWQ-1036) Support user impersonation in PXF for external tables

[Radar Lei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei updated HAWQ-1036:

Fix Version/s: (was: 2.4.0.0-incubating)
   2.3.0.0-incubating

> Support user impersonation in PXF for external tables
> -
>
> Key: HAWQ-1036
> URL: https://issues.apache.org/jira/browse/HAWQ-1036
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: PXF, Security
>Reporter: Alastair "Bell" Turner
>Assignee: Alexander Denissov
>Priority: Critical
> Fix For: 2.3.0.0-incubating
>
> Attachments: HAWQ_Impersonation_rationale.txt
>
>
> Currently HAWQ executes all queries as the user running the HAWQ process or 
> the user running the PXF process, not as the user who issued the query via 
> ODBC/JDBC/... This restricts the options available for integrating with 
> existing security defined in HDFS, Hive, etc.
> Impersonation provides an alternative Ranger integration (as discussed in 
> HAWQ-256 ) for consistent security across HAWQ, HDFS, Hive...
> Implementation High Level steps:
> 1) HAWQ needs to integrate with existing authentication components for the 
> user who invokes the query
> 2) HAWQ needs to pass down the user id to PXF after authorization is passed 
> 3) PXF needs to do "run as ..." the user id to execute APIs to access 
> Hive/HDFS 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HAWQ-1058) Create a separated tarball for libhdfs3

[Radar Lei (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radar Lei updated HAWQ-1058:

Fix Version/s: (was: 2.4.0.0-incubating)
   backlog

> Create a separated tarball for libhdfs3
> ---
>
> Key: HAWQ-1058
> URL: https://issues.apache.org/jira/browse/HAWQ-1058
> Project: Apache HAWQ
>  Issue Type: Test
>  Components: libhdfs
>Affects Versions: 2.0.0.0-incubating
>Reporter: Zhanwei Wang
>Assignee: Lei Chang
>Priority: Major
> Fix For: backlog
>
>
> As discussed in the dev mail list. Proposed by Ramon that create a separated 
> tarball for libhdfs3 at HAWQ release.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)