[jira] [Updated] (HAWQ-1436) Implement RPS High availability on HAWQ
[ https://issues.apache.org/jira/browse/HAWQ-1436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma updated HAWQ-1436: Description: Once Ranger is configured, HAWQ will rely on RPS to connect to Ranger. A single point RPS may influence the robustness of HAWQ. Thus We need to investigate and design out the way to implement RPS High availability. was:Once Ranger is configured, HAWQ will rely on RPS to connect to Ranger. A single point RPS may influence the robustness of HAWQ. Thus We need to investigate and design out the way to implement RPS High availability. > Implement RPS High availability on HAWQ > --- > > Key: HAWQ-1436 > URL: https://issues.apache.org/jira/browse/HAWQ-1436 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Hongxu Ma >Assignee: Hongxu Ma > Fix For: backlog > > Attachments: RPSHADesign_v0.1.pdf > > > Once Ranger is configured, HAWQ will rely on RPS to connect to Ranger. A > single point RPS may influence the robustness of HAWQ. > Thus We need to investigate and design out the way to implement RPS High > availability. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HAWQ-1436) Implement RPS High availability on HAWQ
[ https://issues.apache.org/jira/browse/HAWQ-1436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma updated HAWQ-1436: Attachment: RPSHADesign_v0.1.pdf RPS HA Design > Implement RPS High availability on HAWQ > --- > > Key: HAWQ-1436 > URL: https://issues.apache.org/jira/browse/HAWQ-1436 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Hongxu Ma >Assignee: Hongxu Ma > Fix For: backlog > > Attachments: RPSHADesign_v0.1.pdf > > > Once Ranger is configured, HAWQ will rely on RPS to connect to Ranger. A > single point RPS may influence the robustness of HAWQ. Thus We need to > investigate and design out the way to implement RPS High availability. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HAWQ-1436) Implement RPS High availability on HAWQ
[ https://issues.apache.org/jira/browse/HAWQ-1436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma reassigned HAWQ-1436: --- Assignee: Hongxu Ma (was: Ed Espino) > Implement RPS High availability on HAWQ > --- > > Key: HAWQ-1436 > URL: https://issues.apache.org/jira/browse/HAWQ-1436 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Hongxu Ma >Assignee: Hongxu Ma > Fix For: backlog > > > Once Ranger is configured, HAWQ will rely on RPS to connect to Ranger. A > single point RPS may influence the robustness of HAWQ. Thus We need to > investigate and design out the way to implement RPS High availability. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HAWQ-1436) Implement RPS High availability on HAWQ
Hongxu Ma created HAWQ-1436: --- Summary: Implement RPS High availability on HAWQ Key: HAWQ-1436 URL: https://issues.apache.org/jira/browse/HAWQ-1436 Project: Apache HAWQ Issue Type: Sub-task Components: Security Reporter: Hongxu Ma Assignee: Ed Espino Fix For: backlog Once Ranger is configured, HAWQ will rely on RPS to connect to Ranger. A single point RPS may influence the robustness of HAWQ. Thus We need to investigate and design out the way to implement RPS High availability. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (HAWQ-1410) Add basic test case for hcatalog with ranger
[ https://issues.apache.org/jira/browse/HAWQ-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma closed HAWQ-1410. --- Resolution: Fixed duplicates > Add basic test case for hcatalog with ranger > > > Key: HAWQ-1410 > URL: https://issues.apache.org/jira/browse/HAWQ-1410 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Hongxu Ma >Assignee: Hongxu Ma > Fix For: 2.2.0.0-incubating > > > add test case: > basic operations on hcatalog with ranger > more pxf test with ranger will be added in future. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HAWQ-1410) add basic test case for hcatalog with ranger
[ https://issues.apache.org/jira/browse/HAWQ-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma reassigned HAWQ-1410: --- Assignee: Hongxu Ma (was: Ed Espino) > add basic test case for hcatalog with ranger > > > Key: HAWQ-1410 > URL: https://issues.apache.org/jira/browse/HAWQ-1410 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Hongxu Ma >Assignee: Hongxu Ma > Fix For: 2.2.0.0-incubating > > > add test case: > basic operations on hcatalog with ranger > more pxf test with ranger will be added in future. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HAWQ-1410) Add basic test case for hcatalog with ranger
[ https://issues.apache.org/jira/browse/HAWQ-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma updated HAWQ-1410: Summary: Add basic test case for hcatalog with ranger (was: add basic test case for hcatalog with ranger) > Add basic test case for hcatalog with ranger > > > Key: HAWQ-1410 > URL: https://issues.apache.org/jira/browse/HAWQ-1410 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Hongxu Ma >Assignee: Hongxu Ma > Fix For: 2.2.0.0-incubating > > > add test case: > basic operations on hcatalog with ranger > more pxf test with ranger will be added in future. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HAWQ-1410) add basic test case for hcatalog with ranger
Hongxu Ma created HAWQ-1410: --- Summary: add basic test case for hcatalog with ranger Key: HAWQ-1410 URL: https://issues.apache.org/jira/browse/HAWQ-1410 Project: Apache HAWQ Issue Type: Sub-task Components: Security Reporter: Hongxu Ma Assignee: Ed Espino Fix For: 2.2.0.0-incubating add test case: basic operations on hcatalog with ranger more pxf test with ranger will be added in future. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (HAWQ-1405) 'hawq stop --reload' should not stop RPS
[ https://issues.apache.org/jira/browse/HAWQ-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma closed HAWQ-1405. --- Resolution: Fixed commited. > 'hawq stop --reload' should not stop RPS > > > Key: HAWQ-1405 > URL: https://issues.apache.org/jira/browse/HAWQ-1405 > Project: Apache HAWQ > Issue Type: Bug > Components: Command Line Tools >Reporter: Hongxu Ma >Assignee: Hongxu Ma > Fix For: 2.2.0.0-incubating > > > Currently, running 'hawq stop cluster --reload', RPS is stopped, but not > restarted. > We should not stop RPS when running 'hawq stop --reload'. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HAWQ-304) Support update and delete on non-heap tables
[
https://issues.apache.org/jira/browse/HAWQ-304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929455#comment-15929455
]
Hongxu Ma edited comment on HAWQ-304 at 3/22/17 3:00 AM:
-
Great job @[~tzolov]
I have tried your calcite-sql-rewriter, but failed on HAWQ:
https://github.com/tzolov/calcite-sql-rewriter/issues/1
Not support HAWQ now?
Update: already support HAWQ, thanks Christian
demo:
{code}
$ java -jar ./target/journalled-sql-rewriter-example-1.7-SNAPSHOT.jar
src/main/resources/myTestModel.json
[main] INFO io.pivotal.calcite.sqlrewriter.Main - INSERT INTO hr.depts (deptno,
department_name) VALUES(696, 'Pivotal')
[main] INFO io.pivotal.calcite.sqlrewriter.Main - updated rows: 1
[main] INFO io.pivotal.calcite.sqlrewriter.Main - SELECT * FROM hr.depts
[main] INFO io.pivotal.calcite.sqlrewriter.Main - result: 696 , Pivotal ,
[main] INFO io.pivotal.calcite.sqlrewriter.Main - UPDATE hr.depts SET
department_name='interma' WHERE deptno = 696
[main] INFO io.pivotal.calcite.sqlrewriter.Main - updated rows: 1
[main] INFO io.pivotal.calcite.sqlrewriter.Main - SELECT * FROM hr.depts
[main] INFO io.pivotal.calcite.sqlrewriter.Main - result: 696 , interma ,
[main] INFO io.pivotal.calcite.sqlrewriter.Main - DELETE FROM hr.depts WHERE
deptno = 696
[main] INFO io.pivotal.calcite.sqlrewriter.Main - updated rows: 1
[main] INFO io.pivotal.calcite.sqlrewriter.Main - SELECT * FROM hr.depts
[main] INFO io.pivotal.calcite.sqlrewriter.Main - result:
[main] INFO io.pivotal.calcite.sqlrewriter.Main - Done
{code}
was (Author: hongxu ma):
Great job @[~tzolov]
I have tried your calcite-sql-rewriter, but failed on HAWQ:
https://github.com/tzolov/calcite-sql-rewriter/issues/1
Not support HAWQ now?
> Support update and delete on non-heap tables
>
>
> Key: HAWQ-304
> URL: https://issues.apache.org/jira/browse/HAWQ-304
> Project: Apache HAWQ
> Issue Type: Wish
> Components: Storage
>Reporter: Lei Chang
>Assignee: Lili Ma
> Fix For: 3.0.0.0
>
> Attachments: mutable_table.sql
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HAWQ-1393) 'hawq stop cluster' failed when rps.sh have some path errors (e.g. CATALINA_HOME)
Hongxu Ma created HAWQ-1393:
---
Summary: 'hawq stop cluster' failed when rps.sh have some path
errors (e.g. CATALINA_HOME)
Key: HAWQ-1393
URL: https://issues.apache.org/jira/browse/HAWQ-1393
Project: Apache HAWQ
Issue Type: Bug
Components: Command Line Tools
Reporter: Hongxu Ma
Assignee: Ed Espino
Fix For: 2.2.0.0-incubating
example (hawq master already stopped)
{code}
$ hawq stop cluster -a
20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[INFO]:-Prepare to do 'hawq
stop'
20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[INFO]:-You can find log in:
20170320:16:43:23:038375
hawq_stop:interma-mbp:hma-[INFO]:-/Users/hma/hawqAdminLogs/hawq_stop_20170320.log
20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[INFO]:-GPHOME is set to:
20170320:16:43:23:038375
hawq_stop:interma-mbp:hma-[INFO]:-/Users/hma/hawq/install
20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[INFO]:-Stop hawq with args:
['stop', 'cluster']
20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[INFO]:-No standby host
configured
20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[WARNING]:-Failed to connect
to database, cannot get hawq_acl_type
20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[INFO]:-Stop hawq cluster
20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[WARNING]:-HAWQ process is
not running on localhost, skip
20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[WARNING]:-Master is not
running, skip
20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[INFO]:-Master stopped
successfully
20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[WARNING]:-try to stop RPS
when hawq_acl_type is unknown
20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[INFO]:-Stop Ranger plugin
service
20170320:16:43:24:038375 hawq_stop:interma-mbp:hma-[INFO]:-Cannot find
/usr/lib/bigtop-tomcat/bin/setclasspath.sh
This file is needed to run this program
FATAL: Failed to stop HAWQ Ranger Plugin Service. Check
/Users/hma/hawq/install/ranger/plugin-service/logs/catalina.out for details.
20170320:16:43:24:038375 hawq_stop:interma-mbp:hma-[ERROR]:-Ranger plugin
service stop failed, exit
{code}
We wish to continue to run the remain logic of hawq stop.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HAWQ-1393) 'hawq stop cluster' failed when rps.sh have some path errors (e.g. CATALINA_HOME)
[
https://issues.apache.org/jira/browse/HAWQ-1393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongxu Ma reassigned HAWQ-1393:
---
Assignee: Hongxu Ma (was: Ed Espino)
> 'hawq stop cluster' failed when rps.sh have some path errors (e.g.
> CATALINA_HOME)
> -
>
> Key: HAWQ-1393
> URL: https://issues.apache.org/jira/browse/HAWQ-1393
> Project: Apache HAWQ
> Issue Type: Bug
> Components: Command Line Tools
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: 2.2.0.0-incubating
>
>
> example (hawq master already stopped)
> {code}
> $ hawq stop cluster -a
> 20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[INFO]:-Prepare to do
> 'hawq stop'
> 20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[INFO]:-You can find log
> in:
> 20170320:16:43:23:038375
> hawq_stop:interma-mbp:hma-[INFO]:-/Users/hma/hawqAdminLogs/hawq_stop_20170320.log
> 20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[INFO]:-GPHOME is set to:
> 20170320:16:43:23:038375
> hawq_stop:interma-mbp:hma-[INFO]:-/Users/hma/hawq/install
> 20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[INFO]:-Stop hawq with
> args: ['stop', 'cluster']
> 20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[INFO]:-No standby host
> configured
> 20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[WARNING]:-Failed to
> connect to database, cannot get hawq_acl_type
> 20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[INFO]:-Stop hawq cluster
> 20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[WARNING]:-HAWQ process is
> not running on localhost, skip
> 20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[WARNING]:-Master is not
> running, skip
> 20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[INFO]:-Master stopped
> successfully
> 20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[WARNING]:-try to stop RPS
> when hawq_acl_type is unknown
> 20170320:16:43:23:038375 hawq_stop:interma-mbp:hma-[INFO]:-Stop Ranger plugin
> service
> 20170320:16:43:24:038375 hawq_stop:interma-mbp:hma-[INFO]:-Cannot find
> /usr/lib/bigtop-tomcat/bin/setclasspath.sh
> This file is needed to run this program
> FATAL: Failed to stop HAWQ Ranger Plugin Service. Check
> /Users/hma/hawq/install/ranger/plugin-service/logs/catalina.out for details.
> 20170320:16:43:24:038375 hawq_stop:interma-mbp:hma-[ERROR]:-Ranger plugin
> service stop failed, exit
> {code}
> We wish to continue to run the remain logic of hawq stop.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HAWQ-304) Support update and delete on non-heap tables
[ https://issues.apache.org/jira/browse/HAWQ-304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929455#comment-15929455 ] Hongxu Ma commented on HAWQ-304: Great job [~tzolov] I have tried your calcite-sql-rewriter, but failed on HAWQ: https://github.com/tzolov/calcite-sql-rewriter/issues/1 not supported HAWQ now? > Support update and delete on non-heap tables > > > Key: HAWQ-304 > URL: https://issues.apache.org/jira/browse/HAWQ-304 > Project: Apache HAWQ > Issue Type: Wish > Components: Storage >Reporter: Lei Chang >Assignee: Lili Ma > Fix For: 3.0.0.0 > > Attachments: mutable_table.sql > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HAWQ-304) Support update and delete on non-heap tables
[ https://issues.apache.org/jira/browse/HAWQ-304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929455#comment-15929455 ] Hongxu Ma edited comment on HAWQ-304 at 3/17/17 5:54 AM: - Great job @[~tzolov] I have tried your calcite-sql-rewriter, but failed on HAWQ: https://github.com/tzolov/calcite-sql-rewriter/issues/1 Not support HAWQ now? was (Author: hongxu ma): Great job [~tzolov] I have tried your calcite-sql-rewriter, but failed on HAWQ: https://github.com/tzolov/calcite-sql-rewriter/issues/1 not supported HAWQ now? > Support update and delete on non-heap tables > > > Key: HAWQ-304 > URL: https://issues.apache.org/jira/browse/HAWQ-304 > Project: Apache HAWQ > Issue Type: Wish > Components: Storage >Reporter: Lei Chang >Assignee: Lili Ma > Fix For: 3.0.0.0 > > Attachments: mutable_table.sql > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (HAWQ-1385) hawq_ctl stop failed when master is down
[
https://issues.apache.org/jira/browse/HAWQ-1385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongxu Ma closed HAWQ-1385.
---
Resolution: Fixed
fixed
> hawq_ctl stop failed when master is down
>
>
> Key: HAWQ-1385
> URL: https://issues.apache.org/jira/browse/HAWQ-1385
> Project: Apache HAWQ
> Issue Type: Bug
> Components: Command Line Tools
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: 2.2.0.0-incubating
>
>
> example (master is down):
> hawq activate standby
> hawq stop allsegments
> error output:
> {code}
> 20170314:03:49:36:005905
> hawq_activate:centos7-datanode3:gpadmin-[INFO]:-20170314:03:49:36:005933
> hawq_stop:centos7-datanode3:gpadmin-[ERROR]:-Failed to connect to database,
> this script can only be run when the database is up
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HAWQ-1385) hawq_ctl stop failed when master is down
[
https://issues.apache.org/jira/browse/HAWQ-1385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongxu Ma reassigned HAWQ-1385:
---
Assignee: Hongxu Ma (was: Ed Espino)
> hawq_ctl stop failed when master is down
>
>
> Key: HAWQ-1385
> URL: https://issues.apache.org/jira/browse/HAWQ-1385
> Project: Apache HAWQ
> Issue Type: Bug
> Components: Command Line Tools
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: 2.2.0.0-incubating
>
>
> example (master is down):
> hawq activate standby
> hawq stop allsegments
> error output:
> {code}
> 20170314:03:49:36:005905
> hawq_activate:centos7-datanode3:gpadmin-[INFO]:-20170314:03:49:36:005933
> hawq_stop:centos7-datanode3:gpadmin-[ERROR]:-Failed to connect to database,
> this script can only be run when the database is up
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HAWQ-1385) hawq_ctl stop failed when master is down
Hongxu Ma created HAWQ-1385:
---
Summary: hawq_ctl stop failed when master is down
Key: HAWQ-1385
URL: https://issues.apache.org/jira/browse/HAWQ-1385
Project: Apache HAWQ
Issue Type: Bug
Components: Command Line Tools
Reporter: Hongxu Ma
Assignee: Ed Espino
Fix For: 2.2.0.0-incubating
example (master is down):
hawq activate standby
hawq stop allsegments
error output:
{code}
20170314:03:49:36:005905
hawq_activate:centos7-datanode3:gpadmin-[INFO]:-20170314:03:49:36:005933
hawq_stop:centos7-datanode3:gpadmin-[ERROR]:-Failed to connect to database,
this script can only be run when the database is up
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (HAWQ-1380) Keep hawq_toolkit schema check in HAWQ native side
[ https://issues.apache.org/jira/browse/HAWQ-1380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma closed HAWQ-1380. --- Resolution: Fixed fixed > Keep hawq_toolkit schema check in HAWQ native side > -- > > Key: HAWQ-1380 > URL: https://issues.apache.org/jira/browse/HAWQ-1380 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Hongxu Ma >Assignee: Hongxu Ma > Fix For: 2.2.0.0-incubating > > > Given that *hawq_toolkit* is an administrative schema, let's keep it in HAWQ > Native, outside of Ranger. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (HAWQ-1381) Core dump when execute 'select * from hawq_toolkit.__hawq_log_master_ext;' on macOS
[
https://issues.apache.org/jira/browse/HAWQ-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongxu Ma closed HAWQ-1381.
---
Resolution: Fixed
fixed
> Core dump when execute 'select * from hawq_toolkit.__hawq_log_master_ext;' on
> macOS
> ---
>
> Key: HAWQ-1381
> URL: https://issues.apache.org/jira/browse/HAWQ-1381
> Project: Apache HAWQ
> Issue Type: Bug
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: 2.2.0.0-incubating
>
>
> macOS 10.12.1
> {code}
> Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
> 0 libsystem_kernel.dylib 0x7fffded60dda __pthread_kill + 10
> 1 libsystem_pthread.dylib 0x7fffdee4c787 pthread_kill + 90
> 2 libsystem_c.dylib 0x7fffdecc6420 abort + 129
> 3 libsystem_c.dylib 0x7fffdecc6592 abort_report_np + 181
> 4 libsystem_c.dylib 0x7fffdececf28 __chk_fail + 48
> 5 libsystem_c.dylib 0x7fffdececef8 __chk_fail_overflow +
> 16
> 6 libsystem_c.dylib 0x7fffdeced413 __sprintf_chk + 199
> 7 postgres0x00010e39e394 external_set_env_vars
> + 916
> 8 postgres0x00010e39c335
> open_external_readable_source + 181
> 9 postgres0x00010e39cba6 external_getnext + 70
> 10 postgres0x00010e62a42e ExternalNext + 110
> 11 postgres0x00010e606518 ExecScan + 72
> 12 postgres0x00010e62a3af ExecExternalScan + 31
> 13 postgres0x00010e5f4bd3 ExecProcNode + 739
> 14 postgres0x00010e5e88e2 ExecutePlan + 722
> 15 postgres0x00010e5e82af ExecutorRun + 1471
> 16 postgres0x00010e83531d PortalRunSelect + 317
> 17 postgres0x00010e834dc8 PortalRun + 952
> 18 postgres0x00010e82ad0f exec_simple_query +
> 2367
> 19 postgres0x00010e828eeb PostgresMain + 7979
> 20 postgres0x00010e7c4198 BackendRun + 1048
> 21 postgres0x00010e7c0f35 BackendStartup + 373
> 22 postgres0x00010e7bde10 ServerLoop + 1248
> 23 postgres0x00010e7bc3cb PostmasterMain + 4859
> 24 postgres0x00010e69f22c main + 940
> 25 libdyld.dylib 0x7fffdec32255 start + 1
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HAWQ-1381) Core dump when execute 'select * from hawq_toolkit.__hawq_log_master_ext;' on macOS
[
https://issues.apache.org/jira/browse/HAWQ-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900529#comment-15900529
]
Hongxu Ma edited comment on HAWQ-1381 at 3/8/17 2:11 AM:
-
buffer overflow here:
src/backend/access/external/fileam.c:2610
{code}
sprintf(extvar->GP_SEGMENT_ID, "%d", GetQEIndex());
{code}
GetQEIndex() return -1 and GP_SEGMENT_ID is char[6], no more space for
'\0', so it happend.
was (Author: hongxu ma):
buffer overflow here:
src/backend/access/external/fileam.c:2610
{code}
sprintf(extvar->GP_SEGMENT_ID, "%d", GetQEIndex());
{code}
GetQEIndex() return -1 and GP_SEGMENT_ID is char[6], so
> Core dump when execute 'select * from hawq_toolkit.__hawq_log_master_ext;' on
> macOS
> ---
>
> Key: HAWQ-1381
> URL: https://issues.apache.org/jira/browse/HAWQ-1381
> Project: Apache HAWQ
> Issue Type: Bug
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: 2.2.0.0-incubating
>
>
> macOS 10.12.1
> {code}
> Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
> 0 libsystem_kernel.dylib 0x7fffded60dda __pthread_kill + 10
> 1 libsystem_pthread.dylib 0x7fffdee4c787 pthread_kill + 90
> 2 libsystem_c.dylib 0x7fffdecc6420 abort + 129
> 3 libsystem_c.dylib 0x7fffdecc6592 abort_report_np + 181
> 4 libsystem_c.dylib 0x7fffdececf28 __chk_fail + 48
> 5 libsystem_c.dylib 0x7fffdececef8 __chk_fail_overflow +
> 16
> 6 libsystem_c.dylib 0x7fffdeced413 __sprintf_chk + 199
> 7 postgres0x00010e39e394 external_set_env_vars
> + 916
> 8 postgres0x00010e39c335
> open_external_readable_source + 181
> 9 postgres0x00010e39cba6 external_getnext + 70
> 10 postgres0x00010e62a42e ExternalNext + 110
> 11 postgres0x00010e606518 ExecScan + 72
> 12 postgres0x00010e62a3af ExecExternalScan + 31
> 13 postgres0x00010e5f4bd3 ExecProcNode + 739
> 14 postgres0x00010e5e88e2 ExecutePlan + 722
> 15 postgres0x00010e5e82af ExecutorRun + 1471
> 16 postgres0x00010e83531d PortalRunSelect + 317
> 17 postgres0x00010e834dc8 PortalRun + 952
> 18 postgres0x00010e82ad0f exec_simple_query +
> 2367
> 19 postgres0x00010e828eeb PostgresMain + 7979
> 20 postgres0x00010e7c4198 BackendRun + 1048
> 21 postgres0x00010e7c0f35 BackendStartup + 373
> 22 postgres0x00010e7bde10 ServerLoop + 1248
> 23 postgres0x00010e7bc3cb PostmasterMain + 4859
> 24 postgres0x00010e69f22c main + 940
> 25 libdyld.dylib 0x7fffdec32255 start + 1
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HAWQ-1381) Core dump when execute 'select * from hawq_toolkit.__hawq_log_master_ext;' on macOS
[
https://issues.apache.org/jira/browse/HAWQ-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900529#comment-15900529
]
Hongxu Ma commented on HAWQ-1381:
-
buffer overflow here:
src/backend/access/external/fileam.c:2610
{code}
sprintf(extvar->GP_SEGMENT_ID, "%d", GetQEIndex());
{code}
GetQEIndex() return -1 and GP_SEGMENT_ID is char[6], so
> Core dump when execute 'select * from hawq_toolkit.__hawq_log_master_ext;' on
> macOS
> ---
>
> Key: HAWQ-1381
> URL: https://issues.apache.org/jira/browse/HAWQ-1381
> Project: Apache HAWQ
> Issue Type: Bug
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: 2.2.0.0-incubating
>
>
> macOS 10.12.1
> {code}
> Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
> 0 libsystem_kernel.dylib 0x7fffded60dda __pthread_kill + 10
> 1 libsystem_pthread.dylib 0x7fffdee4c787 pthread_kill + 90
> 2 libsystem_c.dylib 0x7fffdecc6420 abort + 129
> 3 libsystem_c.dylib 0x7fffdecc6592 abort_report_np + 181
> 4 libsystem_c.dylib 0x7fffdececf28 __chk_fail + 48
> 5 libsystem_c.dylib 0x7fffdececef8 __chk_fail_overflow +
> 16
> 6 libsystem_c.dylib 0x7fffdeced413 __sprintf_chk + 199
> 7 postgres0x00010e39e394 external_set_env_vars
> + 916
> 8 postgres0x00010e39c335
> open_external_readable_source + 181
> 9 postgres0x00010e39cba6 external_getnext + 70
> 10 postgres0x00010e62a42e ExternalNext + 110
> 11 postgres0x00010e606518 ExecScan + 72
> 12 postgres0x00010e62a3af ExecExternalScan + 31
> 13 postgres0x00010e5f4bd3 ExecProcNode + 739
> 14 postgres0x00010e5e88e2 ExecutePlan + 722
> 15 postgres0x00010e5e82af ExecutorRun + 1471
> 16 postgres0x00010e83531d PortalRunSelect + 317
> 17 postgres0x00010e834dc8 PortalRun + 952
> 18 postgres0x00010e82ad0f exec_simple_query +
> 2367
> 19 postgres0x00010e828eeb PostgresMain + 7979
> 20 postgres0x00010e7c4198 BackendRun + 1048
> 21 postgres0x00010e7c0f35 BackendStartup + 373
> 22 postgres0x00010e7bde10 ServerLoop + 1248
> 23 postgres0x00010e7bc3cb PostmasterMain + 4859
> 24 postgres0x00010e69f22c main + 940
> 25 libdyld.dylib 0x7fffdec32255 start + 1
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HAWQ-1381) Core dump when execute 'select * from hawq_toolkit.__hawq_log_master_ext;' on macOS
[
https://issues.apache.org/jira/browse/HAWQ-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongxu Ma reassigned HAWQ-1381:
---
Assignee: Hongxu Ma (was: Ed Espino)
> Core dump when execute 'select * from hawq_toolkit.__hawq_log_master_ext;' on
> macOS
> ---
>
> Key: HAWQ-1381
> URL: https://issues.apache.org/jira/browse/HAWQ-1381
> Project: Apache HAWQ
> Issue Type: Bug
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: 2.2.0.0-incubating
>
>
> macOS 10.12.1
> {code}
> Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
> 0 libsystem_kernel.dylib 0x7fffded60dda __pthread_kill + 10
> 1 libsystem_pthread.dylib 0x7fffdee4c787 pthread_kill + 90
> 2 libsystem_c.dylib 0x7fffdecc6420 abort + 129
> 3 libsystem_c.dylib 0x7fffdecc6592 abort_report_np + 181
> 4 libsystem_c.dylib 0x7fffdececf28 __chk_fail + 48
> 5 libsystem_c.dylib 0x7fffdececef8 __chk_fail_overflow +
> 16
> 6 libsystem_c.dylib 0x7fffdeced413 __sprintf_chk + 199
> 7 postgres0x00010e39e394 external_set_env_vars
> + 916
> 8 postgres0x00010e39c335
> open_external_readable_source + 181
> 9 postgres0x00010e39cba6 external_getnext + 70
> 10 postgres0x00010e62a42e ExternalNext + 110
> 11 postgres0x00010e606518 ExecScan + 72
> 12 postgres0x00010e62a3af ExecExternalScan + 31
> 13 postgres0x00010e5f4bd3 ExecProcNode + 739
> 14 postgres0x00010e5e88e2 ExecutePlan + 722
> 15 postgres0x00010e5e82af ExecutorRun + 1471
> 16 postgres0x00010e83531d PortalRunSelect + 317
> 17 postgres0x00010e834dc8 PortalRun + 952
> 18 postgres0x00010e82ad0f exec_simple_query +
> 2367
> 19 postgres0x00010e828eeb PostgresMain + 7979
> 20 postgres0x00010e7c4198 BackendRun + 1048
> 21 postgres0x00010e7c0f35 BackendStartup + 373
> 22 postgres0x00010e7bde10 ServerLoop + 1248
> 23 postgres0x00010e7bc3cb PostmasterMain + 4859
> 24 postgres0x00010e69f22c main + 940
> 25 libdyld.dylib 0x7fffdec32255 start + 1
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HAWQ-1381) Core dump when execute 'select * from hawq_toolkit.__hawq_log_master_ext;' on macOS
Hongxu Ma created HAWQ-1381:
---
Summary: Core dump when execute 'select * from
hawq_toolkit.__hawq_log_master_ext;' on macOS
Key: HAWQ-1381
URL: https://issues.apache.org/jira/browse/HAWQ-1381
Project: Apache HAWQ
Issue Type: Bug
Reporter: Hongxu Ma
Assignee: Ed Espino
Fix For: 2.2.0.0-incubating
macOS 10.12.1
{code}
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 0x7fffded60dda __pthread_kill + 10
1 libsystem_pthread.dylib 0x7fffdee4c787 pthread_kill + 90
2 libsystem_c.dylib 0x7fffdecc6420 abort + 129
3 libsystem_c.dylib 0x7fffdecc6592 abort_report_np + 181
4 libsystem_c.dylib 0x7fffdececf28 __chk_fail + 48
5 libsystem_c.dylib 0x7fffdececef8 __chk_fail_overflow + 16
6 libsystem_c.dylib 0x7fffdeced413 __sprintf_chk + 199
7 postgres0x00010e39e394 external_set_env_vars +
916
8 postgres0x00010e39c335
open_external_readable_source + 181
9 postgres0x00010e39cba6 external_getnext + 70
10 postgres0x00010e62a42e ExternalNext + 110
11 postgres0x00010e606518 ExecScan + 72
12 postgres0x00010e62a3af ExecExternalScan + 31
13 postgres0x00010e5f4bd3 ExecProcNode + 739
14 postgres0x00010e5e88e2 ExecutePlan + 722
15 postgres0x00010e5e82af ExecutorRun + 1471
16 postgres0x00010e83531d PortalRunSelect + 317
17 postgres0x00010e834dc8 PortalRun + 952
18 postgres0x00010e82ad0f exec_simple_query + 2367
19 postgres0x00010e828eeb PostgresMain + 7979
20 postgres0x00010e7c4198 BackendRun + 1048
21 postgres0x00010e7c0f35 BackendStartup + 373
22 postgres0x00010e7bde10 ServerLoop + 1248
23 postgres0x00010e7bc3cb PostmasterMain + 4859
24 postgres0x00010e69f22c main + 940
25 libdyld.dylib 0x7fffdec32255 start + 1
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HAWQ-1380) Keep hawq_toolkit schema check in HAWQ native side
Hongxu Ma created HAWQ-1380: --- Summary: Keep hawq_toolkit schema check in HAWQ native side Key: HAWQ-1380 URL: https://issues.apache.org/jira/browse/HAWQ-1380 Project: Apache HAWQ Issue Type: Sub-task Reporter: Hongxu Ma Assignee: Ed Espino Fix For: 2.2.0.0-incubating Given that *hawq_toolkit* is an administrative schema, let's keep it in HAWQ Native, outside of Ranger. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HAWQ-1380) Keep hawq_toolkit schema check in HAWQ native side
[ https://issues.apache.org/jira/browse/HAWQ-1380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma reassigned HAWQ-1380: --- Assignee: Hongxu Ma (was: Ed Espino) > Keep hawq_toolkit schema check in HAWQ native side > -- > > Key: HAWQ-1380 > URL: https://issues.apache.org/jira/browse/HAWQ-1380 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Hongxu Ma >Assignee: Hongxu Ma > Fix For: 2.2.0.0-incubating > > > Given that *hawq_toolkit* is an administrative schema, let's keep it in HAWQ > Native, outside of Ranger. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (HAWQ-1365) Print out detailed schema information for tables which the user doesn't have privileges
[
https://issues.apache.org/jira/browse/HAWQ-1365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongxu Ma closed HAWQ-1365.
---
Resolution: Fixed
> Print out detailed schema information for tables which the user doesn't have
> privileges
> ---
>
> Key: HAWQ-1365
> URL: https://issues.apache.org/jira/browse/HAWQ-1365
> Project: Apache HAWQ
> Issue Type: Improvement
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: 2.2.0.0-incubating
>
>
> Business Value:
> Current output information for the table name which user doesn't have
> privileges doesn't include schema output. We should print out the schema
> information out, otherwise it's difficult for users to identify the
> identified table.
> {code}
> postgres=# select * from public.a, s1.a;
> ERROR: permission denied for relation(s): a, a
> {code}
> Should print out the schema information for the table.
> {code}
> postgres=# select * from public.a, s1.a;
> ERROR: permission denied for relation(s): public.a, s1.a
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HAWQ-1365) Print out detailed schema information for tables which the user doesn't have privileges
[
https://issues.apache.org/jira/browse/HAWQ-1365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887476#comment-15887476
]
Hongxu Ma commented on HAWQ-1365:
-
new ERROR info:
{code}
postgres=> select * from public.t, test_sa.t;
ERROR: permission denied for relation(s): public.t, test_sa.t
postgres=> select * from t;
ERROR: permission denied for relation(s): public.t
postgres=> select * from test_sa.t;
ERROR: permission denied for relation(s): test_sa.t
postgres=> select * from rank;
ERROR: permission denied for relation(s): public."rank_1_prt_extra" (partition
of relation "rank"), public."rank_1_prt_2" (partition of relation "rank"),
public."rank_1_prt_3" (partition of relation "rank"), public."rank_1_prt_4"
(partition of relation "rank"), public."rank_1_prt_5" (partition of relation
"rank"), public."rank_1_prt_6" (partition of relation "rank"),
public."rank_1_prt_7" (partition of relation "rank"), public."rank_1_prt_8"
(partition of relation "rank”)
postgres=> select * from rank_1_prt_2;
ERROR: permission denied for relation(s): public."rank_1_prt_2" (partition of
relation "rank”)
{code}
> Print out detailed schema information for tables which the user doesn't have
> privileges
> ---
>
> Key: HAWQ-1365
> URL: https://issues.apache.org/jira/browse/HAWQ-1365
> Project: Apache HAWQ
> Issue Type: Improvement
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: 2.2.0.0-incubating
>
>
> Business Value:
> Current output information for the table name which user doesn't have
> privileges doesn't include schema output. We should print out the schema
> information out, otherwise it's difficult for users to identify the
> identified table.
> {code}
> postgres=# select * from public.a, s1.a;
> ERROR: permission denied for relation(s): a, a
> {code}
> Should print out the schema information for the table.
> {code}
> postgres=# select * from public.a, s1.a;
> ERROR: permission denied for relation(s): public.a, s1.a
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HAWQ-1365) Print out detailed schema information for tables which the user doesn't have privileges
[
https://issues.apache.org/jira/browse/HAWQ-1365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongxu Ma updated HAWQ-1365:
Description:
Business Value:
Current output information for the table name which user doesn't have
privileges doesn't include schema output. We should print out the schema
information out, otherwise it's difficult for users to identify the identified
table.
{code}
postgres=# select * from public.a, s1.a;
ERROR: permission denied for relation(s): a, a
{code}
Should print out the schema information for the table.
{code}
postgres=# select * from public.a, s1.a;
ERROR: permission denied for relation(s): public.a, s1.a
{code}
was:
Business Value:
Current output information for the table name which user doesn't have
privileges doesn't include schema output. We should print out the schema
information out, otherwise it's difficult for users to identify the identified
table.
{code}
postgres=# select * from public.a, s1.a;
ERROR: permission denied for relation(s): a, a
{code}
Criteria:
Should print out the schema information for the table.
{code}
postgres=# select * from public.a, s1.a;
ERROR: permission denied for relation(s): public.a, s1.a
{code}
> Print out detailed schema information for tables which the user doesn't have
> privileges
> ---
>
> Key: HAWQ-1365
> URL: https://issues.apache.org/jira/browse/HAWQ-1365
> Project: Apache HAWQ
> Issue Type: Improvement
>Reporter: Hongxu Ma
>Assignee: Ed Espino
> Fix For: 2.2.0.0-incubating
>
>
> Business Value:
> Current output information for the table name which user doesn't have
> privileges doesn't include schema output. We should print out the schema
> information out, otherwise it's difficult for users to identify the
> identified table.
> {code}
> postgres=# select * from public.a, s1.a;
> ERROR: permission denied for relation(s): a, a
> {code}
> Should print out the schema information for the table.
> {code}
> postgres=# select * from public.a, s1.a;
> ERROR: permission denied for relation(s): public.a, s1.a
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HAWQ-1365) Print out detailed schema information for tables which the user doesn't have privileges
[
https://issues.apache.org/jira/browse/HAWQ-1365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongxu Ma reassigned HAWQ-1365:
---
Assignee: Hongxu Ma (was: Ed Espino)
> Print out detailed schema information for tables which the user doesn't have
> privileges
> ---
>
> Key: HAWQ-1365
> URL: https://issues.apache.org/jira/browse/HAWQ-1365
> Project: Apache HAWQ
> Issue Type: Improvement
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: 2.2.0.0-incubating
>
>
> Business Value:
> Current output information for the table name which user doesn't have
> privileges doesn't include schema output. We should print out the schema
> information out, otherwise it's difficult for users to identify the
> identified table.
> {code}
> postgres=# select * from public.a, s1.a;
> ERROR: permission denied for relation(s): a, a
> {code}
> Should print out the schema information for the table.
> {code}
> postgres=# select * from public.a, s1.a;
> ERROR: permission denied for relation(s): public.a, s1.a
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HAWQ-1365) Print out detailed schema information for tables which the user doesn't have privileges
Hongxu Ma created HAWQ-1365:
---
Summary: Print out detailed schema information for tables which
the user doesn't have privileges
Key: HAWQ-1365
URL: https://issues.apache.org/jira/browse/HAWQ-1365
Project: Apache HAWQ
Issue Type: Improvement
Reporter: Hongxu Ma
Assignee: Ed Espino
Fix For: 2.2.0.0-incubating
Business Value:
Current output information for the table name which user doesn't have
privileges doesn't include schema output. We should print out the schema
information out, otherwise it's difficult for users to identify the identified
table.
{code}
postgres=# select * from public.a, s1.a;
ERROR: permission denied for relation(s): a, a
{code}
Criteria:
Should print out the schema information for the table.
{code}
postgres=# select * from public.a, s1.a;
ERROR: permission denied for relation(s): public.a, s1.a
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (HAWQ-1286) Reduce unnecessary calls of namespace check when run \d
[
https://issues.apache.org/jira/browse/HAWQ-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongxu Ma closed HAWQ-1286.
---
Resolution: Fixed
> Reduce unnecessary calls of namespace check when run \d
> ---
>
> Key: HAWQ-1286
> URL: https://issues.apache.org/jira/browse/HAWQ-1286
> Project: Apache HAWQ
> Issue Type: Sub-task
> Components: Security
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: backlog
>
>
> After HAWQ-1279 is done, current schema is no cached in current session.
> But it cause too many calls of namespace check to send in run \d , most of
> them are unnecessary (e.g. repeat check usage right of public schema).
> So we should reduce those unnecessary calls.
> \d case
> {code}
> \d:
> select version()
> SELECT n.nspname as \"Schema\",\n c.relname as \"Name\",\n CASE
> c.relkind WHEN 'r' THEN 'table' WHEN 'v' THEN 'view' WHEN 'i' THEN 'index'
> WHEN 'S' THEN 'sequence' WHEN 's' THEN 'special' END as \"Type\",\n
> pg_catalog.pg_get_userbyid(c.relowner) as \"Owner\"\nFROM pg_catalog.pg_class
> c\n LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace\nWHERE
> c.relkind IN ('r','v','S','')\n AND n.nspname <> 'pg_catalog'\n AND
> n.nspname <> 'information_schema'\n AND n.nspname !~ '^pg_toast'\n AND
> pg_catalog.pg_table_is_visible(c.oid)\nORDER BY 1,2;
> recomputeNamespacePath()
> recomputeNamespacePath()
> recompute many times in this long select sql
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HAWQ-1286) Reduce unnecessary calls of namespace check when run \d
[
https://issues.apache.org/jira/browse/HAWQ-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831323#comment-15831323
]
Hongxu Ma commented on HAWQ-1286:
-
Solution:
Store the last sql query in recomputeNamespacePath(), only recompute if the
query is changed.
> Reduce unnecessary calls of namespace check when run \d
> ---
>
> Key: HAWQ-1286
> URL: https://issues.apache.org/jira/browse/HAWQ-1286
> Project: Apache HAWQ
> Issue Type: Sub-task
> Components: Security
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: backlog
>
>
> After HAWQ-1279 is done, current schema is no cached in current session.
> But it cause too many calls of namespace check to send in run \d , most of
> them are unnecessary (e.g. repeat check usage right of public schema).
> So we should reduce those unnecessary calls.
> \d case
> {code}
> \d:
> select version()
> SELECT n.nspname as \"Schema\",\n c.relname as \"Name\",\n CASE
> c.relkind WHEN 'r' THEN 'table' WHEN 'v' THEN 'view' WHEN 'i' THEN 'index'
> WHEN 'S' THEN 'sequence' WHEN 's' THEN 'special' END as \"Type\",\n
> pg_catalog.pg_get_userbyid(c.relowner) as \"Owner\"\nFROM pg_catalog.pg_class
> c\n LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace\nWHERE
> c.relkind IN ('r','v','S','')\n AND n.nspname <> 'pg_catalog'\n AND
> n.nspname <> 'information_schema'\n AND n.nspname !~ '^pg_toast'\n AND
> pg_catalog.pg_table_is_visible(c.oid)\nORDER BY 1,2;
> recomputeNamespacePath()
> recomputeNamespacePath()
> recompute many times in this long select sql
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HAWQ-1286) Reduce unnecessary calls of namespace check when run \d
[
https://issues.apache.org/jira/browse/HAWQ-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongxu Ma updated HAWQ-1286:
Description:
After HAWQ-1279 is done, current schema is no cached in current session.
But it cause too many calls of namespace check to send in run \d , most of them
are unnecessary (e.g. repeat check usage right of public schema).
So we should reduce those unnecessary calls.
\d case
{code}
\d:
select version()
SELECT n.nspname as \"Schema\",\n c.relname as \"Name\",\n CASE c.relkind
WHEN 'r' THEN 'table' WHEN 'v' THEN 'view' WHEN 'i' THEN 'index' WHEN 'S' THEN
'sequence' WHEN 's' THEN 'special' END as \"Type\",\n
pg_catalog.pg_get_userbyid(c.relowner) as \"Owner\"\nFROM pg_catalog.pg_class
c\n LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace\nWHERE
c.relkind IN ('r','v','S','')\n AND n.nspname <> 'pg_catalog'\n AND
n.nspname <> 'information_schema'\n AND n.nspname !~ '^pg_toast'\n AND
pg_catalog.pg_table_is_visible(c.oid)\nORDER BY 1,2;
recomputeNamespacePath()
recomputeNamespacePath()
recompute many times in this long select sql
{code}
was:
After HAWQ-1279 is done, current schema is no cached in current session.
But it cause too many calls of namespace check to send in run \d , most of them
are unnecessary (e.g. repeat check usage right of public schema).
So we should reduce those unnecessary calls.
> Reduce unnecessary calls of namespace check when run \d
> ---
>
> Key: HAWQ-1286
> URL: https://issues.apache.org/jira/browse/HAWQ-1286
> Project: Apache HAWQ
> Issue Type: Sub-task
> Components: Security
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: backlog
>
>
> After HAWQ-1279 is done, current schema is no cached in current session.
> But it cause too many calls of namespace check to send in run \d , most of
> them are unnecessary (e.g. repeat check usage right of public schema).
> So we should reduce those unnecessary calls.
> \d case
> {code}
> \d:
> select version()
> SELECT n.nspname as \"Schema\",\n c.relname as \"Name\",\n CASE
> c.relkind WHEN 'r' THEN 'table' WHEN 'v' THEN 'view' WHEN 'i' THEN 'index'
> WHEN 'S' THEN 'sequence' WHEN 's' THEN 'special' END as \"Type\",\n
> pg_catalog.pg_get_userbyid(c.relowner) as \"Owner\"\nFROM pg_catalog.pg_class
> c\n LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace\nWHERE
> c.relkind IN ('r','v','S','')\n AND n.nspname <> 'pg_catalog'\n AND
> n.nspname <> 'information_schema'\n AND n.nspname !~ '^pg_toast'\n AND
> pg_catalog.pg_table_is_visible(c.oid)\nORDER BY 1,2;
> recomputeNamespacePath()
> recomputeNamespacePath()
> recompute many times in this long select sql
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HAWQ-1286) Reduce unnecessary calls of namespace check when run \d
[ https://issues.apache.org/jira/browse/HAWQ-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma updated HAWQ-1286: Summary: Reduce unnecessary calls of namespace check when run \d (was: Reduce all unnecessary calls of namespace check when run \d) > Reduce unnecessary calls of namespace check when run \d > --- > > Key: HAWQ-1286 > URL: https://issues.apache.org/jira/browse/HAWQ-1286 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Hongxu Ma >Assignee: Hongxu Ma > Fix For: backlog > > > After HAWQ-1279 is done, current schema is no cached in current session. > But it cause too many calls of namespace check to send in run \d , most of > them are unnecessary (e.g. repeat check usage right of public schema). > So we should reduce those unnecessary calls. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1286) Reduce all unnecessary calls of namespace check when run \d
[ https://issues.apache.org/jira/browse/HAWQ-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma updated HAWQ-1286: Description: After HAWQ-1279 is done, current schema is no cached in current session. But it cause too many calls of namespace check to send in run \d , most of them are unnecessary (e.g. repeat check usage right of public schema). So we should reduce those unnecessary calls. was: After HAWQ-1279 is done, current schema is no cached in current session. But it cause too many calls of namespace check to send in run \d , most of them are unnecessary (e.g. repeat check usage right of public schema). So we should reduct those unnecessary calls. > Reduce all unnecessary calls of namespace check when run \d > --- > > Key: HAWQ-1286 > URL: https://issues.apache.org/jira/browse/HAWQ-1286 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Hongxu Ma >Assignee: Hongxu Ma > Fix For: backlog > > > After HAWQ-1279 is done, current schema is no cached in current session. > But it cause too many calls of namespace check to send in run \d , most of > them are unnecessary (e.g. repeat check usage right of public schema). > So we should reduce those unnecessary calls. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HAWQ-1286) Reduce all unnecessary calls of namespace check when run \d
Hongxu Ma created HAWQ-1286: --- Summary: Reduce all unnecessary calls of namespace check when run \d Key: HAWQ-1286 URL: https://issues.apache.org/jira/browse/HAWQ-1286 Project: Apache HAWQ Issue Type: Sub-task Reporter: Hongxu Ma Assignee: Ed Espino After HAWQ-1279 is done, current schema is no cached in current session. But it cause too many calls of namespace check to send in run \d , most of them are unnecessary (e.g. repeat check usage right of public schema). So we should reduct those unnecessary calls. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HAWQ-1286) Reduce all unnecessary calls of namespace check when run \d
[ https://issues.apache.org/jira/browse/HAWQ-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma reassigned HAWQ-1286: --- Assignee: Hongxu Ma (was: Ed Espino) > Reduce all unnecessary calls of namespace check when run \d > --- > > Key: HAWQ-1286 > URL: https://issues.apache.org/jira/browse/HAWQ-1286 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Hongxu Ma >Assignee: Hongxu Ma > Fix For: backlog > > > After HAWQ-1279 is done, current schema is no cached in current session. > But it cause too many calls of namespace check to send in run \d , most of > them are unnecessary (e.g. repeat check usage right of public schema). > So we should reduct those unnecessary calls. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1193) TDE support in HAWQ
[ https://issues.apache.org/jira/browse/HAWQ-1193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma updated HAWQ-1193: Assignee: Ivan Weng (was: Hongxu Ma) > TDE support in HAWQ > --- > > Key: HAWQ-1193 > URL: https://issues.apache.org/jira/browse/HAWQ-1193 > Project: Apache HAWQ > Issue Type: New Feature > Components: libhdfs >Reporter: Hongxu Ma >Assignee: Ivan Weng > Fix For: backlog > > Attachments: HAWQ_TDE_Design_ver0.1.pdf, HAWQ_TDE_Design_ver0.2 .pdf > > > TDE(transparently data encrypted) has been supported after hadoop 2.6: > http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html > https://issues.apache.org/jira/browse/HDFS-6134 > Use TDE can promise： > 1, hdfs file is encrypted. > 2, network transfer between hdfs and libhdfs client is encrypted. > So hawq will update libhdfs3 to support it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1194) Add EncryptionZones related RPC
[ https://issues.apache.org/jira/browse/HAWQ-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma updated HAWQ-1194: Assignee: Amy (was: Hongxu Ma) > Add EncryptionZones related RPC > --- > > Key: HAWQ-1194 > URL: https://issues.apache.org/jira/browse/HAWQ-1194 > Project: Apache HAWQ > Issue Type: Sub-task > Components: libhdfs >Reporter: Hongxu Ma >Assignee: Amy > Fix For: backlog > > > Add createEncryption, getEZForPath, listEncryptionZones RPC -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1194) Add EncryptionZones related RPC
[ https://issues.apache.org/jira/browse/HAWQ-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma updated HAWQ-1194: Description: Add createEncryption, getEZForPath, listEncryptionZones RPC > Add EncryptionZones related RPC > --- > > Key: HAWQ-1194 > URL: https://issues.apache.org/jira/browse/HAWQ-1194 > Project: Apache HAWQ > Issue Type: Sub-task > Components: libhdfs >Reporter: Hongxu Ma >Assignee: Hongxu Ma > Fix For: backlog > > > Add createEncryption, getEZForPath, listEncryptionZones RPC -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1194) Add EncryptionZones related RPC
[ https://issues.apache.org/jira/browse/HAWQ-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma updated HAWQ-1194: Summary: Add EncryptionZones related RPC (was: add EncryptionZones related RPC) > Add EncryptionZones related RPC > --- > > Key: HAWQ-1194 > URL: https://issues.apache.org/jira/browse/HAWQ-1194 > Project: Apache HAWQ > Issue Type: Sub-task > Components: libhdfs >Reporter: Hongxu Ma >Assignee: Hongxu Ma > Fix For: backlog > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1194) add EncryptionZones related RPC
[ https://issues.apache.org/jira/browse/HAWQ-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma updated HAWQ-1194: Summary: add EncryptionZones related RPC (was: Todo) > add EncryptionZones related RPC > --- > > Key: HAWQ-1194 > URL: https://issues.apache.org/jira/browse/HAWQ-1194 > Project: Apache HAWQ > Issue Type: Sub-task > Components: libhdfs >Reporter: Hongxu Ma >Assignee: Hongxu Ma > Fix For: backlog > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (HAWQ-1279) Force to recompute namespace_path when enable_ranger
[
https://issues.apache.org/jira/browse/HAWQ-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongxu Ma closed HAWQ-1279.
---
Resolution: Fixed
fixed
> Force to recompute namespace_path when enable_ranger
>
>
> Key: HAWQ-1279
> URL: https://issues.apache.org/jira/browse/HAWQ-1279
> Project: Apache HAWQ
> Issue Type: Sub-task
> Components: PXF, Security
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: backlog
>
>
> namespace_path is cached in each psql session and the cache invalidation is
> triggered by Grant/Revoke SQL.
> When enable_ranger, Grant/Revoke SQL is no longer use, so the cache prevent a
> ack-check request sending.
> Example:
> {code}
> // create table t(i int); => failed
> [{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
> [{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
> [{""resource"":{""database"":""postgres"",""schema"":""public""},""privileges"":[""usage""],""allowed"":false}]
> // grant usage and create permissions to public schema in ranger and try
> again => failed again
> [{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
> [{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
> // why not send a request for USAGE??
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HAWQ-1279) Force to recompute namespace_path when enable_ranger
[
https://issues.apache.org/jira/browse/HAWQ-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongxu Ma updated HAWQ-1279:
Description:
namespace_path is cached in each psql session and the cache invalidation is
triggered by Grant/Revoke SQL.
When enable_ranger, Grant/Revoke SQL is no longer use, so the cache prevent a
ack-check request sending.
Example:
{code}
// create table t(i int); => failed
[{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
[{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
[{""resource"":{""database"":""postgres"",""schema"":""public""},""privileges"":[""usage""],""allowed"":false}]
// grant usage and create permissions to public schema in ranger and try again
=> failed again
[{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
[{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
// why not send a request for USAGE??
{code}
was:
namespace_path is cached in each psql session and it the cache invalidation is
triggered by Grant/Revoke SQL.
When enable_ranger, Grant/Revoke SQL is no longer use, so the cache prevent a
ack-check request sending.
Example:
{code}
// create table t(i int); => failed
[{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
[{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
[{""resource"":{""database"":""postgres"",""schema"":""public""},""privileges"":[""usage""],""allowed"":false}]
// grant usage and create permissions to public schema in ranger and try again
=> failed again
[{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
[{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
// why not send a request for USAGE??
{code}
> Force to recompute namespace_path when enable_ranger
>
>
> Key: HAWQ-1279
> URL: https://issues.apache.org/jira/browse/HAWQ-1279
> Project: Apache HAWQ
> Issue Type: Sub-task
> Components: PXF, Security
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: backlog
>
>
> namespace_path is cached in each psql session and the cache invalidation is
> triggered by Grant/Revoke SQL.
> When enable_ranger, Grant/Revoke SQL is no longer use, so the cache prevent a
> ack-check request sending.
> Example:
> {code}
> // create table t(i int); => failed
> [{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
> [{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
> [{""resource"":{""database"":""postgres"",""schema"":""public""},""privileges"":[""usage""],""allowed"":false}]
> // grant usage and create permissions to public schema in ranger and try
> again => failed again
> [{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
> [{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
> // why not send a request for USAGE??
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HAWQ-1279) Force to recompute namespace_path when enable_ranger
[
https://issues.apache.org/jira/browse/HAWQ-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongxu Ma updated HAWQ-1279:
Description:
namespace_path is cached in each psql session and it the cache invalidation is
triggered by Grant/Revoke SQL.
When enable_ranger, Grant/Revoke SQL is no longer use, so the cache prevent a
ack-check request sending.
Example:
{code}
// create table t(i int); => failed
[{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
[{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
[{""resource"":{""database"":""postgres"",""schema"":""public""},""privileges"":[""usage""],""allowed"":false}]
// grant usage and create permissions to public schema in ranger and try again
=> failed again
[{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
[{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
// why not send a request for USAGE??
{code}
was:
namespace_path is cached in each psql session and it the cache invalidation is
triggered by Grant/Revoke SQL.
When enable_ranger, Grant/Revoke SQL is no longer use, so the cache prevent a
ack-check request sending.
Example:
```
// create table t(i int); => failed
[{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
[{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
[{""resource"":{""database"":""postgres"",""schema"":""public""},""privileges"":[""usage""],""allowed"":false}]
// grant usage and create permissions to public schema in ranger and try again
=> failed again
[{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
[{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
// why not send a request for USAGE??
```
> Force to recompute namespace_path when enable_ranger
>
>
> Key: HAWQ-1279
> URL: https://issues.apache.org/jira/browse/HAWQ-1279
> Project: Apache HAWQ
> Issue Type: Sub-task
> Components: PXF, Security
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: backlog
>
>
> namespace_path is cached in each psql session and it the cache invalidation
> is triggered by Grant/Revoke SQL.
> When enable_ranger, Grant/Revoke SQL is no longer use, so the cache prevent a
> ack-check request sending.
> Example:
> {code}
> // create table t(i int); => failed
> [{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
> [{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
> [{""resource"":{""database"":""postgres"",""schema"":""public""},""privileges"":[""usage""],""allowed"":false}]
> // grant usage and create permissions to public schema in ranger and try
> again => failed again
> [{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
> [{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
> // why not send a request for USAGE??
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HAWQ-1279) Force to recompute namespace_path when enable_ranger
Hongxu Ma created HAWQ-1279:
---
Summary: Force to recompute namespace_path when enable_ranger
Key: HAWQ-1279
URL: https://issues.apache.org/jira/browse/HAWQ-1279
Project: Apache HAWQ
Issue Type: Sub-task
Reporter: Hongxu Ma
Assignee: Ed Espino
namespace_path is cached in each psql session and it the cache invalidation is
triggered by Grant/Revoke SQL.
When enable_ranger, Grant/Revoke SQL is no longer use, so the cache prevent a
ack-check request sending.
Example:
```
// create table t(i int); => failed
[{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
[{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
[{""resource"":{""database"":""postgres"",""schema"":""public""},""privileges"":[""usage""],""allowed"":false}]
// grant usage and create permissions to public schema in ranger and try again
=> failed again
[{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
[{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
// why not send a request for USAGE??
```
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HAWQ-1279) Force to recompute namespace_path when enable_ranger
[
https://issues.apache.org/jira/browse/HAWQ-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongxu Ma reassigned HAWQ-1279:
---
Assignee: Hongxu Ma (was: Ed Espino)
> Force to recompute namespace_path when enable_ranger
>
>
> Key: HAWQ-1279
> URL: https://issues.apache.org/jira/browse/HAWQ-1279
> Project: Apache HAWQ
> Issue Type: Sub-task
> Components: PXF, Security
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: backlog
>
>
> namespace_path is cached in each psql session and it the cache invalidation
> is triggered by Grant/Revoke SQL.
> When enable_ranger, Grant/Revoke SQL is no longer use, so the cache prevent a
> ack-check request sending.
> Example:
> ```
> // create table t(i int); => failed
> [{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
> [{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
> [{""resource"":{""database"":""postgres"",""schema"":""public""},""privileges"":[""usage""],""allowed"":false}]
> // grant usage and create permissions to public schema in ranger and try
> again => failed again
> [{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
> [{""resource"":{""database"":""postgres"",""schema"":""pg_catalog""},""privileges"":[""usage""],""allowed"":true}]
> // why not send a request for USAGE??
> ```
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (HAWQ-1257) If user doesn't have privileges on certain objects, need return user which specific table he doesn't have right.
[
https://issues.apache.org/jira/browse/HAWQ-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongxu Ma closed HAWQ-1257.
---
Resolution: Fixed
already opened a PR: https://github.com/apache/incubator-hawq/pull/1081
> If user doesn't have privileges on certain objects, need return user which
> specific table he doesn't have right.
> -
>
> Key: HAWQ-1257
> URL: https://issues.apache.org/jira/browse/HAWQ-1257
> Project: Apache HAWQ
> Issue Type: Sub-task
> Components: Security
>Reporter: Lili Ma
>Assignee: Hongxu Ma
> Fix For: 2.2.0.0-incubating
>
>
> If user doesn't have privileges on certain objects, need return user all the
> objects he doesn't have right, to avoid the user modify one privilege, and
> then find another privilege constraint, and then another... which may bother
> the user a lot.
> For example:
> user didn't have the rights of t1 and t2.
> {code}
> postgres=> select * from test_sa.t1 left join test_sa.t2 on
> test_sa.t1.i=test_sa.t2.i;
> ERROR: permission denied for relation t1
> {code}
> We wish to prompt user didn't have the rights of t2 also.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HAWQ-1226) HAWQ core dump due to enable ranger while RPS is down
[
https://issues.apache.org/jira/browse/HAWQ-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongxu Ma updated HAWQ-1226:
Summary: HAWQ core dump due to enable ranger while RPS is down (was: HAWQ
core dump due to enable ranger while RPS is down. )
> HAWQ core dump due to enable ranger while RPS is down
> --
>
> Key: HAWQ-1226
> URL: https://issues.apache.org/jira/browse/HAWQ-1226
> Project: Apache HAWQ
> Issue Type: Bug
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
>
> Core dump when starting hawq without starting RPS with enable_ranger is true.
> {code}
> (lldb) bt
> * thread #1: tid = 0x, 0x7fffe9264dda
> libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
> * frame #0: 0x7fffe9264dda libsystem_kernel.dylib`__pthread_kill + 10
> frame #1: 0x7fffe9350787 libsystem_pthread.dylib`pthread_kill + 90
> frame #2: 0x7fffe917b497 libsystem_c.dylib`raise + 26
> frame #3: 0x00010cbdd93f
> postgres`SafeHandlerForSegvBusIll(processName="Process",
> postgres_signal_arg=11) + 591 at elog.c:4519
> frame #4: 0x00010cbdd6cb
> postgres`StandardHandlerForSigillSigsegvSigbus_OnMainThread(processName="Process",
> postgres_signal_arg=11) + 43 at elog.c:4597
> frame #5: 0x00010cab66ef
> postgres`CdbProgramErrorHandler(postgres_signal_arg=11) + 319 at
> postgres.c:3512
> frame #6: 0x7fffe9343bba libsystem_platform.dylib`_sigtramp + 26
> frame #7: 0x7fffe916cb53 libsystem_c.dylib`strlen + 19
> frame #8: 0x00010c951215
> postgres`parse_ranger_response(buffer=0x) + 21 at
> rangerrest.c:59
> frame #9: 0x00010c951c8e
> postgres`check_privilege_from_ranger(user="wuhong", kind=ACL_KIND_NAMESPACE,
> object="template1", actions=0x7ff338036598, how=0x) + 254
> at rangerrest.c:339
> frame #10: 0x00010c7890c4
> postgres`pg_rangercheck(objkind=ACL_KIND_NAMESPACE, object_oid=2200,
> roleid=10, mask=256, how=ACLMASK_ANY) + 164 at aclchk.c:2678
> frame #11: 0x00010c78b085
> postgres`pg_namespace_aclcheck(nsp_oid=2200, roleid=10, mode=256) + 101 at
> aclchk.c:3752
> frame #12: 0x00010c77ea48 postgres`recomputeNamespacePath + 600 at
> namespace.c:2010
> frame #13: 0x00010c77dec1
> postgres`RelnameGetRelid(relname="gp_segment_configuration") + 17 at
> namespace.c:423
> frame #14: 0x00010c77db61
> postgres`RangeVarGetRelid(relation=0x7ff3388cef60, failOK='\0',
> allowHcatalog='\x01') + 449 at namespace.c:278
> frame #15: 0x00010c671f6e
> postgres`CdbOpenRelationRv(relation=0x7ff3388cef60, reqmode=3,
> noWait='\0', lockUpgraded=0x) + 46 at heapam.c:1075
> frame #16: 0x00010c7f718c
> postgres`setTargetTable(pstate=0x7ff3388cf460,
> relation=0x7ff3388cef60, inh='\x01', alsoSource='\x01', requiredPerms=8)
> + 140 at parse_clause.c:866
> frame #17: 0x00010c7b9a6e
> postgres`transformDeleteStmt(pstate=0x7ff3388cf460,
> stmt=0x7ff3388cf3c0) + 174 at analyze.c:943
> frame #18: 0x00010c7b5249
> postgres`transformStmt(pstate=0x7ff3388cf460,
> parseTree=0x7ff3388cf3c0, extras_before=0x7fff535bcba0,
> extras_after=0x7fff535bcb98) + 841 at analyze.c:769
> frame #19: 0x00010c7a5b60
> postgres`do_parse_analyze(parseTree=0x7ff3388cf3c0,
> pstate=0x7ff3388cf460) + 112 at analyze.c:497
> frame #20: 0x00010c7a5acf
> postgres`parse_analyze(parseTree=0x7ff3388cf3c0, sourceText="DELETE FROM
> gp_segment_configuration WHERE role = 'p' or role = 'm'",
> paramTypes=0x, numParams=0) + 95 at analyze.c:351
> frame #21: 0x00010cab5960
> postgres`pg_analyze_and_rewrite(parsetree=0x7ff3388cf3c0,
> query_string="DELETE FROM gp_segment_configuration WHERE role = 'p' or role =
> 'm'", paramTypes=0x, numParams=0) + 64 at postgres.c:812
> frame #22: 0x00010caba448
> postgres`exec_simple_query(query_string="DELETE FROM gp_segment_configuration
> WHERE role = 'p' or role = 'm'", seqServerHost=0x,
> seqServerPort=-1) + 1416 at postgres.c:1739
> frame #23: 0x00010cab8b15 postgres`PostgresMain(argc=8,
> argv=0x7ff338818698, username="wuhong") + 7189 at postgres.c:4840
> frame #24: 0x00010ca62295
> postgres`BackendRun(port=0x7ff337416a60) + 981 at postmaster.c:5915
> frame #25: 0x00010ca5f6e5
> postgres`BackendStartup(port=0x7ff337416a60) + 373 at postmaster.c:5484
> frame #26: 0x00010ca5cbb0 postgres`ServerLoop + 1248 at
> postmaster.c:2163
> frame #27: 0x00010ca5b2e3 postgres`PostmasterMain(argc=9,
> argv=0x7ff337415af0) + 4835 at postmaster.c:1454
> frame #28: 0x00010c96742c postgres`main(argc=9,
> argv=0x7ff337415af0)
[jira] [Updated] (HAWQ-1226) HAWQ core dump due to enable ranger while RPS is down.
[
https://issues.apache.org/jira/browse/HAWQ-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongxu Ma updated HAWQ-1226:
Summary: HAWQ core dump due to enable ranger while RPS is down. (was:
HAWQ core dump due to Enable ranger while RPS is down. )
> HAWQ core dump due to enable ranger while RPS is down.
> ---
>
> Key: HAWQ-1226
> URL: https://issues.apache.org/jira/browse/HAWQ-1226
> Project: Apache HAWQ
> Issue Type: Bug
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
>
> Core dump when starting hawq without starting RPS with enable_ranger is true.
> {code}
> (lldb) bt
> * thread #1: tid = 0x, 0x7fffe9264dda
> libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
> * frame #0: 0x7fffe9264dda libsystem_kernel.dylib`__pthread_kill + 10
> frame #1: 0x7fffe9350787 libsystem_pthread.dylib`pthread_kill + 90
> frame #2: 0x7fffe917b497 libsystem_c.dylib`raise + 26
> frame #3: 0x00010cbdd93f
> postgres`SafeHandlerForSegvBusIll(processName="Process",
> postgres_signal_arg=11) + 591 at elog.c:4519
> frame #4: 0x00010cbdd6cb
> postgres`StandardHandlerForSigillSigsegvSigbus_OnMainThread(processName="Process",
> postgres_signal_arg=11) + 43 at elog.c:4597
> frame #5: 0x00010cab66ef
> postgres`CdbProgramErrorHandler(postgres_signal_arg=11) + 319 at
> postgres.c:3512
> frame #6: 0x7fffe9343bba libsystem_platform.dylib`_sigtramp + 26
> frame #7: 0x7fffe916cb53 libsystem_c.dylib`strlen + 19
> frame #8: 0x00010c951215
> postgres`parse_ranger_response(buffer=0x) + 21 at
> rangerrest.c:59
> frame #9: 0x00010c951c8e
> postgres`check_privilege_from_ranger(user="wuhong", kind=ACL_KIND_NAMESPACE,
> object="template1", actions=0x7ff338036598, how=0x) + 254
> at rangerrest.c:339
> frame #10: 0x00010c7890c4
> postgres`pg_rangercheck(objkind=ACL_KIND_NAMESPACE, object_oid=2200,
> roleid=10, mask=256, how=ACLMASK_ANY) + 164 at aclchk.c:2678
> frame #11: 0x00010c78b085
> postgres`pg_namespace_aclcheck(nsp_oid=2200, roleid=10, mode=256) + 101 at
> aclchk.c:3752
> frame #12: 0x00010c77ea48 postgres`recomputeNamespacePath + 600 at
> namespace.c:2010
> frame #13: 0x00010c77dec1
> postgres`RelnameGetRelid(relname="gp_segment_configuration") + 17 at
> namespace.c:423
> frame #14: 0x00010c77db61
> postgres`RangeVarGetRelid(relation=0x7ff3388cef60, failOK='\0',
> allowHcatalog='\x01') + 449 at namespace.c:278
> frame #15: 0x00010c671f6e
> postgres`CdbOpenRelationRv(relation=0x7ff3388cef60, reqmode=3,
> noWait='\0', lockUpgraded=0x) + 46 at heapam.c:1075
> frame #16: 0x00010c7f718c
> postgres`setTargetTable(pstate=0x7ff3388cf460,
> relation=0x7ff3388cef60, inh='\x01', alsoSource='\x01', requiredPerms=8)
> + 140 at parse_clause.c:866
> frame #17: 0x00010c7b9a6e
> postgres`transformDeleteStmt(pstate=0x7ff3388cf460,
> stmt=0x7ff3388cf3c0) + 174 at analyze.c:943
> frame #18: 0x00010c7b5249
> postgres`transformStmt(pstate=0x7ff3388cf460,
> parseTree=0x7ff3388cf3c0, extras_before=0x7fff535bcba0,
> extras_after=0x7fff535bcb98) + 841 at analyze.c:769
> frame #19: 0x00010c7a5b60
> postgres`do_parse_analyze(parseTree=0x7ff3388cf3c0,
> pstate=0x7ff3388cf460) + 112 at analyze.c:497
> frame #20: 0x00010c7a5acf
> postgres`parse_analyze(parseTree=0x7ff3388cf3c0, sourceText="DELETE FROM
> gp_segment_configuration WHERE role = 'p' or role = 'm'",
> paramTypes=0x, numParams=0) + 95 at analyze.c:351
> frame #21: 0x00010cab5960
> postgres`pg_analyze_and_rewrite(parsetree=0x7ff3388cf3c0,
> query_string="DELETE FROM gp_segment_configuration WHERE role = 'p' or role =
> 'm'", paramTypes=0x, numParams=0) + 64 at postgres.c:812
> frame #22: 0x00010caba448
> postgres`exec_simple_query(query_string="DELETE FROM gp_segment_configuration
> WHERE role = 'p' or role = 'm'", seqServerHost=0x,
> seqServerPort=-1) + 1416 at postgres.c:1739
> frame #23: 0x00010cab8b15 postgres`PostgresMain(argc=8,
> argv=0x7ff338818698, username="wuhong") + 7189 at postgres.c:4840
> frame #24: 0x00010ca62295
> postgres`BackendRun(port=0x7ff337416a60) + 981 at postmaster.c:5915
> frame #25: 0x00010ca5f6e5
> postgres`BackendStartup(port=0x7ff337416a60) + 373 at postmaster.c:5484
> frame #26: 0x00010ca5cbb0 postgres`ServerLoop + 1248 at
> postmaster.c:2163
> frame #27: 0x00010ca5b2e3 postgres`PostmasterMain(argc=9,
> argv=0x7ff337415af0) + 4835 at postmaster.c:1454
> frame #28: 0x00010c96742c postgres`main(argc=9,
>
[jira] [Updated] (HAWQ-1226) HAWQ core dump due to Enable ranger while RPS is down.
[
https://issues.apache.org/jira/browse/HAWQ-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongxu Ma updated HAWQ-1226:
Summary: HAWQ core dump due to Enable ranger while RPS is down. (was:
Enable ranger while RPS is down cause HAWQ to core dump)
> HAWQ core dump due to Enable ranger while RPS is down.
> ---
>
> Key: HAWQ-1226
> URL: https://issues.apache.org/jira/browse/HAWQ-1226
> Project: Apache HAWQ
> Issue Type: Bug
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
>
> Core dump when starting hawq without starting RPS with enable_ranger is true.
> {code}
> (lldb) bt
> * thread #1: tid = 0x, 0x7fffe9264dda
> libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
> * frame #0: 0x7fffe9264dda libsystem_kernel.dylib`__pthread_kill + 10
> frame #1: 0x7fffe9350787 libsystem_pthread.dylib`pthread_kill + 90
> frame #2: 0x7fffe917b497 libsystem_c.dylib`raise + 26
> frame #3: 0x00010cbdd93f
> postgres`SafeHandlerForSegvBusIll(processName="Process",
> postgres_signal_arg=11) + 591 at elog.c:4519
> frame #4: 0x00010cbdd6cb
> postgres`StandardHandlerForSigillSigsegvSigbus_OnMainThread(processName="Process",
> postgres_signal_arg=11) + 43 at elog.c:4597
> frame #5: 0x00010cab66ef
> postgres`CdbProgramErrorHandler(postgres_signal_arg=11) + 319 at
> postgres.c:3512
> frame #6: 0x7fffe9343bba libsystem_platform.dylib`_sigtramp + 26
> frame #7: 0x7fffe916cb53 libsystem_c.dylib`strlen + 19
> frame #8: 0x00010c951215
> postgres`parse_ranger_response(buffer=0x) + 21 at
> rangerrest.c:59
> frame #9: 0x00010c951c8e
> postgres`check_privilege_from_ranger(user="wuhong", kind=ACL_KIND_NAMESPACE,
> object="template1", actions=0x7ff338036598, how=0x) + 254
> at rangerrest.c:339
> frame #10: 0x00010c7890c4
> postgres`pg_rangercheck(objkind=ACL_KIND_NAMESPACE, object_oid=2200,
> roleid=10, mask=256, how=ACLMASK_ANY) + 164 at aclchk.c:2678
> frame #11: 0x00010c78b085
> postgres`pg_namespace_aclcheck(nsp_oid=2200, roleid=10, mode=256) + 101 at
> aclchk.c:3752
> frame #12: 0x00010c77ea48 postgres`recomputeNamespacePath + 600 at
> namespace.c:2010
> frame #13: 0x00010c77dec1
> postgres`RelnameGetRelid(relname="gp_segment_configuration") + 17 at
> namespace.c:423
> frame #14: 0x00010c77db61
> postgres`RangeVarGetRelid(relation=0x7ff3388cef60, failOK='\0',
> allowHcatalog='\x01') + 449 at namespace.c:278
> frame #15: 0x00010c671f6e
> postgres`CdbOpenRelationRv(relation=0x7ff3388cef60, reqmode=3,
> noWait='\0', lockUpgraded=0x) + 46 at heapam.c:1075
> frame #16: 0x00010c7f718c
> postgres`setTargetTable(pstate=0x7ff3388cf460,
> relation=0x7ff3388cef60, inh='\x01', alsoSource='\x01', requiredPerms=8)
> + 140 at parse_clause.c:866
> frame #17: 0x00010c7b9a6e
> postgres`transformDeleteStmt(pstate=0x7ff3388cf460,
> stmt=0x7ff3388cf3c0) + 174 at analyze.c:943
> frame #18: 0x00010c7b5249
> postgres`transformStmt(pstate=0x7ff3388cf460,
> parseTree=0x7ff3388cf3c0, extras_before=0x7fff535bcba0,
> extras_after=0x7fff535bcb98) + 841 at analyze.c:769
> frame #19: 0x00010c7a5b60
> postgres`do_parse_analyze(parseTree=0x7ff3388cf3c0,
> pstate=0x7ff3388cf460) + 112 at analyze.c:497
> frame #20: 0x00010c7a5acf
> postgres`parse_analyze(parseTree=0x7ff3388cf3c0, sourceText="DELETE FROM
> gp_segment_configuration WHERE role = 'p' or role = 'm'",
> paramTypes=0x, numParams=0) + 95 at analyze.c:351
> frame #21: 0x00010cab5960
> postgres`pg_analyze_and_rewrite(parsetree=0x7ff3388cf3c0,
> query_string="DELETE FROM gp_segment_configuration WHERE role = 'p' or role =
> 'm'", paramTypes=0x, numParams=0) + 64 at postgres.c:812
> frame #22: 0x00010caba448
> postgres`exec_simple_query(query_string="DELETE FROM gp_segment_configuration
> WHERE role = 'p' or role = 'm'", seqServerHost=0x,
> seqServerPort=-1) + 1416 at postgres.c:1739
> frame #23: 0x00010cab8b15 postgres`PostgresMain(argc=8,
> argv=0x7ff338818698, username="wuhong") + 7189 at postgres.c:4840
> frame #24: 0x00010ca62295
> postgres`BackendRun(port=0x7ff337416a60) + 981 at postmaster.c:5915
> frame #25: 0x00010ca5f6e5
> postgres`BackendStartup(port=0x7ff337416a60) + 373 at postmaster.c:5484
> frame #26: 0x00010ca5cbb0 postgres`ServerLoop + 1248 at
> postmaster.c:2163
> frame #27: 0x00010ca5b2e3 postgres`PostmasterMain(argc=9,
> argv=0x7ff337415af0) + 4835 at postmaster.c:1454
> frame #28: 0x00010c96742c postgres`main(argc=9,
>
[jira] [Updated] (HAWQ-1226) Enable ranger while RPS is down will cause HAWQ to core dump
[
https://issues.apache.org/jira/browse/HAWQ-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongxu Ma updated HAWQ-1226:
Summary: Enable ranger while RPS is down will cause HAWQ to core dump
(was: Enable ranger while RPS is down will cause HAWQ core dump)
> Enable ranger while RPS is down will cause HAWQ to core dump
>
>
> Key: HAWQ-1226
> URL: https://issues.apache.org/jira/browse/HAWQ-1226
> Project: Apache HAWQ
> Issue Type: Bug
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
>
> Core dump when starting hawq without starting RPS with enable_ranger is true.
> {code}
> (lldb) bt
> * thread #1: tid = 0x, 0x7fffe9264dda
> libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
> * frame #0: 0x7fffe9264dda libsystem_kernel.dylib`__pthread_kill + 10
> frame #1: 0x7fffe9350787 libsystem_pthread.dylib`pthread_kill + 90
> frame #2: 0x7fffe917b497 libsystem_c.dylib`raise + 26
> frame #3: 0x00010cbdd93f
> postgres`SafeHandlerForSegvBusIll(processName="Process",
> postgres_signal_arg=11) + 591 at elog.c:4519
> frame #4: 0x00010cbdd6cb
> postgres`StandardHandlerForSigillSigsegvSigbus_OnMainThread(processName="Process",
> postgres_signal_arg=11) + 43 at elog.c:4597
> frame #5: 0x00010cab66ef
> postgres`CdbProgramErrorHandler(postgres_signal_arg=11) + 319 at
> postgres.c:3512
> frame #6: 0x7fffe9343bba libsystem_platform.dylib`_sigtramp + 26
> frame #7: 0x7fffe916cb53 libsystem_c.dylib`strlen + 19
> frame #8: 0x00010c951215
> postgres`parse_ranger_response(buffer=0x) + 21 at
> rangerrest.c:59
> frame #9: 0x00010c951c8e
> postgres`check_privilege_from_ranger(user="wuhong", kind=ACL_KIND_NAMESPACE,
> object="template1", actions=0x7ff338036598, how=0x) + 254
> at rangerrest.c:339
> frame #10: 0x00010c7890c4
> postgres`pg_rangercheck(objkind=ACL_KIND_NAMESPACE, object_oid=2200,
> roleid=10, mask=256, how=ACLMASK_ANY) + 164 at aclchk.c:2678
> frame #11: 0x00010c78b085
> postgres`pg_namespace_aclcheck(nsp_oid=2200, roleid=10, mode=256) + 101 at
> aclchk.c:3752
> frame #12: 0x00010c77ea48 postgres`recomputeNamespacePath + 600 at
> namespace.c:2010
> frame #13: 0x00010c77dec1
> postgres`RelnameGetRelid(relname="gp_segment_configuration") + 17 at
> namespace.c:423
> frame #14: 0x00010c77db61
> postgres`RangeVarGetRelid(relation=0x7ff3388cef60, failOK='\0',
> allowHcatalog='\x01') + 449 at namespace.c:278
> frame #15: 0x00010c671f6e
> postgres`CdbOpenRelationRv(relation=0x7ff3388cef60, reqmode=3,
> noWait='\0', lockUpgraded=0x) + 46 at heapam.c:1075
> frame #16: 0x00010c7f718c
> postgres`setTargetTable(pstate=0x7ff3388cf460,
> relation=0x7ff3388cef60, inh='\x01', alsoSource='\x01', requiredPerms=8)
> + 140 at parse_clause.c:866
> frame #17: 0x00010c7b9a6e
> postgres`transformDeleteStmt(pstate=0x7ff3388cf460,
> stmt=0x7ff3388cf3c0) + 174 at analyze.c:943
> frame #18: 0x00010c7b5249
> postgres`transformStmt(pstate=0x7ff3388cf460,
> parseTree=0x7ff3388cf3c0, extras_before=0x7fff535bcba0,
> extras_after=0x7fff535bcb98) + 841 at analyze.c:769
> frame #19: 0x00010c7a5b60
> postgres`do_parse_analyze(parseTree=0x7ff3388cf3c0,
> pstate=0x7ff3388cf460) + 112 at analyze.c:497
> frame #20: 0x00010c7a5acf
> postgres`parse_analyze(parseTree=0x7ff3388cf3c0, sourceText="DELETE FROM
> gp_segment_configuration WHERE role = 'p' or role = 'm'",
> paramTypes=0x, numParams=0) + 95 at analyze.c:351
> frame #21: 0x00010cab5960
> postgres`pg_analyze_and_rewrite(parsetree=0x7ff3388cf3c0,
> query_string="DELETE FROM gp_segment_configuration WHERE role = 'p' or role =
> 'm'", paramTypes=0x, numParams=0) + 64 at postgres.c:812
> frame #22: 0x00010caba448
> postgres`exec_simple_query(query_string="DELETE FROM gp_segment_configuration
> WHERE role = 'p' or role = 'm'", seqServerHost=0x,
> seqServerPort=-1) + 1416 at postgres.c:1739
> frame #23: 0x00010cab8b15 postgres`PostgresMain(argc=8,
> argv=0x7ff338818698, username="wuhong") + 7189 at postgres.c:4840
> frame #24: 0x00010ca62295
> postgres`BackendRun(port=0x7ff337416a60) + 981 at postmaster.c:5915
> frame #25: 0x00010ca5f6e5
> postgres`BackendStartup(port=0x7ff337416a60) + 373 at postmaster.c:5484
> frame #26: 0x00010ca5cbb0 postgres`ServerLoop + 1248 at
> postmaster.c:2163
> frame #27: 0x00010ca5b2e3 postgres`PostmasterMain(argc=9,
> argv=0x7ff337415af0) + 4835 at postmaster.c:1454
> frame #28: 0x00010c96742c postgres`main(argc=9,
>
[jira] [Assigned] (HAWQ-1226) Enable ranger while RPS is down will cause HAWQ core dump
[
https://issues.apache.org/jira/browse/HAWQ-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongxu Ma reassigned HAWQ-1226:
---
Assignee: Hongxu Ma (was: Ed Espino)
> Enable ranger while RPS is down will cause HAWQ core dump
> -
>
> Key: HAWQ-1226
> URL: https://issues.apache.org/jira/browse/HAWQ-1226
> Project: Apache HAWQ
> Issue Type: Bug
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
>
> Core dump when starting hawq without starting RPS with enable_ranger is true.
> {code}
> (lldb) bt
> * thread #1: tid = 0x, 0x7fffe9264dda
> libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
> * frame #0: 0x7fffe9264dda libsystem_kernel.dylib`__pthread_kill + 10
> frame #1: 0x7fffe9350787 libsystem_pthread.dylib`pthread_kill + 90
> frame #2: 0x7fffe917b497 libsystem_c.dylib`raise + 26
> frame #3: 0x00010cbdd93f
> postgres`SafeHandlerForSegvBusIll(processName="Process",
> postgres_signal_arg=11) + 591 at elog.c:4519
> frame #4: 0x00010cbdd6cb
> postgres`StandardHandlerForSigillSigsegvSigbus_OnMainThread(processName="Process",
> postgres_signal_arg=11) + 43 at elog.c:4597
> frame #5: 0x00010cab66ef
> postgres`CdbProgramErrorHandler(postgres_signal_arg=11) + 319 at
> postgres.c:3512
> frame #6: 0x7fffe9343bba libsystem_platform.dylib`_sigtramp + 26
> frame #7: 0x7fffe916cb53 libsystem_c.dylib`strlen + 19
> frame #8: 0x00010c951215
> postgres`parse_ranger_response(buffer=0x) + 21 at
> rangerrest.c:59
> frame #9: 0x00010c951c8e
> postgres`check_privilege_from_ranger(user="wuhong", kind=ACL_KIND_NAMESPACE,
> object="template1", actions=0x7ff338036598, how=0x) + 254
> at rangerrest.c:339
> frame #10: 0x00010c7890c4
> postgres`pg_rangercheck(objkind=ACL_KIND_NAMESPACE, object_oid=2200,
> roleid=10, mask=256, how=ACLMASK_ANY) + 164 at aclchk.c:2678
> frame #11: 0x00010c78b085
> postgres`pg_namespace_aclcheck(nsp_oid=2200, roleid=10, mode=256) + 101 at
> aclchk.c:3752
> frame #12: 0x00010c77ea48 postgres`recomputeNamespacePath + 600 at
> namespace.c:2010
> frame #13: 0x00010c77dec1
> postgres`RelnameGetRelid(relname="gp_segment_configuration") + 17 at
> namespace.c:423
> frame #14: 0x00010c77db61
> postgres`RangeVarGetRelid(relation=0x7ff3388cef60, failOK='\0',
> allowHcatalog='\x01') + 449 at namespace.c:278
> frame #15: 0x00010c671f6e
> postgres`CdbOpenRelationRv(relation=0x7ff3388cef60, reqmode=3,
> noWait='\0', lockUpgraded=0x) + 46 at heapam.c:1075
> frame #16: 0x00010c7f718c
> postgres`setTargetTable(pstate=0x7ff3388cf460,
> relation=0x7ff3388cef60, inh='\x01', alsoSource='\x01', requiredPerms=8)
> + 140 at parse_clause.c:866
> frame #17: 0x00010c7b9a6e
> postgres`transformDeleteStmt(pstate=0x7ff3388cf460,
> stmt=0x7ff3388cf3c0) + 174 at analyze.c:943
> frame #18: 0x00010c7b5249
> postgres`transformStmt(pstate=0x7ff3388cf460,
> parseTree=0x7ff3388cf3c0, extras_before=0x7fff535bcba0,
> extras_after=0x7fff535bcb98) + 841 at analyze.c:769
> frame #19: 0x00010c7a5b60
> postgres`do_parse_analyze(parseTree=0x7ff3388cf3c0,
> pstate=0x7ff3388cf460) + 112 at analyze.c:497
> frame #20: 0x00010c7a5acf
> postgres`parse_analyze(parseTree=0x7ff3388cf3c0, sourceText="DELETE FROM
> gp_segment_configuration WHERE role = 'p' or role = 'm'",
> paramTypes=0x, numParams=0) + 95 at analyze.c:351
> frame #21: 0x00010cab5960
> postgres`pg_analyze_and_rewrite(parsetree=0x7ff3388cf3c0,
> query_string="DELETE FROM gp_segment_configuration WHERE role = 'p' or role =
> 'm'", paramTypes=0x, numParams=0) + 64 at postgres.c:812
> frame #22: 0x00010caba448
> postgres`exec_simple_query(query_string="DELETE FROM gp_segment_configuration
> WHERE role = 'p' or role = 'm'", seqServerHost=0x,
> seqServerPort=-1) + 1416 at postgres.c:1739
> frame #23: 0x00010cab8b15 postgres`PostgresMain(argc=8,
> argv=0x7ff338818698, username="wuhong") + 7189 at postgres.c:4840
> frame #24: 0x00010ca62295
> postgres`BackendRun(port=0x7ff337416a60) + 981 at postmaster.c:5915
> frame #25: 0x00010ca5f6e5
> postgres`BackendStartup(port=0x7ff337416a60) + 373 at postmaster.c:5484
> frame #26: 0x00010ca5cbb0 postgres`ServerLoop + 1248 at
> postmaster.c:2163
> frame #27: 0x00010ca5b2e3 postgres`PostmasterMain(argc=9,
> argv=0x7ff337415af0) + 4835 at postmaster.c:1454
> frame #28: 0x00010c96742c postgres`main(argc=9,
> argv=0x7ff337415af0) + 940 at main.c:226
> frame #29: 0x7fffe9136255 libdyld.dylib`start +
[jira] [Updated] (HAWQ-1226) Enable ranger while RPS is down will cause HAWQ core dump
[
https://issues.apache.org/jira/browse/HAWQ-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongxu Ma updated HAWQ-1226:
Description:
Core dump when starting hawq without starting RPS with enable_ranger is true.
{{
(lldb) bt
* thread #1: tid = 0x, 0x7fffe9264dda
libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
* frame #0: 0x7fffe9264dda libsystem_kernel.dylib`__pthread_kill + 10
frame #1: 0x7fffe9350787 libsystem_pthread.dylib`pthread_kill + 90
frame #2: 0x7fffe917b497 libsystem_c.dylib`raise + 26
frame #3: 0x00010cbdd93f
postgres`SafeHandlerForSegvBusIll(processName="Process",
postgres_signal_arg=11) + 591 at elog.c:4519
frame #4: 0x00010cbdd6cb
postgres`StandardHandlerForSigillSigsegvSigbus_OnMainThread(processName="Process",
postgres_signal_arg=11) + 43 at elog.c:4597
frame #5: 0x00010cab66ef
postgres`CdbProgramErrorHandler(postgres_signal_arg=11) + 319 at postgres.c:3512
frame #6: 0x7fffe9343bba libsystem_platform.dylib`_sigtramp + 26
frame #7: 0x7fffe916cb53 libsystem_c.dylib`strlen + 19
frame #8: 0x00010c951215
postgres`parse_ranger_response(buffer=0x) + 21 at
rangerrest.c:59
frame #9: 0x00010c951c8e
postgres`check_privilege_from_ranger(user="wuhong", kind=ACL_KIND_NAMESPACE,
object="template1", actions=0x7ff338036598, how=0x) + 254
at rangerrest.c:339
frame #10: 0x00010c7890c4
postgres`pg_rangercheck(objkind=ACL_KIND_NAMESPACE, object_oid=2200, roleid=10,
mask=256, how=ACLMASK_ANY) + 164 at aclchk.c:2678
frame #11: 0x00010c78b085 postgres`pg_namespace_aclcheck(nsp_oid=2200,
roleid=10, mode=256) + 101 at aclchk.c:3752
frame #12: 0x00010c77ea48 postgres`recomputeNamespacePath + 600 at
namespace.c:2010
frame #13: 0x00010c77dec1
postgres`RelnameGetRelid(relname="gp_segment_configuration") + 17 at
namespace.c:423
frame #14: 0x00010c77db61
postgres`RangeVarGetRelid(relation=0x7ff3388cef60, failOK='\0',
allowHcatalog='\x01') + 449 at namespace.c:278
frame #15: 0x00010c671f6e
postgres`CdbOpenRelationRv(relation=0x7ff3388cef60, reqmode=3, noWait='\0',
lockUpgraded=0x) + 46 at heapam.c:1075
frame #16: 0x00010c7f718c
postgres`setTargetTable(pstate=0x7ff3388cf460, relation=0x7ff3388cef60,
inh='\x01', alsoSource='\x01', requiredPerms=8) + 140 at parse_clause.c:866
frame #17: 0x00010c7b9a6e
postgres`transformDeleteStmt(pstate=0x7ff3388cf460,
stmt=0x7ff3388cf3c0) + 174 at analyze.c:943
frame #18: 0x00010c7b5249
postgres`transformStmt(pstate=0x7ff3388cf460, parseTree=0x7ff3388cf3c0,
extras_before=0x7fff535bcba0, extras_after=0x7fff535bcb98) + 841 at
analyze.c:769
frame #19: 0x00010c7a5b60
postgres`do_parse_analyze(parseTree=0x7ff3388cf3c0,
pstate=0x7ff3388cf460) + 112 at analyze.c:497
frame #20: 0x00010c7a5acf
postgres`parse_analyze(parseTree=0x7ff3388cf3c0, sourceText="DELETE FROM
gp_segment_configuration WHERE role = 'p' or role = 'm'",
paramTypes=0x, numParams=0) + 95 at analyze.c:351
frame #21: 0x00010cab5960
postgres`pg_analyze_and_rewrite(parsetree=0x7ff3388cf3c0,
query_string="DELETE FROM gp_segment_configuration WHERE role = 'p' or role =
'm'", paramTypes=0x, numParams=0) + 64 at postgres.c:812
frame #22: 0x00010caba448
postgres`exec_simple_query(query_string="DELETE FROM gp_segment_configuration
WHERE role = 'p' or role = 'm'", seqServerHost=0x,
seqServerPort=-1) + 1416 at postgres.c:1739
frame #23: 0x00010cab8b15 postgres`PostgresMain(argc=8,
argv=0x7ff338818698, username="wuhong") + 7189 at postgres.c:4840
frame #24: 0x00010ca62295 postgres`BackendRun(port=0x7ff337416a60)
+ 981 at postmaster.c:5915
frame #25: 0x00010ca5f6e5
postgres`BackendStartup(port=0x7ff337416a60) + 373 at postmaster.c:5484
frame #26: 0x00010ca5cbb0 postgres`ServerLoop + 1248 at
postmaster.c:2163
frame #27: 0x00010ca5b2e3 postgres`PostmasterMain(argc=9,
argv=0x7ff337415af0) + 4835 at postmaster.c:1454
frame #28: 0x00010c96742c postgres`main(argc=9,
argv=0x7ff337415af0) + 940 at main.c:226
frame #29: 0x7fffe9136255 libdyld.dylib`start + 1
frame #30: 0x7fffe9136255 libdyld.dylib`start + 1
}}
was:
Core dump when starting hawq without starting RPS with enable_ranger is true.
bq. {{
(lldb) bt
* thread #1: tid = 0x, 0x7fffe9264dda
libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
* frame #0: 0x7fffe9264dda libsystem_kernel.dylib`__pthread_kill + 10
frame #1: 0x7fffe9350787 libsystem_pthread.dylib`pthread_kill + 90
frame #2: 0x7fffe917b497 libsystem_c.dylib`raise + 26
frame #3: 0x00010cbdd93f
[jira] [Updated] (HAWQ-1226) Enable ranger while RPS is down will cause HAWQ core dump
[
https://issues.apache.org/jira/browse/HAWQ-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongxu Ma updated HAWQ-1226:
Description:
Core dump when starting hawq without starting RPS with enable_ranger is true.
{code}
(lldb) bt
* thread #1: tid = 0x, 0x7fffe9264dda
libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
* frame #0: 0x7fffe9264dda libsystem_kernel.dylib`__pthread_kill + 10
frame #1: 0x7fffe9350787 libsystem_pthread.dylib`pthread_kill + 90
frame #2: 0x7fffe917b497 libsystem_c.dylib`raise + 26
frame #3: 0x00010cbdd93f
postgres`SafeHandlerForSegvBusIll(processName="Process",
postgres_signal_arg=11) + 591 at elog.c:4519
frame #4: 0x00010cbdd6cb
postgres`StandardHandlerForSigillSigsegvSigbus_OnMainThread(processName="Process",
postgres_signal_arg=11) + 43 at elog.c:4597
frame #5: 0x00010cab66ef
postgres`CdbProgramErrorHandler(postgres_signal_arg=11) + 319 at postgres.c:3512
frame #6: 0x7fffe9343bba libsystem_platform.dylib`_sigtramp + 26
frame #7: 0x7fffe916cb53 libsystem_c.dylib`strlen + 19
frame #8: 0x00010c951215
postgres`parse_ranger_response(buffer=0x) + 21 at
rangerrest.c:59
frame #9: 0x00010c951c8e
postgres`check_privilege_from_ranger(user="wuhong", kind=ACL_KIND_NAMESPACE,
object="template1", actions=0x7ff338036598, how=0x) + 254
at rangerrest.c:339
frame #10: 0x00010c7890c4
postgres`pg_rangercheck(objkind=ACL_KIND_NAMESPACE, object_oid=2200, roleid=10,
mask=256, how=ACLMASK_ANY) + 164 at aclchk.c:2678
frame #11: 0x00010c78b085 postgres`pg_namespace_aclcheck(nsp_oid=2200,
roleid=10, mode=256) + 101 at aclchk.c:3752
frame #12: 0x00010c77ea48 postgres`recomputeNamespacePath + 600 at
namespace.c:2010
frame #13: 0x00010c77dec1
postgres`RelnameGetRelid(relname="gp_segment_configuration") + 17 at
namespace.c:423
frame #14: 0x00010c77db61
postgres`RangeVarGetRelid(relation=0x7ff3388cef60, failOK='\0',
allowHcatalog='\x01') + 449 at namespace.c:278
frame #15: 0x00010c671f6e
postgres`CdbOpenRelationRv(relation=0x7ff3388cef60, reqmode=3, noWait='\0',
lockUpgraded=0x) + 46 at heapam.c:1075
frame #16: 0x00010c7f718c
postgres`setTargetTable(pstate=0x7ff3388cf460, relation=0x7ff3388cef60,
inh='\x01', alsoSource='\x01', requiredPerms=8) + 140 at parse_clause.c:866
frame #17: 0x00010c7b9a6e
postgres`transformDeleteStmt(pstate=0x7ff3388cf460,
stmt=0x7ff3388cf3c0) + 174 at analyze.c:943
frame #18: 0x00010c7b5249
postgres`transformStmt(pstate=0x7ff3388cf460, parseTree=0x7ff3388cf3c0,
extras_before=0x7fff535bcba0, extras_after=0x7fff535bcb98) + 841 at
analyze.c:769
frame #19: 0x00010c7a5b60
postgres`do_parse_analyze(parseTree=0x7ff3388cf3c0,
pstate=0x7ff3388cf460) + 112 at analyze.c:497
frame #20: 0x00010c7a5acf
postgres`parse_analyze(parseTree=0x7ff3388cf3c0, sourceText="DELETE FROM
gp_segment_configuration WHERE role = 'p' or role = 'm'",
paramTypes=0x, numParams=0) + 95 at analyze.c:351
frame #21: 0x00010cab5960
postgres`pg_analyze_and_rewrite(parsetree=0x7ff3388cf3c0,
query_string="DELETE FROM gp_segment_configuration WHERE role = 'p' or role =
'm'", paramTypes=0x, numParams=0) + 64 at postgres.c:812
frame #22: 0x00010caba448
postgres`exec_simple_query(query_string="DELETE FROM gp_segment_configuration
WHERE role = 'p' or role = 'm'", seqServerHost=0x,
seqServerPort=-1) + 1416 at postgres.c:1739
frame #23: 0x00010cab8b15 postgres`PostgresMain(argc=8,
argv=0x7ff338818698, username="wuhong") + 7189 at postgres.c:4840
frame #24: 0x00010ca62295 postgres`BackendRun(port=0x7ff337416a60)
+ 981 at postmaster.c:5915
frame #25: 0x00010ca5f6e5
postgres`BackendStartup(port=0x7ff337416a60) + 373 at postmaster.c:5484
frame #26: 0x00010ca5cbb0 postgres`ServerLoop + 1248 at
postmaster.c:2163
frame #27: 0x00010ca5b2e3 postgres`PostmasterMain(argc=9,
argv=0x7ff337415af0) + 4835 at postmaster.c:1454
frame #28: 0x00010c96742c postgres`main(argc=9,
argv=0x7ff337415af0) + 940 at main.c:226
frame #29: 0x7fffe9136255 libdyld.dylib`start + 1
frame #30: 0x7fffe9136255 libdyld.dylib`start + 1
{code}}
was:
Core dump when starting hawq without starting RPS with enable_ranger is true.
{{
(lldb) bt
* thread #1: tid = 0x, 0x7fffe9264dda
libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
* frame #0: 0x7fffe9264dda libsystem_kernel.dylib`__pthread_kill + 10
frame #1: 0x7fffe9350787 libsystem_pthread.dylib`pthread_kill + 90
frame #2: 0x7fffe917b497 libsystem_c.dylib`raise + 26
frame #3: 0x00010cbdd93f
[jira] [Created] (HAWQ-1226) Enable ranger while RPS is down will cause HAWQ core dump
Hongxu Ma created HAWQ-1226:
---
Summary: Enable ranger while RPS is down will cause HAWQ core dump
Key: HAWQ-1226
URL: https://issues.apache.org/jira/browse/HAWQ-1226
Project: Apache HAWQ
Issue Type: Bug
Reporter: Hongxu Ma
Assignee: Ed Espino
Core dump when starting hawq without starting RPS with enable_ranger is true.
bq. {{
(lldb) bt
* thread #1: tid = 0x, 0x7fffe9264dda
libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
* frame #0: 0x7fffe9264dda libsystem_kernel.dylib`__pthread_kill + 10
frame #1: 0x7fffe9350787 libsystem_pthread.dylib`pthread_kill + 90
frame #2: 0x7fffe917b497 libsystem_c.dylib`raise + 26
frame #3: 0x00010cbdd93f
postgres`SafeHandlerForSegvBusIll(processName="Process",
postgres_signal_arg=11) + 591 at elog.c:4519
frame #4: 0x00010cbdd6cb
postgres`StandardHandlerForSigillSigsegvSigbus_OnMainThread(processName="Process",
postgres_signal_arg=11) + 43 at elog.c:4597
frame #5: 0x00010cab66ef
postgres`CdbProgramErrorHandler(postgres_signal_arg=11) + 319 at postgres.c:3512
frame #6: 0x7fffe9343bba libsystem_platform.dylib`_sigtramp + 26
frame #7: 0x7fffe916cb53 libsystem_c.dylib`strlen + 19
frame #8: 0x00010c951215
postgres`parse_ranger_response(buffer=0x) + 21 at
rangerrest.c:59
frame #9: 0x00010c951c8e
postgres`check_privilege_from_ranger(user="wuhong", kind=ACL_KIND_NAMESPACE,
object="template1", actions=0x7ff338036598, how=0x) + 254
at rangerrest.c:339
frame #10: 0x00010c7890c4
postgres`pg_rangercheck(objkind=ACL_KIND_NAMESPACE, object_oid=2200, roleid=10,
mask=256, how=ACLMASK_ANY) + 164 at aclchk.c:2678
frame #11: 0x00010c78b085 postgres`pg_namespace_aclcheck(nsp_oid=2200,
roleid=10, mode=256) + 101 at aclchk.c:3752
frame #12: 0x00010c77ea48 postgres`recomputeNamespacePath + 600 at
namespace.c:2010
frame #13: 0x00010c77dec1
postgres`RelnameGetRelid(relname="gp_segment_configuration") + 17 at
namespace.c:423
frame #14: 0x00010c77db61
postgres`RangeVarGetRelid(relation=0x7ff3388cef60, failOK='\0',
allowHcatalog='\x01') + 449 at namespace.c:278
frame #15: 0x00010c671f6e
postgres`CdbOpenRelationRv(relation=0x7ff3388cef60, reqmode=3, noWait='\0',
lockUpgraded=0x) + 46 at heapam.c:1075
frame #16: 0x00010c7f718c
postgres`setTargetTable(pstate=0x7ff3388cf460, relation=0x7ff3388cef60,
inh='\x01', alsoSource='\x01', requiredPerms=8) + 140 at parse_clause.c:866
frame #17: 0x00010c7b9a6e
postgres`transformDeleteStmt(pstate=0x7ff3388cf460,
stmt=0x7ff3388cf3c0) + 174 at analyze.c:943
frame #18: 0x00010c7b5249
postgres`transformStmt(pstate=0x7ff3388cf460, parseTree=0x7ff3388cf3c0,
extras_before=0x7fff535bcba0, extras_after=0x7fff535bcb98) + 841 at
analyze.c:769
frame #19: 0x00010c7a5b60
postgres`do_parse_analyze(parseTree=0x7ff3388cf3c0,
pstate=0x7ff3388cf460) + 112 at analyze.c:497
frame #20: 0x00010c7a5acf
postgres`parse_analyze(parseTree=0x7ff3388cf3c0, sourceText="DELETE FROM
gp_segment_configuration WHERE role = 'p' or role = 'm'",
paramTypes=0x, numParams=0) + 95 at analyze.c:351
frame #21: 0x00010cab5960
postgres`pg_analyze_and_rewrite(parsetree=0x7ff3388cf3c0,
query_string="DELETE FROM gp_segment_configuration WHERE role = 'p' or role =
'm'", paramTypes=0x, numParams=0) + 64 at postgres.c:812
frame #22: 0x00010caba448
postgres`exec_simple_query(query_string="DELETE FROM gp_segment_configuration
WHERE role = 'p' or role = 'm'", seqServerHost=0x,
seqServerPort=-1) + 1416 at postgres.c:1739
frame #23: 0x00010cab8b15 postgres`PostgresMain(argc=8,
argv=0x7ff338818698, username="wuhong") + 7189 at postgres.c:4840
frame #24: 0x00010ca62295 postgres`BackendRun(port=0x7ff337416a60)
+ 981 at postmaster.c:5915
frame #25: 0x00010ca5f6e5
postgres`BackendStartup(port=0x7ff337416a60) + 373 at postmaster.c:5484
frame #26: 0x00010ca5cbb0 postgres`ServerLoop + 1248 at
postmaster.c:2163
frame #27: 0x00010ca5b2e3 postgres`PostmasterMain(argc=9,
argv=0x7ff337415af0) + 4835 at postmaster.c:1454
frame #28: 0x00010c96742c postgres`main(argc=9,
argv=0x7ff337415af0) + 940 at main.c:226
frame #29: 0x7fffe9136255 libdyld.dylib`start + 1
frame #30: 0x7fffe9136255 libdyld.dylib`start + 1
}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (HAWQ-870) Allocate target's tuple table slot in PortalHeapMemory during split partition
[ https://issues.apache.org/jira/browse/HAWQ-870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma closed HAWQ-870. -- Resolution: Fixed already commit in. > Allocate target's tuple table slot in PortalHeapMemory during split partition > - > > Key: HAWQ-870 > URL: https://issues.apache.org/jira/browse/HAWQ-870 > Project: Apache HAWQ > Issue Type: Bug > Components: Query Execution >Reporter: Venkatesh >Assignee: Hongxu Ma > Fix For: backlog > > > This is a nice fix from QP team on GPDB. Please port this fix into HAWQ. Th > GPDB Commit: > https://github.com/greenplum-db/gpdb/commit/c0e1f00c2532d1e2ef8d3b409dc8fee901a7cfe2 > PR: https://github.com/greenplum-db/gpdb/pull/866 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1193) TDE support in HAWQ
[ https://issues.apache.org/jira/browse/HAWQ-1193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma updated HAWQ-1193: Attachment: HAWQ_TDE_Design_ver0.1.pdf design doc ver0.1, better format > TDE support in HAWQ > --- > > Key: HAWQ-1193 > URL: https://issues.apache.org/jira/browse/HAWQ-1193 > Project: Apache HAWQ > Issue Type: New Feature > Components: libhdfs >Reporter: Hongxu Ma >Assignee: Hongxu Ma > Fix For: backlog > > Attachments: HAWQ_TDE_Design_ver0.1.pdf > > > TDE(transparently data encrypted) has been supported after hadoop 2.6: > http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html > https://issues.apache.org/jira/browse/HDFS-6134 > Use TDE can promise： > 1, hdfs file is encrypted. > 2, network transfer between hdfs and libhdfs client is encrypted. > So hawq will update libhdfs3 to support it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1193) TDE support in HAWQ
[ https://issues.apache.org/jira/browse/HAWQ-1193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma updated HAWQ-1193: Attachment: (was: HAWQ_TDE_Design_ver0.1.pdf) > TDE support in HAWQ > --- > > Key: HAWQ-1193 > URL: https://issues.apache.org/jira/browse/HAWQ-1193 > Project: Apache HAWQ > Issue Type: New Feature > Components: libhdfs >Reporter: Hongxu Ma >Assignee: Hongxu Ma > Fix For: backlog > > > TDE(transparently data encrypted) has been supported after hadoop 2.6: > http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html > https://issues.apache.org/jira/browse/HDFS-6134 > Use TDE can promise： > 1, hdfs file is encrypted. > 2, network transfer between hdfs and libhdfs client is encrypted. > So hawq will update libhdfs3 to support it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1193) TDE support in HAWQ
[ https://issues.apache.org/jira/browse/HAWQ-1193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma updated HAWQ-1193: Attachment: HAWQ_TDE_Design_ver0.1.pdf design doc ver0.1 updated > TDE support in HAWQ > --- > > Key: HAWQ-1193 > URL: https://issues.apache.org/jira/browse/HAWQ-1193 > Project: Apache HAWQ > Issue Type: New Feature > Components: libhdfs >Reporter: Hongxu Ma >Assignee: Hongxu Ma > Fix For: backlog > > Attachments: HAWQ_TDE_Design_ver0.1.pdf > > > TDE(transparently data encrypted) has been supported after hadoop 2.6: > http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html > https://issues.apache.org/jira/browse/HDFS-6134 > Use TDE can promise： > 1, hdfs file is encrypted. > 2, network transfer between hdfs and libhdfs client is encrypted. > So hawq will update libhdfs3 to support it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1193) TDE support in HAWQ
[ https://issues.apache.org/jira/browse/HAWQ-1193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma updated HAWQ-1193: Attachment: (was: HAWQ_TDE_Design_ver0.1.pdf) > TDE support in HAWQ > --- > > Key: HAWQ-1193 > URL: https://issues.apache.org/jira/browse/HAWQ-1193 > Project: Apache HAWQ > Issue Type: New Feature > Components: libhdfs >Reporter: Hongxu Ma >Assignee: Hongxu Ma > Fix For: backlog > > > TDE(transparently data encrypted) has been supported after hadoop 2.6: > http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html > https://issues.apache.org/jira/browse/HDFS-6134 > Use TDE can promise： > 1, hdfs file is encrypted. > 2, network transfer between hdfs and libhdfs client is encrypted. > So hawq will update libhdfs3 to support it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1193) TDE support in HAWQ
[ https://issues.apache.org/jira/browse/HAWQ-1193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma updated HAWQ-1193: Description: TDE(transparently data encrypted) has been supported after hadoop 2.6: http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html https://issues.apache.org/jira/browse/HDFS-6134 Use TDE can promise： 1, hdfs file is encrypted. 2, network transfer between hdfs and libhdfs client is encrypted. So hawq will update libhdfs3 to support it. was: TDE(transparently data encrypted) has been supported after hadoop 2.6: http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html https://issues.apache.org/jira/browse/HDFS-6134 Use TDE can promise： 1, hdfs file is encrypted. 2, network transfer is encrypted. So hawq will update libhdfs3 to support it. > TDE support in HAWQ > --- > > Key: HAWQ-1193 > URL: https://issues.apache.org/jira/browse/HAWQ-1193 > Project: Apache HAWQ > Issue Type: New Feature > Components: libhdfs >Reporter: Hongxu Ma >Assignee: Hongxu Ma > Fix For: backlog > > Attachments: HAWQ_TDE_Design_ver0.1.pdf > > > TDE(transparently data encrypted) has been supported after hadoop 2.6: > http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html > https://issues.apache.org/jira/browse/HDFS-6134 > Use TDE can promise： > 1, hdfs file is encrypted. > 2, network transfer between hdfs and libhdfs client is encrypted. > So hawq will update libhdfs3 to support it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1194) XXXX Todo
[ https://issues.apache.org/jira/browse/HAWQ-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma updated HAWQ-1194: Summary: Todo (was: Design document) > Todo > - > > Key: HAWQ-1194 > URL: https://issues.apache.org/jira/browse/HAWQ-1194 > Project: Apache HAWQ > Issue Type: Sub-task > Components: libhdfs >Reporter: Hongxu Ma >Assignee: Hongxu Ma > Fix For: backlog > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1193) TDE support in HAWQ
[ https://issues.apache.org/jira/browse/HAWQ-1193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma updated HAWQ-1193: Attachment: HAWQ_TDE_Design_ver0.1.pdf design doc ver0.1 > TDE support in HAWQ > --- > > Key: HAWQ-1193 > URL: https://issues.apache.org/jira/browse/HAWQ-1193 > Project: Apache HAWQ > Issue Type: New Feature > Components: libhdfs >Reporter: Hongxu Ma >Assignee: Hongxu Ma > Fix For: backlog > > Attachments: HAWQ_TDE_Design_ver0.1.pdf > > > TDE(transparently data encrypted) has been supported after hadoop 2.6: > http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html > https://issues.apache.org/jira/browse/HDFS-6134 > Use TDE can promise： > 1, hdfs file is encrypted. > 2, network transfer is encrypted. > So hawq will update libhdfs3 to support it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-1193) TDE support in HAWQ
[ https://issues.apache.org/jira/browse/HAWQ-1193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15727431#comment-15727431 ] Hongxu Ma commented on HAWQ-1193: - I have seen bdrosen96's token implementation. It will be considered in this feature. If you have more ideas, let me know. Thanks alastair. > TDE support in HAWQ > --- > > Key: HAWQ-1193 > URL: https://issues.apache.org/jira/browse/HAWQ-1193 > Project: Apache HAWQ > Issue Type: New Feature > Components: libhdfs >Reporter: Hongxu Ma >Assignee: Hongxu Ma > Fix For: backlog > > > TDE(transparently data encrypted) has been supported after hadoop 2.6: > http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html > https://issues.apache.org/jira/browse/HDFS-6134 > Use TDE can promise： > 1, hdfs file is encrypted. > 2, network transfer is encrypted. > So hawq will update libhdfs3 to support it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HAWQ-1194) Design document
[ https://issues.apache.org/jira/browse/HAWQ-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma reassigned HAWQ-1194: --- Assignee: Hongxu Ma (was: Lei Chang) > Design document > --- > > Key: HAWQ-1194 > URL: https://issues.apache.org/jira/browse/HAWQ-1194 > Project: Apache HAWQ > Issue Type: Sub-task > Components: libhdfs >Reporter: Hongxu Ma >Assignee: Hongxu Ma > Fix For: backlog > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1193) TDE support in HAWQ
[ https://issues.apache.org/jira/browse/HAWQ-1193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma updated HAWQ-1193: Description: TDE(transparently data encrypted) has been supported after hadoop 2.6: http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html https://issues.apache.org/jira/browse/HDFS-6134 Use TDE can promise： 1, hdfs file is encrypted. 2, network transfer is encrypted. So hawq will update libhdfs3 to support it. was: TDE(transparently data encrypted) has been supported after hadoop 2.6: http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html https://issues.apache.org/jira/browse/HDFS-6134 Use tde can promise： 1, hdfs file is encrypted. 2, network transfer is encrypted. So hawq will update libhdfs3 to support it. > TDE support in HAWQ > --- > > Key: HAWQ-1193 > URL: https://issues.apache.org/jira/browse/HAWQ-1193 > Project: Apache HAWQ > Issue Type: New Feature > Components: libhdfs >Reporter: Hongxu Ma >Assignee: Lei Chang > Fix For: backlog > > > TDE(transparently data encrypted) has been supported after hadoop 2.6: > http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html > https://issues.apache.org/jira/browse/HDFS-6134 > Use TDE can promise： > 1, hdfs file is encrypted. > 2, network transfer is encrypted. > So hawq will update libhdfs3 to support it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HAWQ-1194) Design document
Hongxu Ma created HAWQ-1194: --- Summary: Design document Key: HAWQ-1194 URL: https://issues.apache.org/jira/browse/HAWQ-1194 Project: Apache HAWQ Issue Type: Sub-task Reporter: Hongxu Ma Assignee: Lei Chang -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HAWQ-1193) TDE support in HAWQ
Hongxu Ma created HAWQ-1193: --- Summary: TDE support in HAWQ Key: HAWQ-1193 URL: https://issues.apache.org/jira/browse/HAWQ-1193 Project: Apache HAWQ Issue Type: New Feature Components: libhdfs Reporter: Hongxu Ma Assignee: Lei Chang Fix For: backlog TDE(transparently data encrypted) has been supported after hadoop 2.6: http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html https://issues.apache.org/jira/browse/HDFS-6134 Use tde can promise： 1, hdfs file is encrypted. 2, network transfer is encrypted. So hawq will update libhdfs3 to support it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HAWQ-1193) TDE support in HAWQ
[ https://issues.apache.org/jira/browse/HAWQ-1193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma reassigned HAWQ-1193: --- Assignee: Hongxu Ma (was: Lei Chang) > TDE support in HAWQ > --- > > Key: HAWQ-1193 > URL: https://issues.apache.org/jira/browse/HAWQ-1193 > Project: Apache HAWQ > Issue Type: New Feature > Components: libhdfs >Reporter: Hongxu Ma >Assignee: Hongxu Ma > Fix For: backlog > > > TDE(transparently data encrypted) has been supported after hadoop 2.6: > http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html > https://issues.apache.org/jira/browse/HDFS-6134 > Use TDE can promise： > 1, hdfs file is encrypted. > 2, network transfer is encrypted. > So hawq will update libhdfs3 to support it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-870) Allocate target's tuple table slot in PortalHeapMemory during split partition
[ https://issues.apache.org/jira/browse/HAWQ-870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15670112#comment-15670112 ] Hongxu Ma commented on HAWQ-870: Test Ok. But there is a small charge in partition.sql, the statement "appendonly=false" of PARTITION statement: +PARTITION BY RANGE(log_id) +( + START (1::int) END (100::int) EVERY (5) WITH (appendonly=false), + PARTITION "Old" START (101::int) END (201::int) WITH (appendonly=false), + DEFAULT PARTITION other_log_ids WITH (statement) +); Since HAWQ do not support "appendonly=false" table, it should be remove the appendonly statement. So the rest of test make sense. > Allocate target's tuple table slot in PortalHeapMemory during split partition > - > > Key: HAWQ-870 > URL: https://issues.apache.org/jira/browse/HAWQ-870 > Project: Apache HAWQ > Issue Type: Bug > Components: Query Execution >Reporter: Venkatesh >Assignee: Hongxu Ma > Fix For: backlog > > > This is a nice fix from QP team on GPDB. Please port this fix into HAWQ. Th > GPDB Commit: > https://github.com/greenplum-db/gpdb/commit/c0e1f00c2532d1e2ef8d3b409dc8fee901a7cfe2 > PR: https://github.com/greenplum-db/gpdb/pull/866 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HAWQ-513) initdb.c failed on OSX 10.11.3 due to fgets error
[
https://issues.apache.org/jira/browse/HAWQ-513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15666182#comment-15666182
]
Hongxu Ma edited comment on HAWQ-513 at 11/16/16 7:58 AM:
--
The problem is caused by the Mac OSX "Recovery Mode" (env variable
"DYLD_LIBRARY_PATH" work abnormally)
Solved it by:
Enter the mac "Recovery Mode", and set csrutil disable.
More detail:
https://cwiki.apache.org/confluence/display/HAWQ/Build+and+Install
See the "Turning Off Rootless System Integrity Protection in OS X El Capitan
10.11+" section.
was (Author: hongxu ma):
The problem is caused by the Mac OSX "Recovery Mode" (env variable
"DYLD_LIBRARY_PATH" work abnormally)
And solved it by do:
Enter the mac "Recovery Mode", and set csrutil disable.
More detail:
https://cwiki.apache.org/confluence/display/HAWQ/Build+and+Install
See the "Turning Off Rootless System Integrity Protection in OS X El Capitan
10.11+" section.
> initdb.c failed on OSX 10.11.3 due to fgets error
> -
>
> Key: HAWQ-513
> URL: https://issues.apache.org/jira/browse/HAWQ-513
> Project: Apache HAWQ
> Issue Type: Bug
> Components: Unknown
>Reporter: xin zhang
>Assignee: Hongxu Ma
> Fix For: backlog
>
>
> we hit following strange issue on OSX 10.11.3:
> The error message in the initdb is:
> {code}
> 20160301:00:00:26:075823 hawq_init:This-MacBook-Pro:vagrant-[INFO]:-Start to
> init master node: 'localhost'
> sh: line 1: 76106 Trace/BPT trap: 5 "/usr/local/hawq/bin/postgres" -V
> 2> /dev/null
> fgets failure: Undefined error: 0
> The program "postgres" is needed by initdb but was either not found in the
> same directory as "/usr/local/hawq/bin/initdb" or failed unexpectedly.
> Check your installation; "postgres -V" may have more information.
> Master postgres initdb failed
> {code}
> We suspect the issue due to the newer version of the libSystem.B.dylib on OSX
> 10.11.3.
> Here is the details of the dependencies of `postgres` and `initdb`:
> 10.10.5, postgres can start, and initdb succeed:
> {code}
> [bin: xzhang{master}]$ otool -L postgres
> postgres:
> /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version
> 120.0.0)
> libhdfs3.1.dylib (compatibility version 1.0.0, current version 2.2.30)
> /usr/lib/libxml2.2.dylib (compatibility version 10.0.0, current version
> 10.9.0)
> libyarn.1.dylib (compatibility version 1.0.0, current version 0.1.13)
> /usr/local/opt/json-c/lib/libjson-c.2.dylib (compatibility version 3.0.0,
> current version 3.1.0)
> /usr/local/opt/snappy/lib/libsnappy.1.dylib (compatibility version 5.0.0,
> current version 5.0.0)
> /usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version
> 1.0.5)
> /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
> 1213.0.0)
> /usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version
> 8.0.0)
> /System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos
> (compatibility version 5.0.0, current version 6.0.0)
> libdxltranslators.dylib (compatibility version 0.0.0, current version
> 0.0.0)
> /usr/local/opt/thrift/lib/libthrift-0.9.3.dylib (compatibility version
> 0.0.0, current version 0.0.0)
> This-MacBook-Pro:bin vagrant$ otool -L initdb
> initdb:
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
> 1226.10.1)
> {code}
> 10.11.3, postgres can start, but initdb failed:
> {code}
> This-MacBook-Pro:bin vagrant$ otool -L postgres
> postgres:
> /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version
> 120.1.0)
> libhdfs3.1.dylib (compatibility version 1.0.0, current version 2.2.30)
> /usr/lib/libxml2.2.dylib (compatibility version 10.0.0, current version
> 10.9.0)
> libyarn.1.dylib (compatibility version 1.0.0, current version 0.1.13)
> /usr/local/opt/json-c/lib/libjson-c.2.dylib (compatibility version 3.0.0,
> current version 3.1.0)
> /usr/local/opt/snappy/lib/libsnappy.1.dylib (compatibility version 5.0.0,
> current version 5.0.0)
> /usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version
> 1.0.5)
> /usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current version
> 0.9.8)
> /usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current
> version 0.9.8)
> /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
> 1226.10.1)
> /usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version
> 8.0.0)
> /System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos
> (compatibility version 5.0.0, current version 6.0.0)
>
[jira] [Resolved] (HAWQ-513) initdb.c failed on OSX 10.11.3 due to fgets error
[
https://issues.apache.org/jira/browse/HAWQ-513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongxu Ma resolved HAWQ-513.
Resolution: Fixed
The problem is caused by the Mac OSX "Recovery Mode" (env variable
"DYLD_LIBRARY_PATH" work abnormally)
And solved it by do:
Enter the mac "Recovery Mode", and set csrutil disable.
More detail:
https://cwiki.apache.org/confluence/display/HAWQ/Build+and+Install
See the "Turning Off Rootless System Integrity Protection in OS X El Capitan
10.11+" section.
> initdb.c failed on OSX 10.11.3 due to fgets error
> -
>
> Key: HAWQ-513
> URL: https://issues.apache.org/jira/browse/HAWQ-513
> Project: Apache HAWQ
> Issue Type: Bug
> Components: Unknown
>Reporter: xin zhang
>Assignee: Hongxu Ma
> Fix For: backlog
>
>
> we hit following strange issue on OSX 10.11.3:
> The error message in the initdb is:
> {code}
> 20160301:00:00:26:075823 hawq_init:This-MacBook-Pro:vagrant-[INFO]:-Start to
> init master node: 'localhost'
> sh: line 1: 76106 Trace/BPT trap: 5 "/usr/local/hawq/bin/postgres" -V
> 2> /dev/null
> fgets failure: Undefined error: 0
> The program "postgres" is needed by initdb but was either not found in the
> same directory as "/usr/local/hawq/bin/initdb" or failed unexpectedly.
> Check your installation; "postgres -V" may have more information.
> Master postgres initdb failed
> {code}
> We suspect the issue due to the newer version of the libSystem.B.dylib on OSX
> 10.11.3.
> Here is the details of the dependencies of `postgres` and `initdb`:
> 10.10.5, postgres can start, and initdb succeed:
> {code}
> [bin: xzhang{master}]$ otool -L postgres
> postgres:
> /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version
> 120.0.0)
> libhdfs3.1.dylib (compatibility version 1.0.0, current version 2.2.30)
> /usr/lib/libxml2.2.dylib (compatibility version 10.0.0, current version
> 10.9.0)
> libyarn.1.dylib (compatibility version 1.0.0, current version 0.1.13)
> /usr/local/opt/json-c/lib/libjson-c.2.dylib (compatibility version 3.0.0,
> current version 3.1.0)
> /usr/local/opt/snappy/lib/libsnappy.1.dylib (compatibility version 5.0.0,
> current version 5.0.0)
> /usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version
> 1.0.5)
> /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
> 1213.0.0)
> /usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version
> 8.0.0)
> /System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos
> (compatibility version 5.0.0, current version 6.0.0)
> libdxltranslators.dylib (compatibility version 0.0.0, current version
> 0.0.0)
> /usr/local/opt/thrift/lib/libthrift-0.9.3.dylib (compatibility version
> 0.0.0, current version 0.0.0)
> This-MacBook-Pro:bin vagrant$ otool -L initdb
> initdb:
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
> 1226.10.1)
> {code}
> 10.11.3, postgres can start, but initdb failed:
> {code}
> This-MacBook-Pro:bin vagrant$ otool -L postgres
> postgres:
> /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version
> 120.1.0)
> libhdfs3.1.dylib (compatibility version 1.0.0, current version 2.2.30)
> /usr/lib/libxml2.2.dylib (compatibility version 10.0.0, current version
> 10.9.0)
> libyarn.1.dylib (compatibility version 1.0.0, current version 0.1.13)
> /usr/local/opt/json-c/lib/libjson-c.2.dylib (compatibility version 3.0.0,
> current version 3.1.0)
> /usr/local/opt/snappy/lib/libsnappy.1.dylib (compatibility version 5.0.0,
> current version 5.0.0)
> /usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version
> 1.0.5)
> /usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current version
> 0.9.8)
> /usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current
> version 0.9.8)
> /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
> 1226.10.1)
> /usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version
> 8.0.0)
> /System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos
> (compatibility version 5.0.0, current version 6.0.0)
> libdxltranslators.dylib (compatibility version 0.0.0, current version
> 0.0.0)
> /usr/local/opt/thrift/lib/libthrift-0.9.3.dylib (compatibility version
> 0.0.0, current version 0.0.0)
> [bin: xzhang{master}]$ otool -L initdb
> initdb:
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
> 1213.0.0)
> {code}
> In this case, there is a difference between the two OS regarding to the
> libSystem.B.dylib.
>
[jira] [Assigned] (HAWQ-870) Allocate target's tuple table slot in PortalHeapMemory during split partition
[ https://issues.apache.org/jira/browse/HAWQ-870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongxu Ma reassigned HAWQ-870: -- Assignee: Hongxu Ma (was: Lei Chang) > Allocate target's tuple table slot in PortalHeapMemory during split partition > - > > Key: HAWQ-870 > URL: https://issues.apache.org/jira/browse/HAWQ-870 > Project: Apache HAWQ > Issue Type: Bug > Components: Query Execution >Reporter: Venkatesh >Assignee: Hongxu Ma > Fix For: backlog > > > This is a nice fix from QP team on GPDB. Please port this fix into HAWQ. Th > GPDB Commit: > https://github.com/greenplum-db/gpdb/commit/c0e1f00c2532d1e2ef8d3b409dc8fee901a7cfe2 > PR: https://github.com/greenplum-db/gpdb/pull/866 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HAWQ-513) initdb.c failed on OSX 10.11.3 due to fgets error
[
https://issues.apache.org/jira/browse/HAWQ-513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongxu Ma reassigned HAWQ-513:
--
Assignee: Hongxu Ma (was: Lei Chang)
> initdb.c failed on OSX 10.11.3 due to fgets error
> -
>
> Key: HAWQ-513
> URL: https://issues.apache.org/jira/browse/HAWQ-513
> Project: Apache HAWQ
> Issue Type: Bug
> Components: Unknown
>Reporter: xin zhang
>Assignee: Hongxu Ma
> Fix For: backlog
>
>
> we hit following strange issue on OSX 10.11.3:
> The error message in the initdb is:
> {code}
> 20160301:00:00:26:075823 hawq_init:This-MacBook-Pro:vagrant-[INFO]:-Start to
> init master node: 'localhost'
> sh: line 1: 76106 Trace/BPT trap: 5 "/usr/local/hawq/bin/postgres" -V
> 2> /dev/null
> fgets failure: Undefined error: 0
> The program "postgres" is needed by initdb but was either not found in the
> same directory as "/usr/local/hawq/bin/initdb" or failed unexpectedly.
> Check your installation; "postgres -V" may have more information.
> Master postgres initdb failed
> {code}
> We suspect the issue due to the newer version of the libSystem.B.dylib on OSX
> 10.11.3.
> Here is the details of the dependencies of `postgres` and `initdb`:
> 10.10.5, postgres can start, and initdb succeed:
> {code}
> [bin: xzhang{master}]$ otool -L postgres
> postgres:
> /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version
> 120.0.0)
> libhdfs3.1.dylib (compatibility version 1.0.0, current version 2.2.30)
> /usr/lib/libxml2.2.dylib (compatibility version 10.0.0, current version
> 10.9.0)
> libyarn.1.dylib (compatibility version 1.0.0, current version 0.1.13)
> /usr/local/opt/json-c/lib/libjson-c.2.dylib (compatibility version 3.0.0,
> current version 3.1.0)
> /usr/local/opt/snappy/lib/libsnappy.1.dylib (compatibility version 5.0.0,
> current version 5.0.0)
> /usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version
> 1.0.5)
> /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
> 1213.0.0)
> /usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version
> 8.0.0)
> /System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos
> (compatibility version 5.0.0, current version 6.0.0)
> libdxltranslators.dylib (compatibility version 0.0.0, current version
> 0.0.0)
> /usr/local/opt/thrift/lib/libthrift-0.9.3.dylib (compatibility version
> 0.0.0, current version 0.0.0)
> This-MacBook-Pro:bin vagrant$ otool -L initdb
> initdb:
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
> 1226.10.1)
> {code}
> 10.11.3, postgres can start, but initdb failed:
> {code}
> This-MacBook-Pro:bin vagrant$ otool -L postgres
> postgres:
> /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version
> 120.1.0)
> libhdfs3.1.dylib (compatibility version 1.0.0, current version 2.2.30)
> /usr/lib/libxml2.2.dylib (compatibility version 10.0.0, current version
> 10.9.0)
> libyarn.1.dylib (compatibility version 1.0.0, current version 0.1.13)
> /usr/local/opt/json-c/lib/libjson-c.2.dylib (compatibility version 3.0.0,
> current version 3.1.0)
> /usr/local/opt/snappy/lib/libsnappy.1.dylib (compatibility version 5.0.0,
> current version 5.0.0)
> /usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version
> 1.0.5)
> /usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current version
> 0.9.8)
> /usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current
> version 0.9.8)
> /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
> 1226.10.1)
> /usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version
> 8.0.0)
> /System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos
> (compatibility version 5.0.0, current version 6.0.0)
> libdxltranslators.dylib (compatibility version 0.0.0, current version
> 0.0.0)
> /usr/local/opt/thrift/lib/libthrift-0.9.3.dylib (compatibility version
> 0.0.0, current version 0.0.0)
> [bin: xzhang{master}]$ otool -L initdb
> initdb:
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
> 1213.0.0)
> {code}
> In this case, there is a difference between the two OS regarding to the
> libSystem.B.dylib.
> Question is how to fix it? For example, how to change the libSystem.B.dylib
> to an older version? or, how to fix the postgres or initdb so that they works
> on the new OSX 10.11.3?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HAWQ-513) initdb.c failed on OSX 10.11.3 due to fgets error
[
https://issues.apache.org/jira/browse/HAWQ-513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15650385#comment-15650385
]
Hongxu Ma commented on HAWQ-513:
I had the same problems on osx 10.11.16, and solved it by this:
enter the mac "Recovery Mode", and set csrutil disable.
more detail in:
https://cwiki.apache.org/confluence/display/HAWQ/Build+and+Install
see the "Turning Off Rootless System Integrity Protection in OS X El Capitan
10.11+" section.
after do that, the "DYLD_LIBRARY_PATH" can work well, that's the key to cause
of this problem.
Try it.
> initdb.c failed on OSX 10.11.3 due to fgets error
> -
>
> Key: HAWQ-513
> URL: https://issues.apache.org/jira/browse/HAWQ-513
> Project: Apache HAWQ
> Issue Type: Bug
> Components: Unknown
>Reporter: xin zhang
>Assignee: Lei Chang
> Fix For: backlog
>
>
> we hit following strange issue on OSX 10.11.3:
> The error message in the initdb is:
> {code}
> 20160301:00:00:26:075823 hawq_init:This-MacBook-Pro:vagrant-[INFO]:-Start to
> init master node: 'localhost'
> sh: line 1: 76106 Trace/BPT trap: 5 "/usr/local/hawq/bin/postgres" -V
> 2> /dev/null
> fgets failure: Undefined error: 0
> The program "postgres" is needed by initdb but was either not found in the
> same directory as "/usr/local/hawq/bin/initdb" or failed unexpectedly.
> Check your installation; "postgres -V" may have more information.
> Master postgres initdb failed
> {code}
> We suspect the issue due to the newer version of the libSystem.B.dylib on OSX
> 10.11.3.
> Here is the details of the dependencies of `postgres` and `initdb`:
> 10.10.5, postgres can start, and initdb succeed:
> {code}
> [bin: xzhang{master}]$ otool -L postgres
> postgres:
> /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version
> 120.0.0)
> libhdfs3.1.dylib (compatibility version 1.0.0, current version 2.2.30)
> /usr/lib/libxml2.2.dylib (compatibility version 10.0.0, current version
> 10.9.0)
> libyarn.1.dylib (compatibility version 1.0.0, current version 0.1.13)
> /usr/local/opt/json-c/lib/libjson-c.2.dylib (compatibility version 3.0.0,
> current version 3.1.0)
> /usr/local/opt/snappy/lib/libsnappy.1.dylib (compatibility version 5.0.0,
> current version 5.0.0)
> /usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version
> 1.0.5)
> /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
> 1213.0.0)
> /usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version
> 8.0.0)
> /System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos
> (compatibility version 5.0.0, current version 6.0.0)
> libdxltranslators.dylib (compatibility version 0.0.0, current version
> 0.0.0)
> /usr/local/opt/thrift/lib/libthrift-0.9.3.dylib (compatibility version
> 0.0.0, current version 0.0.0)
> This-MacBook-Pro:bin vagrant$ otool -L initdb
> initdb:
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
> 1226.10.1)
> {code}
> 10.11.3, postgres can start, but initdb failed:
> {code}
> This-MacBook-Pro:bin vagrant$ otool -L postgres
> postgres:
> /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version
> 120.1.0)
> libhdfs3.1.dylib (compatibility version 1.0.0, current version 2.2.30)
> /usr/lib/libxml2.2.dylib (compatibility version 10.0.0, current version
> 10.9.0)
> libyarn.1.dylib (compatibility version 1.0.0, current version 0.1.13)
> /usr/local/opt/json-c/lib/libjson-c.2.dylib (compatibility version 3.0.0,
> current version 3.1.0)
> /usr/local/opt/snappy/lib/libsnappy.1.dylib (compatibility version 5.0.0,
> current version 5.0.0)
> /usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version
> 1.0.5)
> /usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current version
> 0.9.8)
> /usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current
> version 0.9.8)
> /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
> 1226.10.1)
> /usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version
> 8.0.0)
> /System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos
> (compatibility version 5.0.0, current version 6.0.0)
> libdxltranslators.dylib (compatibility version 0.0.0, current version
> 0.0.0)
> /usr/local/opt/thrift/lib/libthrift-0.9.3.dylib (compatibility version
> 0.0.0, current version 0.0.0)
> [bin: xzhang{master}]$ otool -L initdb
> initdb:
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
> 1213.0.0)
> {code}
> In this case, there is a difference
