[jira] [Created] (HAWQ-1450) New HAWQ executor with vectorization & possible code generation
Lei Chang created HAWQ-1450: --- Summary: New HAWQ executor with vectorization & possible code generation Key: HAWQ-1450 URL: https://issues.apache.org/jira/browse/HAWQ-1450 Project: Apache HAWQ Issue Type: New Feature Components: Query Execution Reporter: Lei Chang Assignee: Lei Chang Fix For: backlog Most HAWQ executor code is inherited from postgres & gpdb. Let's discuss how to build a new hawq executor with vectorization and possibly code generation. These optimization may potentially improve the query performance a lot. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HAWQ-1449) HAWQ start/stop cluster should be able to start/stop RPS on standby node
[ https://issues.apache.org/jira/browse/HAWQ-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiang Sheng reassigned HAWQ-1449: - Assignee: Xiang Sheng (was: Ed Espino) > HAWQ start/stop cluster should be able to start/stop RPS on standby node > > > Key: HAWQ-1449 > URL: https://issues.apache.org/jira/browse/HAWQ-1449 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Command Line Tools >Reporter: Xiang Sheng >Assignee: Xiang Sheng > Fix For: 2.3.0.0-incubating > > > To improve the robustness of HAWQ, we need to implement HA for RPS side to > avoid the case that single RPS failure may cause the failure of the whole > HAWQ cluster. > We need to implement start/stop RPS on standby master so that we can support > the HA of RPS side. > When Ranger HA is configured, hawq start cluster can also start RPS on > standby master; hawq stop cluster can also stop RPS on standby master. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HAWQ-1449) HAWQ start/stop cluster should be able to start/stop RPS on standby node
Xiang Sheng created HAWQ-1449: - Summary: HAWQ start/stop cluster should be able to start/stop RPS on standby node Key: HAWQ-1449 URL: https://issues.apache.org/jira/browse/HAWQ-1449 Project: Apache HAWQ Issue Type: Sub-task Components: Command Line Tools Reporter: Xiang Sheng Assignee: Ed Espino Fix For: 2.3.0.0-incubating To improve the robustness of HAWQ, we need to implement HA for RPS side to avoid the case that single RPS failure may cause the failure of the whole HAWQ cluster. We need to implement start/stop RPS on standby master so that we can support the HA of RPS side. When Ranger HA is configured, hawq start cluster can also start RPS on standby master; hawq stop cluster can also stop RPS on standby master. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HAWQ-1448) Postmaster process hung at recv () on segment
[ https://issues.apache.org/jira/browse/HAWQ-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992470#comment-15992470 ] Ming LI commented on HAWQ-1448: --- Instead of change connection type, here just change the hawq stop script, so that we keep the change minimal affect. > Postmaster process hung at recv () on segment > - > > Key: HAWQ-1448 > URL: https://issues.apache.org/jira/browse/HAWQ-1448 > Project: Apache HAWQ > Issue Type: Bug > Components: Dispatcher >Reporter: Ming LI >Assignee: Ming LI > Fix For: backlog > > > Some process hung for almost 2 hours before quit. > 4/13/17 8:13:36 AM PDT: Thread 1 (Thread 0x7f9c78eae920 (LWP 177517)): > 4/13/17 8:13:36 AM PDT: #0 0x00322180ec2c in recv () from > /lib64/libpthread.so.0 > 4/13/17 8:13:36 AM PDT: #1 0x007847e8 in secure_read () > 4/13/17 8:13:36 AM PDT: #2 0x00793735 in pq_recvbuf () > 4/13/17 8:13:36 AM PDT: #3 0x007939b9 in pq_getbyte () > 4/13/17 8:13:36 AM PDT: #4 0x008e39a4 in SocketBackend () > 4/13/17 8:13:36 AM PDT: #5 0x008e3ddc in ReadCommand () > 4/13/17 8:13:36 AM PDT: #6 0x008ea8c3 in PostgresMain () > 4/13/17 8:13:36 AM PDT: #7 0x008944ff in BackendRun () > 4/13/17 8:13:36 AM PDT: #8 0x0089391e in BackendStartup () > 4/13/17 8:13:36 AM PDT: #9 0x0088d99a in ServerLoop () > 4/13/17 8:13:36 AM PDT: #10 0x0088c9a7 in PostmasterMain () > 4/13/17 8:13:36 AM PDT: #11 0x007a9d63 in main () > 4/13/17 8:13:36 AM PDT: - > All postgres processes on all host are quit, only postmaster on seg3 hung. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HAWQ-1448) Postmaster process hung at recv () on segment
[
https://issues.apache.org/jira/browse/HAWQ-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992314#comment-15992314
]
Ming LI edited comment on HAWQ-1448 at 5/2/17 7:16 AM:
---
Below is the related log segmentdd/pg_log/hawq-2017-04-13_071837.csv on seg3
{code}
2017-04-13 08:08:15.998769
PDT,,,p23303,th20286610240,,,seg-1,"LOG","0","received smart
shutdown request",,,0,,"postmaster.c",3447,
...
2017-04-13 08:08:43.228325
PDT,,,p23310,th20286610240,,,seg-1,"WARNING","01000","FD 4 having
errors raised. errno 111",,,0,,"rmcomm_AsyncComm.c",188,
2017-04-13 08:08:43.228347
PDT,,,p23310,th20286610240,,,seg-1,"WARNING","01000","Resource
manager socket connect has error raised.",,,0,,"rmcomm_Connect.c",100,
2017-04-13 08:08:43.228364
PDT,,,p23310,th20286610240,,,seg-1,"WARNING","01000","Segment's
resource manager sending IMAlive message switches from master to
standby",,,0,,"rmcomm_RMSEG2RM.c",168,
2017-04-13 08:08:43.228383
PDT,,,p23310,th20286610240,,,seg-1,"LOG","0","segment will send
heart-beat to standby from now on",,,0,,"resourcemanager_RMSEG.c",285,
2017-04-13 08:09:13.280237
PDT,,,p23310,th20286610240,,,seg-1,"LOG","0","Resource manager
discovered local host IPv4 address 127.0.0.1",,,0,,"network_utils.c",210,
2017-04-13 08:09:13.280294
PDT,,,p23310,th20286610240,,,seg-1,"LOG","0","Resource manager
discovered local host IPv4 address 10.32.34.6",,,0,,"network_utils.c",210,
... LOOP THESE 6 LINES
...
2017-04-13 10:03:55.869252
PDT,,,p23310,th20286610240,,,seg-1,"WARNING","01000","FD 4 having
errors raised. errno 111",,,0,,"rmcomm_AsyncComm.c",188,
2017-04-13 10:03:55.869277
PDT,,,p23310,th20286610240,,,seg-1,"WARNING","01000","Resource
manager socket connect has error raised.",,,0,,"rmcomm_Connect.c",100,
2017-04-13 10:03:55.869293
PDT,,,p23310,th20286610240,,,seg-1,"WARNING","01000","Segment's
resource manager sending IMAlive message switches from master to
standby",,,0,,"rmcomm_RMSEG2RM.c",168,
2017-04-13 10:03:55.869323
PDT,,,p23310,th20286610240,,,seg-1,"LOG","0","segment will send
heart-beat to standby from now on",,,0,,"resourcemanager_RMSEG.c",285,
2017-04-13 10:04:01.249461
PDT,"hawqsuperuser","olap_winowerr",p177517,th2028661024,"10.32.35.251","45247",2017-04-13
08:04:00 PDT,0,con4354,,seg6,"LOG","08006","could not receive data from
client: Connection reset by peer",,,0,,"pqcomm.c",842,
2017-04-13 10:04:01.249522
PDT,"hawqsuperuser","olap_winowerr",p177517,th2028661024,"10.32.35.251","45247",2017-04-13
08:04:00 PDT,0,con4354,,seg6,"LOG","08P01","unexpected EOF on client
connection",,,0,,"postgres.c",443,
2017-04-13 10:04:01.252964
PDT,,,p23310,th20286610240,,,seg-1,"LOG","0","Segment RM
exits.",,,0,,"resourcemanager.c",347,
2017-04-13 10:04:01.253027
PDT,,,p23310,th20286610240,,,seg-1,"LOG","0","Clean up handler
in message server is called.",,,0,,"rmcomm_MessageServer.c",105,
2017-04-13 10:04:01.255779
PDT,,,p23308,th20286610240,,,seg-1,"LOG","0","shutting
down",,,0,,"xlog.c",7861,
2017-04-13 10:04:01.257902
PDT,,,p23308,th20286610240,,,seg-1,"LOG","0","database system
is shut down",,,0,,"xlog.c",7882,
{code}
was (Author: mli):
Below is the related log
{code}
2017-04-13 08:08:15.998769
PDT,,,p23303,th20286610240,,,seg-1,"LOG","0","received smart
shutdown request",,,0,,"postmaster.c",3447,
...
2017-04-13 08:08:43.228325
PDT,,,p23310,th20286610240,,,seg-1,"WARNING","01000","FD 4 having
errors raised. errno 111",,,0,,"rmcomm_AsyncComm.c",188,
2017-04-13 08:08:43.228347
PDT,,,p23310,th20286610240,,,seg-1,"WARNING","01000","Resource
manager socket connect has error raised.",,,0,,"rmcomm_Connect.c",100,
2017-04-13 08:08:43.228364
PDT,,,p23310,th20286610240,,,seg-1,"WARNING","01000","Segment's
resource manager sending IMAlive message switches from master to
standby",,,0,,"rmcomm_RMSEG2RM.c",168,
2017-04-13 08:08:43.228383
PDT,,,p23310,th20286610240,,,seg-1,"LOG","0","segment will send
heart-beat to standby from now on",,,0,,"resourcemanager_RMSEG.c",285,
2017-04-13 08:09:13.280237
PDT,,,p23310,th20286610240,,,seg-1,"LOG","0","Resource manager
discovered local host IPv4 address 127.0.0.1",,,0,,"network_utils.c",210,
2017-04-13 08:09:13.280294
PDT,,,p23310,th20286610240,,,seg-1,"LOG","0","Resource manager
discovered local host IPv4 address 10.32.34.6",,,0,,"network_utils.c",210,
... LOOP THESE 6 LINES
[jira] [Updated] (HAWQ-1448) Postmaster process hung at recv () on segment
[ https://issues.apache.org/jira/browse/HAWQ-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming LI updated HAWQ-1448: -- Description: Some process hung for almost 2 hours before quit. 4/13/17 8:13:36 AM PDT: Thread 1 (Thread 0x7f9c78eae920 (LWP 177517)): 4/13/17 8:13:36 AM PDT: #0 0x00322180ec2c in recv () from /lib64/libpthread.so.0 4/13/17 8:13:36 AM PDT: #1 0x007847e8 in secure_read () 4/13/17 8:13:36 AM PDT: #2 0x00793735 in pq_recvbuf () 4/13/17 8:13:36 AM PDT: #3 0x007939b9 in pq_getbyte () 4/13/17 8:13:36 AM PDT: #4 0x008e39a4 in SocketBackend () 4/13/17 8:13:36 AM PDT: #5 0x008e3ddc in ReadCommand () 4/13/17 8:13:36 AM PDT: #6 0x008ea8c3 in PostgresMain () 4/13/17 8:13:36 AM PDT: #7 0x008944ff in BackendRun () 4/13/17 8:13:36 AM PDT: #8 0x0089391e in BackendStartup () 4/13/17 8:13:36 AM PDT: #9 0x0088d99a in ServerLoop () 4/13/17 8:13:36 AM PDT: #10 0x0088c9a7 in PostmasterMain () 4/13/17 8:13:36 AM PDT: #11 0x007a9d63 in main () 4/13/17 8:13:36 AM PDT: - All postgres processes on all host are quit, only postmaster on seg3 hung. was: Some process hung for almost 2 hours before quit. 4/13/17 8:13:36 AM PDT: Thread 1 (Thread 0x7f9c78eae920 (LWP 177517)): 4/13/17 8:13:36 AM PDT: #0 0x00322180ec2c in recv () from /lib64/libpthread.so.0 4/13/17 8:13:36 AM PDT: #1 0x007847e8 in secure_read () 4/13/17 8:13:36 AM PDT: #2 0x00793735 in pq_recvbuf () 4/13/17 8:13:36 AM PDT: #3 0x007939b9 in pq_getbyte () 4/13/17 8:13:36 AM PDT: #4 0x008e39a4 in SocketBackend () 4/13/17 8:13:36 AM PDT: #5 0x008e3ddc in ReadCommand () 4/13/17 8:13:36 AM PDT: #6 0x008ea8c3 in PostgresMain () 4/13/17 8:13:36 AM PDT: #7 0x008944ff in BackendRun () 4/13/17 8:13:36 AM PDT: #8 0x0089391e in BackendStartup () 4/13/17 8:13:36 AM PDT: #9 0x0088d99a in ServerLoop () 4/13/17 8:13:36 AM PDT: #10 0x0088c9a7 in PostmasterMain () 4/13/17 8:13:36 AM PDT: #11 0x007a9d63 in main () 4/13/17 8:13:36 AM PDT: - All postgres processes on all host are quit, > Postmaster process hung at recv () on segment > - > > Key: HAWQ-1448 > URL: https://issues.apache.org/jira/browse/HAWQ-1448 > Project: Apache HAWQ > Issue Type: Bug > Components: Dispatcher >Reporter: Ming LI >Assignee: Ming LI > Fix For: backlog > > > Some process hung for almost 2 hours before quit. > 4/13/17 8:13:36 AM PDT: Thread 1 (Thread 0x7f9c78eae920 (LWP 177517)): > 4/13/17 8:13:36 AM PDT: #0 0x00322180ec2c in recv () from > /lib64/libpthread.so.0 > 4/13/17 8:13:36 AM PDT: #1 0x007847e8 in secure_read () > 4/13/17 8:13:36 AM PDT: #2 0x00793735 in pq_recvbuf () > 4/13/17 8:13:36 AM PDT: #3 0x007939b9 in pq_getbyte () > 4/13/17 8:13:36 AM PDT: #4 0x008e39a4 in SocketBackend () > 4/13/17 8:13:36 AM PDT: #5 0x008e3ddc in ReadCommand () > 4/13/17 8:13:36 AM PDT: #6 0x008ea8c3 in PostgresMain () > 4/13/17 8:13:36 AM PDT: #7 0x008944ff in BackendRun () > 4/13/17 8:13:36 AM PDT: #8 0x0089391e in BackendStartup () > 4/13/17 8:13:36 AM PDT: #9 0x0088d99a in ServerLoop () > 4/13/17 8:13:36 AM PDT: #10 0x0088c9a7 in PostmasterMain () > 4/13/17 8:13:36 AM PDT: #11 0x007a9d63 in main () > 4/13/17 8:13:36 AM PDT: - > All postgres processes on all host are quit, only postmaster on seg3 hung. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HAWQ-1443) Implement Ranger lookup for HAWQ with Kerberos enabled.
[ https://issues.apache.org/jira/browse/HAWQ-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992459#comment-15992459 ] Xiang Sheng commented on HAWQ-1443: --- For the lookup access from ranger admin to HAWQ, we could use kerberos to securitised it. Since HAWQ support kerberos authentication and Ranger admin lookup using JDBC connection, so we could use kerberos through HAWQ with kerberos for JDBC. As Hubert wrote above, we have 2 works to do: 1. update the ranger ui through update the ranger-servicedef-hawq.json file. Add an drop-down box for authentication method choose and an input box for principal name. 2. update rps lookup logical code. change the jdbc string to the required string that HAWQ kerberos supports. > Implement Ranger lookup for HAWQ with Kerberos enabled. > --- > > Key: HAWQ-1443 > URL: https://issues.apache.org/jira/browse/HAWQ-1443 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: backlog > > Attachments: Kerberos Support for Ranger Lookup HAWQ.pdf > > > When add a HAWQ service in Ranger, we also need to configure Ranger look up > service for HAWQ. Lookup service can be done through JDBC with username and > password. But It cannot support Kerberos authentication currently. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HAWQ-1443) Implement Ranger lookup for HAWQ with Kerberos enabled.
[ https://issues.apache.org/jira/browse/HAWQ-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiang Sheng updated HAWQ-1443: -- Attachment: Kerberos Support for Ranger Lookup HAWQ.pdf > Implement Ranger lookup for HAWQ with Kerberos enabled. > --- > > Key: HAWQ-1443 > URL: https://issues.apache.org/jira/browse/HAWQ-1443 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Security >Reporter: Hubert Zhang >Assignee: Hubert Zhang > Fix For: backlog > > Attachments: Kerberos Support for Ranger Lookup HAWQ.pdf > > > When add a HAWQ service in Ranger, we also need to configure Ranger look up > service for HAWQ. Lookup service can be done through JDBC with username and > password. But It cannot support Kerberos authentication currently. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
