[GitHub] incubator-hawq pull request #1234: HAWQ-1436. Implement ranger pulgin servic...

[stanlyxiang]

Github user stanlyxiang commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1234#discussion_r115410444
  
--- Diff: src/backend/libpq/rangerrest.c ---
@@ -383,58 +383,100 @@ static size_t write_callback(char *contents, size_t 
size, size_t nitems,
 static int call_ranger_rest(CURL_HANDLE curl_handle, const char* request)
 {
int ret = -1;
+   int retry = 2;
CURLcode res;
Assert(request != NULL);
 
/*
-* Re-initializes all options previously set on a specified CURL handle
-* to the default values. This puts back the handle to the same state as
-* it was in when it was just created with curl_easy_init.It does not
-* change the following information kept in the handle: live 
connections,
-* the Session ID cache, the DNS cache, the cookies and shares.
+* If master is talking with standby RPS, for every predefined interval
+* (controlled by a GUC hawq_rps_check_local_interval) it will check if 
local RPS works now.
 */
-   curl_easy_reset(curl_handle->curl_handle);
-   /* timeout: hard-coded temporarily and maybe should be a guc in future 
*/
-   curl_easy_setopt(curl_handle->curl_handle, CURLOPT_TIMEOUT, 30L);
-
-   /* specify URL to get */
-   StringInfoData tname;
-   initStringInfo();
-   appendStringInfo(, "http://;);
-   appendStringInfo(, "%s", master_addr_host);
-   appendStringInfo(, ":");
-   appendStringInfo(, "%d", rps_addr_port);
-   appendStringInfo(, "/");
-   appendStringInfo(, "%s", "rps");
-   curl_easy_setopt(curl_handle->curl_handle, CURLOPT_URL, tname.data);
-   pfree(tname.data);  
-
-   struct curl_slist *headers = NULL;
-   headers = curl_slist_append(headers, "Content-Type:application/json");
-   curl_easy_setopt(curl_handle->curl_handle, CURLOPT_HTTPHEADER, headers);
-
-   curl_easy_setopt(curl_handle->curl_handle, CURLOPT_POSTFIELDS,request);
-   /* send all data to this function  */
-   curl_easy_setopt(curl_handle->curl_handle, CURLOPT_WRITEFUNCTION, 
write_callback);
-   curl_easy_setopt(curl_handle->curl_handle, CURLOPT_WRITEDATA, (void 
*)curl_handle);
-
-   res = curl_easy_perform(curl_handle->curl_handle);
-   if(request_id == INT_MAX)
+   if (curl_handle->talkingWithStandby)
{
-   request_id = 0;
-   }
-   request_id++;
-   /* check for errors */
-   if(res != CURLE_OK)
-   {
-   elog(ERROR, "ranger plugin service from http://%s:%d/rps is 
unavailable : %s.\n",
-   master_addr_host, rps_addr_port, 
curl_easy_strerror(res));
+   uint64_t current_time = gettime_microsec();
+   if ((current_time - curl_handle->lastCheckTimestamp) > 
100LL * rps_check_local_interval)
+   {
+   curl_handle->talkingWithStandby = false;
+   curl_handle->lastCheckTimestamp = 0;
+   elog(RANGER_LOG,
+   "master has been talking to standby RPS 
for a predefined period, try switching to master RPS");
+   }
}
-   else
+
+   /*
+* try to connect standby's RPS if fail in connecting master's RPS
+*/
+   while(retry > 0 && ret != 0)
{
-   ret = 0;
-   elog(RANGER_LOG, "retrieved %d bytes data from ranger restful 
response.",
-   curl_handle->response.response_size);
+   /*
+* Re-initializes all options previously set on a specified 
CURL handle
+* to the default values. This puts back the handle to the same 
state as
+* it was in when it was just created with curl_easy_init.It 
does not
+* change the following information kept in the handle: live 
connections,
+* the Session ID cache, the DNS cache, the cookies and shares.
+*/
+   curl_easy_reset(curl_handle->curl_handle);
+   /* timeout: hard-coded temporarily and maybe should be a guc in 
future */
+   curl_easy_setopt(curl_handle->curl_handle, CURLOPT_TIMEOUT, 
30L);
+
+   /* specify URL to get */
+   StringInfoData tname;
+   initStringInfo();
+   appendStringInfo(, "http://;);
+   appendStringInfo(, "%s", 
curl_handle->talkingWithStandby?standby_addr_host:master_addr_host);
+   appendStringInfo(, ":");
+   appendStringInfo(, "%d", rps_addr_port);
+   appendStringInfo(, "/");
+   appendStringInfo(, "%s", "rps");
--- End diff --

These 2 lines can squash to 1 line . "appendStringInfo(, "/rps");"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. 

[GitHub] incubator-hawq pull request #1234: HAWQ-1436. Implement ranger pulgin servic...

[stanlyxiang]

Github user stanlyxiang commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/1234#discussion_r115410368
  
--- Diff: src/backend/libpq/rangerrest.c ---
@@ -383,58 +383,100 @@ static size_t write_callback(char *contents, size_t 
size, size_t nitems,
 static int call_ranger_rest(CURL_HANDLE curl_handle, const char* request)
 {
int ret = -1;
+   int retry = 2;
CURLcode res;
Assert(request != NULL);
 
/*
-* Re-initializes all options previously set on a specified CURL handle
-* to the default values. This puts back the handle to the same state as
-* it was in when it was just created with curl_easy_init.It does not
-* change the following information kept in the handle: live 
connections,
-* the Session ID cache, the DNS cache, the cookies and shares.
+* If master is talking with standby RPS, for every predefined interval
+* (controlled by a GUC hawq_rps_check_local_interval) it will check if 
local RPS works now.
 */
-   curl_easy_reset(curl_handle->curl_handle);
-   /* timeout: hard-coded temporarily and maybe should be a guc in future 
*/
-   curl_easy_setopt(curl_handle->curl_handle, CURLOPT_TIMEOUT, 30L);
-
-   /* specify URL to get */
-   StringInfoData tname;
-   initStringInfo();
-   appendStringInfo(, "http://;);
-   appendStringInfo(, "%s", master_addr_host);
-   appendStringInfo(, ":");
-   appendStringInfo(, "%d", rps_addr_port);
-   appendStringInfo(, "/");
-   appendStringInfo(, "%s", "rps");
-   curl_easy_setopt(curl_handle->curl_handle, CURLOPT_URL, tname.data);
-   pfree(tname.data);  
-
-   struct curl_slist *headers = NULL;
-   headers = curl_slist_append(headers, "Content-Type:application/json");
-   curl_easy_setopt(curl_handle->curl_handle, CURLOPT_HTTPHEADER, headers);
-
-   curl_easy_setopt(curl_handle->curl_handle, CURLOPT_POSTFIELDS,request);
-   /* send all data to this function  */
-   curl_easy_setopt(curl_handle->curl_handle, CURLOPT_WRITEFUNCTION, 
write_callback);
-   curl_easy_setopt(curl_handle->curl_handle, CURLOPT_WRITEDATA, (void 
*)curl_handle);
-
-   res = curl_easy_perform(curl_handle->curl_handle);
-   if(request_id == INT_MAX)
+   if (curl_handle->talkingWithStandby)
{
-   request_id = 0;
-   }
-   request_id++;
-   /* check for errors */
-   if(res != CURLE_OK)
-   {
-   elog(ERROR, "ranger plugin service from http://%s:%d/rps is 
unavailable : %s.\n",
-   master_addr_host, rps_addr_port, 
curl_easy_strerror(res));
+   uint64_t current_time = gettime_microsec();
+   if ((current_time - curl_handle->lastCheckTimestamp) > 
100LL * rps_check_local_interval)
+   {
+   curl_handle->talkingWithStandby = false;
+   curl_handle->lastCheckTimestamp = 0;
+   elog(RANGER_LOG,
+   "master has been talking to standby RPS 
for a predefined period, try switching to master RPS");
--- End diff --

we could print the rps_check_local_interval time instead of "predefined 
period".


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1234: HAWQ-1436. Implement ranger pulgin servic...

[linwen]

GitHub user linwen opened a pull request:

https://github.com/apache/incubator-hawq/pull/1234

HAWQ-1436. Implement ranger pulgin service High Availability.

1. master will the connect to standby RPS for policy search if RPS on 
master failed;
2. if master has been talking to standby RPS for a period(controlled by GUC 
hawq_rps_check_local_interval, 5 minutes by default), it will try to connect 
local RPS again.

Please review, thanks! 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/linwen/incubator-hawq hawq_1436

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/1234.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1234


commit baf088d7187de4c757c7d44e7b174615c84303b1
Author: Wen Lin 
Date:   2017-05-09T03:36:13Z

HAWQ-1436. Implement ranger pulgin service High Availability.
1. master will the connect to standby RPS for policy search if RPS on 
master failed;
2. if master has been talking to standby RPS for a period(controlled by GUC 
hawq_rps_check_local_interval, 5 minutes by default), it will try to connect 
local RPS again.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #1233: HAWQ-1454. Exclude certain jars from Ranger Plug...

[linwen]

Github user linwen commented on the issue:

https://github.com/apache/incubator-hawq/pull/1233
  
+1 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (HAWQ-1082) Do not send filter information on fragmenter call

[Vineet Goel (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001812#comment-16001812
 ] 

Vineet Goel commented on HAWQ-1082:
---

Looks like there are two "vineet goel". Sorry about assigning you the issue by 
mistake.

> Do not send filter information on fragmenter call
> -
>
> Key: HAWQ-1082
> URL: https://issues.apache.org/jira/browse/HAWQ-1082
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: PXF
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
> Fix For: backlog
>
>
> As for now HAWQ master evaluates and sends X-GP-FILTER value. Only one 
> fragmenter in PXF uses this value - HiveDataFragmenter, and loads metadata of 
> partitions which satisfy filter condition.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] incubator-hawq pull request #1218: HAWQ-1431: Do not use StatsAccessor when ...

[sansanichfb]

Github user sansanichfb closed the pull request at:

https://github.com/apache/incubator-hawq/pull/1218


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (HAWQ-1444) Need to replace gettimeofday() with clock_gettime() for related timeout checking code

[Paul Guo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Guo closed HAWQ-1444.
--
   Resolution: Not A Problem
Fix Version/s: (was: backlog)
   2.3.0.0-incubating

> Need to replace gettimeofday() with clock_gettime() for related timeout 
> checking code
> -
>
> Key: HAWQ-1444
> URL: https://issues.apache.org/jira/browse/HAWQ-1444
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Core
>Reporter: Paul Guo
>Assignee: Paul Guo
> Fix For: 2.3.0.0-incubating
>
>
> gettimeofday() could be affected by ntp kinda things. If using it for timeout 
> logic there could be wrong, e.g. time goes backwards. We could 
> clock_gettime() with CLOCK_MONOTONIC as an alternative.
> For some platforms/oses that does not have the support for clock_gettime(), 
> we can fall back to use gettimeofday().
> Note getCurrentTime() in code is a good example.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HAWQ-1444) Need to replace gettimeofday() with clock_gettime() for related timeout checking code

[Paul Guo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-1444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16000498#comment-16000498
 ] 

Paul Guo commented on HAWQ-1444:


I went through the callers of gettimeofday() in code and did not find obvious 
bad use of it since the following code change, so I'm closing this JIRA.

HAWQ-1439. tolerate system time being changed to earlier point when 
checking resource context timeout

> Need to replace gettimeofday() with clock_gettime() for related timeout 
> checking code
> -
>
> Key: HAWQ-1444
> URL: https://issues.apache.org/jira/browse/HAWQ-1444
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Core
>Reporter: Paul Guo
>Assignee: Paul Guo
> Fix For: backlog
>
>
> gettimeofday() could be affected by ntp kinda things. If using it for timeout 
> logic there could be wrong, e.g. time goes backwards. We could 
> clock_gettime() with CLOCK_MONOTONIC as an alternative.
> For some platforms/oses that does not have the support for clock_gettime(), 
> we can fall back to use gettimeofday().
> Note getCurrentTime() in code is a good example.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HAWQ-1444) Need to replace gettimeofday() with clock_gettime() for related timeout checking code

[Paul Guo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Guo reassigned HAWQ-1444:
--

Assignee: Paul Guo  (was: Ed Espino)

> Need to replace gettimeofday() with clock_gettime() for related timeout 
> checking code
> -
>
> Key: HAWQ-1444
> URL: https://issues.apache.org/jira/browse/HAWQ-1444
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Core
>Reporter: Paul Guo
>Assignee: Paul Guo
> Fix For: backlog
>
>
> gettimeofday() could be affected by ntp kinda things. If using it for timeout 
> logic there could be wrong, e.g. time goes backwards. We could 
> clock_gettime() with CLOCK_MONOTONIC as an alternative.
> For some platforms/oses that does not have the support for clock_gettime(), 
> we can fall back to use gettimeofday().
> Note getCurrentTime() in code is a good example.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HAWQ-1456) Copy RPS configuration files to standby in specific scenarios

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1456:
---

 Summary: Copy RPS configuration files to standby in specific 
scenarios
 Key: HAWQ-1456
 URL: https://issues.apache.org/jira/browse/HAWQ-1456
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: Security
Reporter: Hongxu Ma
Assignee: Ed Espino
 Fix For: backlog


Copy rps releated configuration files under file /usr/local/hawq/ranger/etc/ 
from HAWQ master to standby master in specific scenarios.

* scenario 1: hawq init standby
may does not has a standby before, so should copy latest confguration files to 
standby.

* scenario 2: execute enable-ranger-plugin.sh 
this script will modify rps confguration files, so should copy updated 
confguration files to standby.





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HAWQ-1456) Copy RPS configuration files to standby in specific scenarios

[Hongxu Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hongxu Ma reassigned HAWQ-1456:
---

Assignee: Hongxu Ma  (was: Ed Espino)

> Copy RPS configuration files to standby in specific scenarios
> -
>
> Key: HAWQ-1456
> URL: https://issues.apache.org/jira/browse/HAWQ-1456
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Security
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: backlog
>
>
> Copy rps releated configuration files under file /usr/local/hawq/ranger/etc/ 
> from HAWQ master to standby master in specific scenarios.
> * scenario 1: hawq init standby
> may does not has a standby before, so should copy latest confguration files 
> to standby.
> * scenario 2: execute enable-ranger-plugin.sh 
> this script will modify rps confguration files, so should copy updated 
> confguration files to standby.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)