[GitHub] incubator-hawq issue #1079: HAWQ-1256. Enhance libcurl connection to Ranger ...
Github user stanlyxiang commented on the issue: https://github.com/apache/incubator-hawq/pull/1079 merged into master. closed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1078: HAWQ-1258. segment resource manager does ...
Github user paul-guo- commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1078#discussion_r95526548 --- Diff: src/backend/utils/misc/guc.c --- @@ -8200,7 +8200,7 @@ static struct config_string ConfigureNamesString[] = NULL }, &standby_addr_host, - "localhost", NULL, NULL + "none", NULL, NULL --- End diff -- OK. Agree, we could start another JIRA to track the hostname "none" issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1078: HAWQ-1258. segment resource manager does not swi...
Github user paul-guo- commented on the issue: https://github.com/apache/incubator-hawq/pull/1078 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1078: HAWQ-1258. segment resource manager does ...
Github user radarwave commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1078#discussion_r95525426 --- Diff: src/backend/utils/misc/guc.c --- @@ -8200,7 +8200,7 @@ static struct config_string ConfigureNamesString[] = NULL }, &standby_addr_host, - "localhost", NULL, NULL + "none", NULL, NULL --- End diff -- Yeah, it's not so good to use 'none' to indicate there is no standby. But I don't think use '' or ' ' or ' ' are good too. Still let user feel confusion. Maybe we can add a new guc to indicate if we have a standby master defined, the value can be yes/no. Better to start a new Jira to talk about this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1079: HAWQ-1256. Enhance libcurl connection to Ranger ...
Github user linwen commented on the issue: https://github.com/apache/incubator-hawq/pull/1079 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1079: HAWQ-1256. Enhance libcurl connection to ...
Github user stanlyxiang commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1079#discussion_r95524672
--- Diff: src/include/utils/rangerrest.h ---
@@ -60,9 +61,12 @@ typedef struct curl_context_t
{
char* buffer;
int size;
+int buffer_size;
--- End diff --
fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1079: HAWQ-1256. Enhance libcurl connection to ...
Github user stanlyxiang commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1079#discussion_r95524634 --- Diff: src/backend/tcop/postgres.c --- @@ -133,6 +135,7 @@ extern char *optarg; extern char *savedSeqServerHost; extern int savedSeqServerPort; +struct curl_context_t curl_context; --- End diff -- fixed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1079: HAWQ-1256. Enhance libcurl connection to ...
Github user linwen commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1079#discussion_r95521589 --- Diff: src/backend/tcop/postgres.c --- @@ -133,6 +135,7 @@ extern char *optarg; extern char *savedSeqServerHost; extern int savedSeqServerPort; +struct curl_context_t curl_context; --- End diff -- The name of this global variable "curl_context", seems ambiguous, code readers can't know what is this curl context used for. maybe "curl_context_rps" or "curl_context_ranger" is better, indicates this curl context is used for Ranger Plugin Service. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1079: HAWQ-1256. Enhance libcurl connection to ...
Github user linwen commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1079#discussion_r95519578 --- Diff: src/backend/tcop/postgres.c --- @@ -5314,6 +5337,24 @@ PostgresMain(int argc, char *argv[], const char *username) return 1; /* keep compiler quiet */ } +static void +curl_finalize(int code, Datum arg __MAYBE_UNUSED) --- End diff -- Make sure the initialization and finalization for libcurl context won't be called on standby and segment side. The libcurl connection to RPS is triggered from master side . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1079: HAWQ-1256. Enhance libcurl connection to ...
Github user linwen commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1079#discussion_r95521692
--- Diff: src/include/utils/rangerrest.h ---
@@ -60,9 +61,12 @@ typedef struct curl_context_t
{
char* buffer;
int size;
+int buffer_size;
--- End diff --
The name size and buffer_size are ambiguous, should have a more clear name.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1078: HAWQ-1258. segment resource manager does ...
Github user jiny2 commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1078#discussion_r95522787 --- Diff: src/backend/utils/misc/guc.c --- @@ -8200,7 +8200,7 @@ static struct config_string ConfigureNamesString[] = NULL }, &standby_addr_host, - "localhost", NULL, NULL + "none", NULL, NULL --- End diff -- No, segment rm process does not care what are saved in python code. So, it only knows there is a standby host named "none". So deleting from hawq_dict does not help. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1081: HAWQ-1257. Prompt all tables which user doesn't ...
Github user interma commented on the issue: https://github.com/apache/incubator-hawq/pull/1081 My main modified is commented overhead. @paul-guo- @stanlyxiang @linwen @zhangh43 pls review, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1079: HAWQ-1256. Enhance libcurl connection to Ranger ...
Github user zhangh43 commented on the issue: https://github.com/apache/incubator-hawq/pull/1079 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1081: HAWQ-1257. Prompt all tables which user d...
Github user interma commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1081#discussion_r95522001
--- Diff: src/backend/parser/parse_relation.c ---
@@ -2714,12 +2714,12 @@ warnAutoRange(ParseState *pstate, RangeVar
*relation, int location)
void
ExecCheckRTPerms(List *rangeTable)
{
- if (enable_ranger &&
!fallBackToNativeChecks(ACL_KIND_CLASS,rangeTable,GetUserId()))
- {
-if(rangeTable!=NULL)
- ExecCheckRTPermsWithRanger(rangeTable);
-return;
- }
+ if (enable_ranger &&
!fallBackToNativeChecks(ACL_KIND_CLASS,rangeTable,GetUserId()))
+ {
+ if(rangeTable!=NULL)
+ ExecCheckRTPermsWithRanger(rangeTable);
+ return;
+ }
--- End diff --
Only the two function used 2space ident, and other used Tab.
So I unify them to Tab.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1081: HAWQ-1257. Prompt all tables which user d...
Github user interma commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1081#discussion_r95522139
--- Diff: src/backend/libpq/rangerrest.c ---
@@ -68,51 +69,99 @@ static void getClientIP(char *remote_host)
}
}
-RangerACLResult parse_ranger_response(char* buffer)
+/*
+ * parse ranger response
+ * @param buffer ranger response
+ * @param result_list List of RangerPrivilegeResults
+ * @return 0 parse success; -1 other error
+ */
+static int parse_ranger_response(char* buffer, List *result_list)
{
if (buffer == NULL || strlen(buffer) == 0)
- return RANGERCHECK_UNKNOWN;
+ return -1;
--- End diff --
I modified this function's return value.
The acl check results will be stored in the param `result_list`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1081: HAWQ-1257. Prompt all tables which user d...
Github user interma commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1081#discussion_r95522472
--- Diff: src/include/utils/rangerrest.h ---
@@ -90,9 +97,6 @@ typedef struct RangerRequestJsonArgs {
bool isAll;
} RangerRequestJsonArgs;
-RangerACLResult parse_ranger_response(char *);
-json_object *create_ranger_request_json(List *);
-int call_ranger_rest(CURL_HANDLE curl_handle, const char *request);
-extern int check_privilege_from_ranger(List *);
+int check_privilege_from_ranger(List *request_list, List *result_list);
--- End diff --
Hide no outer used function.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1081: HAWQ-1257. Prompt all tables which user d...
Github user interma commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1081#discussion_r95522261
--- Diff: src/backend/libpq/rangerrest.c ---
@@ -68,51 +69,99 @@ static void getClientIP(char *remote_host)
}
}
-RangerACLResult parse_ranger_response(char* buffer)
+/*
+ * parse ranger response
+ * @param buffer ranger response
+ * @param result_list List of RangerPrivilegeResults
+ * @return 0 parse success; -1 other error
+ */
+static int parse_ranger_response(char* buffer, List *result_list)
{
if (buffer == NULL || strlen(buffer) == 0)
- return RANGERCHECK_UNKNOWN;
+ return -1;
elog(LOG, "read from Ranger Restful API: %s", buffer);
struct json_object *response = json_tokener_parse(buffer);
if (response == NULL)
{
elog(WARNING, "json_tokener_parse failed");
- return RANGERCHECK_NO_PRIV;
+ return -1;
}
struct json_object *accessObj = NULL;
if (!json_object_object_get_ex(response, "access", &accessObj))
{
elog(WARNING, "get json access field failed");
- return RANGERCHECK_NO_PRIV;
+ return -1;
}
int arraylen = json_object_array_length(accessObj);
elog(LOG, "Array Length: %d",arraylen);
-
- // here should return which table's acl check failed in future.
+
for (int i=0; i< arraylen; i++){
struct json_object *jvalue = NULL;
struct json_object *jallow = NULL;
+ struct json_object *jresource = NULL;
+ struct json_object *jprivilege = NULL;
jvalue = json_object_array_get_idx(accessObj, i);
+ if (jvalue == NULL)
+ return -1;
if (!json_object_object_get_ex(jvalue, "allowed", &jallow))
- {
- return RANGERCHECK_NO_PRIV;
- }
- json_bool result = json_object_get_boolean(jallow);
- if(result != 1){
- return RANGERCHECK_NO_PRIV;
+ return -1;
+ if (!json_object_object_get_ex(jvalue, "resource", &jresource))
+ return -1;
+ if (!json_object_object_get_ex(jvalue, "privileges",
&jprivilege))
+ return -1;
+
+ json_bool ok = json_object_get_boolean(jallow);
+
+ const char *resource_str = json_object_get_string(jresource);
+ const char *privilege_str = json_object_get_string(jprivilege);
+ uint32 resource_sign = string_hash(resource_str,
strlen(resource_str));
+ uint32 privilege_sign = string_hash(privilege_str,
strlen(privilege_str));
+ elog(DEBUG3, "ranger reponse access sign, resource_str:%s,
privilege_str:%s",
+ resource_str, privilege_str);
+
+ ListCell *result;
+ /* get each resource result by use sign */
+ foreach(result, result_list) {
+ /* loop find is enough for performence*/
+ RangerPrivilegeResults *result_ptr =
(RangerPrivilegeResults *) lfirst(result);
+ if (result_ptr->resource_sign != resource_sign ||
result_ptr->privilege_sign != privilege_sign)
+ continue;
+
+ if (ok == 1)
+ result_ptr->result = RANGERCHECK_OK;
+ else
+ result_ptr->result = RANGERCHECK_NO_PRIV;
--- End diff --
Match response sign to request, and judge a **pair**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1081: HAWQ-1257. Prompt all tables which user d...
Github user interma commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1081#discussion_r95522410
--- Diff: src/backend/libpq/rangerrest.c ---
@@ -247,12 +299,27 @@ json_object *create_ranger_request_json(List *args)
ListCell *cell;
foreach(cell, arg_ptr->actions)
{
- json_object* jaction = json_object_new_string((char
*)cell->data.ptr_value);
+ /* need more normalization in future */
--- End diff --
If the json string changed in future, we should do more normalization works.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1081: HAWQ-1257. Prompt all tables which user d...
Github user interma commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1081#discussion_r95521660
--- Diff: src/include/utils/rangerrest.h ---
@@ -80,6 +80,13 @@ typedef struct RangerPrivilegeResults
{
RangerACLResult result;
Oid relOid;
+
+ /*
+ * string_hash of access[i] field of ranger request
+ * use the sigh to identify each resource result
+ */
+ uint32 resource_sign;
+ uint32 privilege_sign;
--- End diff --
On a batch ranger acl check request/response:
The acl check result order isn't the same as request order.
When reponse's resource&privilege field is same as request, then we
consider them is a pair.
So we need storage resource&privilege sign in here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1079: HAWQ-1256. Enhance libcurl connection to ...
Github user stanlyxiang commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1079#discussion_r95522271
--- Diff: src/backend/libpq/rangerrest.c ---
@@ -284,7 +284,7 @@ static size_t write_callback(char *contents, size_t
size, size_t nitems,
Assert(curl != NULL);
elog(DEBUG3, "response size is %d. response buffer size is %d.",
curl->response.size, curl->response.buffer_size);
- if(curl->response.size + realsize >= curl->response.buffer_size)
+ while(curl->response.size + realsize >= curl->response.buffer_size)
{
/*
--- End diff --
updated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1079: HAWQ-1256. Enhance libcurl connection to ...
Github user stanlyxiang commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1079#discussion_r95522252 --- Diff: src/backend/tcop/postgres.c --- @@ -4641,11 +4641,12 @@ PostgresMain(int argc, char *argv[], const char *username) /* cleanup curl stuff */ /* no need to cleanup curl_handle since it's null. just cleanup curl global.*/ curl_global_cleanup(); + elog(ERROR, "init curl handle failed."); } curl_context.hasInited = true; curl_context.response.buffer = palloc0(CURL_RES_BUFFER_SIZE); curl_context.response.buffer_size = CURL_RES_BUFFER_SIZE; - elog(LOG, "when enable ranger, init global struct for privileges check."); + elog(DEBUG3, "init global curl context for privileges check."); --- End diff -- fixed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1081: HAWQ-1257. Prompt all tables which user doesn't ...
Github user paul-guo- commented on the issue: https://github.com/apache/incubator-hawq/pull/1081 I think some coding style (please follow the pg coding style) should be uniform. At least, e.g.: 1) do not mix // and /* 2) do not have the function return type be on the same line of the declaration. List * deconstruct_jointree(PlannerInfo *root) 3) Please make sure whether ereport instead of elog should be used. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1081: HAWQ-1257. Prompt all tables which user d...
Github user interma commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1081#discussion_r95521591
--- Diff: src/include/utils/rangerrest.h ---
@@ -80,6 +80,13 @@ typedef struct RangerPrivilegeResults
{
RangerACLResult result;
Oid relOid;
+
+ /*
--- End diff --
On a *batch* ranger acl check request/response:
The acl check result order isn't the same as request.
When reponse's resource&privilege field is same as request, then we
consider them is a pair.
So we need storage resource&privilege sign in here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Closed] (HAWQ-1257) If user doesn't have privileges on certain objects, need return user which specific table he doesn't have right.
[
https://issues.apache.org/jira/browse/HAWQ-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongxu Ma closed HAWQ-1257.
---
Resolution: Fixed
already opened a PR: https://github.com/apache/incubator-hawq/pull/1081
> If user doesn't have privileges on certain objects, need return user which
> specific table he doesn't have right.
> -
>
> Key: HAWQ-1257
> URL: https://issues.apache.org/jira/browse/HAWQ-1257
> Project: Apache HAWQ
> Issue Type: Sub-task
> Components: Security
>Reporter: Lili Ma
>Assignee: Hongxu Ma
> Fix For: 2.2.0.0-incubating
>
>
> If user doesn't have privileges on certain objects, need return user all the
> objects he doesn't have right, to avoid the user modify one privilege, and
> then find another privilege constraint, and then another... which may bother
> the user a lot.
> For example:
> user didn't have the rights of t1 and t2.
> {code}
> postgres=> select * from test_sa.t1 left join test_sa.t2 on
> test_sa.t1.i=test_sa.t2.i;
> ERROR: permission denied for relation t1
> {code}
> We wish to prompt user didn't have the rights of t2 also.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] incubator-hawq pull request #1081: HAWQ-1257. Prompt all tables which user d...
GitHub user interma opened a pull request: https://github.com/apache/incubator-hawq/pull/1081 HAWQ-1257. Prompt all tables which user doesn't have right once You can merge this pull request into a Git repository by running: $ git pull https://github.com/interma/interma-hawq HAWQ-1257 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1081.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1081 commit baef5a3925b370270dbb8d7ed659df705b422e89 Author: interma Date: 2017-01-11T06:51:21Z HAWQ-1257. Prompt all tables which user doesn't have right once --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1080: HAWQ-1257. Prompt all tables which user doesn't ...
Github user interma commented on the issue: https://github.com/apache/incubator-hawq/pull/1080 wcl's commit also be included, very strange, I will open a new PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1080: HAWQ-1257. Prompt all tables which user d...
Github user interma closed the pull request at: https://github.com/apache/incubator-hawq/pull/1080 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1080: HAWQ-1257. Prompt all tables which user d...
GitHub user interma opened a pull request: https://github.com/apache/incubator-hawq/pull/1080 HAWQ-1257. Prompt all tables which user doesn't have right once You can merge this pull request into a Git repository by running: $ git pull https://github.com/interma/interma-hawq HAWQ-1257 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1080.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1080 commit b9e8b1ccd0e5d8f3453df92eebc9a3c18ed37e9e Author: Chunling Wang Date: 2017-01-09T06:35:11Z HAWQ-1249. Don't do ACL checks on segments commit df35712a1b88fd4999a8b1ef0ba5ff7fee2a39fd Author: interma Date: 2017-01-09T09:02:29Z HAWQ-1257. Prompt all tables which user doesn't have right once --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1078: HAWQ-1258. segment resource manager does ...
Github user paul-guo- commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1078#discussion_r95515076 --- Diff: src/backend/utils/misc/guc.c --- @@ -8200,7 +8200,7 @@ static struct config_string ConfigureNamesString[] = NULL }, &standby_addr_host, - "localhost", NULL, NULL + "none", NULL, NULL --- End diff -- I suspect this does not affect your code. See the code below, def get_all_values(self): if 'hawq_standby_address_host' in self.hawq_dict: if self.hawq_dict['hawq_standby_address_host'].lower() in ['none', '', 'localhost']: del self.hawq_dict['hawq_standby_address_host'] However hawq_ctl should remove the "none" case, I think. @radarwave Please confirm the logic. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1079: HAWQ-1256. Enhance libcurl connection to ...
Github user interma commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1079#discussion_r95514955
--- Diff: src/backend/tcop/postgres.c ---
@@ -5314,6 +5336,24 @@ PostgresMain(int argc, char *argv[], const char
*username)
return 1; /* keep compiler quiet
*/
}
+static void
+curl_finalize(int code, Datum arg __attribute__((unused)))
+{
+ if (curl_context.hasInited)
+ {
+ if (curl_context.response.buffer != NULL) {
+ pfree(curl_context.response.buffer);
+ }
+ /* cleanup curl stuff */
+ if (curl_context.curl_handle) {
+ curl_easy_cleanup(curl_context.curl_handle);
+ }
+ /* we're done with libcurl, so clean it up */
+ curl_global_cleanup();
+ curl_context.hasInited = false;
+ elog(LOG, "finalize the global struct for curl handle
context.");
--- End diff --
got it, TKS all!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1079: HAWQ-1256. Enhance libcurl connection to ...
Github user interma commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1079#discussion_r95514836
--- Diff: src/backend/libpq/rangerrest.c ---
@@ -284,7 +284,7 @@ static size_t write_callback(char *contents, size_t
size, size_t nitems,
Assert(curl != NULL);
elog(DEBUG3, "response size is %d. response buffer size is %d.",
curl->response.size, curl->response.buffer_size);
- if(curl->response.size + realsize >= curl->response.buffer_size)
+ while(curl->response.size + realsize >= curl->response.buffer_size)
{
/*
--- End diff --
Take `repalloc()` out of while and *do one alloc* after determining the
buffer size is better~
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1078: HAWQ-1258. segment resource manager does ...
Github user jiny2 commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1078#discussion_r95513917 --- Diff: src/backend/utils/misc/guc.c --- @@ -8200,7 +8200,7 @@ static struct config_string ConfigureNamesString[] = NULL }, &standby_addr_host, - "localhost", NULL, NULL + "none", NULL, NULL --- End diff -- So that's why we need another fix if necessary. the fix should fix script and code together. @paul-guo- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1078: HAWQ-1258. segment resource manager does ...
Github user jiny2 commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1078#discussion_r95513882 --- Diff: src/backend/utils/misc/guc.c --- @@ -8200,7 +8200,7 @@ static struct config_string ConfigureNamesString[] = NULL }, &standby_addr_host, - "localhost", NULL, NULL + "none", NULL, NULL --- End diff -- This will cause behavior inconsistency, when "none" is set script thinks there is no standby, but rm segment will try to resolve "none", which is not necessary. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1079: HAWQ-1256. Enhance libcurl connection to ...
Github user linwen commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1079#discussion_r95513132 --- Diff: src/backend/tcop/postgres.c --- @@ -5351,7 +5352,7 @@ curl_finalize(int code, Datum arg __attribute__((unused))) /* we're done with libcurl, so clean it up */ curl_global_cleanup(); curl_context.hasInited = false; - elog(LOG, "finalize the global struct for curl handle context."); + elog(DEBUG3, "finalize the global struct for curl handle context."); --- End diff -- This line for finalize should correspond to line:4649, "finalize the global curl context for privileges check." So that initialize and finalize appear in pair. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1079: HAWQ-1256. Enhance libcurl connection to ...
Github user linwen commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1079#discussion_r95513053 --- Diff: src/backend/tcop/postgres.c --- @@ -4641,11 +4641,12 @@ PostgresMain(int argc, char *argv[], const char *username) /* cleanup curl stuff */ /* no need to cleanup curl_handle since it's null. just cleanup curl global.*/ curl_global_cleanup(); + elog(ERROR, "init curl handle failed."); } curl_context.hasInited = true; curl_context.response.buffer = palloc0(CURL_RES_BUFFER_SIZE); curl_context.response.buffer_size = CURL_RES_BUFFER_SIZE; - elog(LOG, "when enable ranger, init global struct for privileges check."); + elog(DEBUG3, "init global curl context for privileges check."); --- End diff -- Please use initialize instead of init. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1079: HAWQ-1256. Enhance libcurl connection to ...
Github user stanlyxiang commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1079#discussion_r95512049
--- Diff: src/backend/libpq/rangerrest.c ---
@@ -281,29 +283,27 @@ static size_t write_callback(char *contents, size_t
size, size_t nitems,
CURL_HANDLE curl = (CURL_HANDLE) userp;
Assert(curl != NULL);
- if (curl->response.buffer == NULL)
- {
- curl->response.buffer = palloc0(realsize + 1);
- }
- else
+ elog(DEBUG3, "response size is %d. response buffer size is %d.",
curl->response.size, curl->response.buffer_size);
+ if(curl->response.size + realsize >= curl->response.buffer_size)
{
- /*Note:*/
- /*our repalloc is not same as realloc, repalloc's first
param(buffer) can not be NULL*/
- curl->response.buffer = repalloc(curl->response.buffer,
curl->response.size + realsize + 1);
+ /*
+* our repalloc is not same as realloc, repalloc's first
param(buffer) can not be NULL
+* double the buffer size if the buffer is not enough.
+*/
+ curl->response.buffer = repalloc(curl->response.buffer,
curl->response.buffer_size * 2);
--- End diff --
fixed. thks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1079: HAWQ-1256. Enhance libcurl connection to ...
Github user stanlyxiang commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1079#discussion_r95510736
--- Diff: src/backend/libpq/rangerrest.c ---
@@ -281,29 +283,27 @@ static size_t write_callback(char *contents, size_t
size, size_t nitems,
CURL_HANDLE curl = (CURL_HANDLE) userp;
Assert(curl != NULL);
- if (curl->response.buffer == NULL)
- {
- curl->response.buffer = palloc0(realsize + 1);
- }
- else
+ elog(DEBUG3, "response size is %d. response buffer size is %d.",
curl->response.size, curl->response.buffer_size);
+ if(curl->response.size + realsize >= curl->response.buffer_size)
{
- /*Note:*/
- /*our repalloc is not same as realloc, repalloc's first
param(buffer) can not be NULL*/
- curl->response.buffer = repalloc(curl->response.buffer,
curl->response.size + realsize + 1);
+ /*
+* our repalloc is not same as realloc, repalloc's first
param(buffer) can not be NULL
+* double the buffer size if the buffer is not enough.
+*/
+ curl->response.buffer = repalloc(curl->response.buffer,
curl->response.buffer_size * 2);
--- End diff --
oh yes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1079: HAWQ-1256. Enhance libcurl connection to ...
Github user interma commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1079#discussion_r95508628
--- Diff: src/backend/libpq/rangerrest.c ---
@@ -281,29 +283,27 @@ static size_t write_callback(char *contents, size_t
size, size_t nitems,
CURL_HANDLE curl = (CURL_HANDLE) userp;
Assert(curl != NULL);
- if (curl->response.buffer == NULL)
- {
- curl->response.buffer = palloc0(realsize + 1);
- }
- else
+ elog(DEBUG3, "response size is %d. response buffer size is %d.",
curl->response.size, curl->response.buffer_size);
+ if(curl->response.size + realsize >= curl->response.buffer_size)
{
- /*Note:*/
- /*our repalloc is not same as realloc, repalloc's first
param(buffer) can not be NULL*/
- curl->response.buffer = repalloc(curl->response.buffer,
curl->response.size + realsize + 1);
+ /*
+* our repalloc is not same as realloc, repalloc's first
param(buffer) can not be NULL
+* double the buffer size if the buffer is not enough.
+*/
+ curl->response.buffer = repalloc(curl->response.buffer,
curl->response.buffer_size * 2);
--- End diff --
Oh, buffer*2 maybe not enough for current content.
Should use a loop to determine the next buffer_size:
```
while (curl->response.size + realsize >= curl->response.buffer_size)
{ curl->response.buffer_size *= 2; }
//do realloc
//...
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1079: HAWQ-1256. Enhance libcurl connection to ...
Github user stanlyxiang commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1079#discussion_r95510088
--- Diff: src/backend/tcop/postgres.c ---
@@ -4626,6 +4630,24 @@ PostgresMain(int argc, char *argv[], const char
*username)
if (!ignore_till_sync)
send_ready_for_query = true;/* initially, or after error */
+ /* for enable ranger*/
+ if (enable_ranger && !curl_context.hasInited)
+ {
+ memset(&curl_context, 0, sizeof(curl_context_t));
+ curl_global_init(CURL_GLOBAL_ALL);
+ /* init the curl session */
+ curl_context.curl_handle = curl_easy_init();
+ if (curl_context.curl_handle == NULL) {
+ /* cleanup curl stuff */
+ /* no need to cleanup curl_handle since it's null. just
cleanup curl global.*/
+ curl_global_cleanup();
+ }
+ curl_context.hasInited = true;
+ curl_context.response.buffer = palloc0(CURL_RES_BUFFER_SIZE);
+ curl_context.response.buffer_size = CURL_RES_BUFFER_SIZE;
+ elog(LOG, "when enable ranger, init global struct for
privileges check.");
--- End diff --
fixed. thks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1079: HAWQ-1256. Enhance libcurl connection to ...
Github user stanlyxiang commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1079#discussion_r95509760 --- Diff: src/backend/tcop/postgres.c --- @@ -5314,6 +5336,24 @@ PostgresMain(int argc, char *argv[], const char *username) return 1; /* keep compiler quiet */ } +static void +curl_finalize(int code, Datum arg __attribute__((unused))) --- End diff -- fixed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1079: HAWQ-1256. Enhance libcurl connection to ...
Github user paul-guo- commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1079#discussion_r95507983
--- Diff: src/backend/tcop/postgres.c ---
@@ -5314,6 +5336,24 @@ PostgresMain(int argc, char *argv[], const char
*username)
return 1; /* keep compiler quiet
*/
}
+static void
+curl_finalize(int code, Datum arg __attribute__((unused)))
+{
+ if (curl_context.hasInited)
+ {
+ if (curl_context.response.buffer != NULL) {
+ pfree(curl_context.response.buffer);
+ }
+ /* cleanup curl stuff */
+ if (curl_context.curl_handle) {
+ curl_easy_cleanup(curl_context.curl_handle);
+ }
+ /* we're done with libcurl, so clean it up */
+ curl_global_cleanup();
+ curl_context.hasInited = false;
+ elog(LOG, "finalize the global struct for curl handle
context.");
--- End diff --
I did not look into the where should be set as DEBUG*, but at least if the
logs could be shown on our log files by default, they should 1) be accurate and
helpful. 2) benefit not just developers 3) avoid potential flooding.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1079: HAWQ-1256. Enhance libcurl connection to ...
Github user stanlyxiang commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1079#discussion_r95507913
--- Diff: src/backend/tcop/postgres.c ---
@@ -5314,6 +5336,24 @@ PostgresMain(int argc, char *argv[], const char
*username)
return 1; /* keep compiler quiet
*/
}
+static void
+curl_finalize(int code, Datum arg __attribute__((unused)))
+{
+ if (curl_context.hasInited)
+ {
+ if (curl_context.response.buffer != NULL) {
+ pfree(curl_context.response.buffer);
+ }
+ /* cleanup curl stuff */
+ if (curl_context.curl_handle) {
+ curl_easy_cleanup(curl_context.curl_handle);
+ }
+ /* we're done with libcurl, so clean it up */
+ curl_global_cleanup();
+ curl_context.hasInited = false;
+ elog(LOG, "finalize the global struct for curl handle
context.");
--- End diff --
cool, I'll fix these log level. @interma use guc "log_min_messages"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1079: HAWQ-1256. Enhance libcurl connection to ...
Github user interma commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1079#discussion_r95507805
--- Diff: src/backend/tcop/postgres.c ---
@@ -5314,6 +5336,24 @@ PostgresMain(int argc, char *argv[], const char
*username)
return 1; /* keep compiler quiet
*/
}
+static void
+curl_finalize(int code, Datum arg __attribute__((unused)))
+{
+ if (curl_context.hasInited)
+ {
+ if (curl_context.response.buffer != NULL) {
+ pfree(curl_context.response.buffer);
+ }
+ /* cleanup curl stuff */
+ if (curl_context.curl_handle) {
+ curl_easy_cleanup(curl_context.curl_handle);
+ }
+ /* we're done with libcurl, so clean it up */
+ curl_global_cleanup();
+ curl_context.hasInited = false;
+ elog(LOG, "finalize the global struct for curl handle
context.");
--- End diff --
Agreed with paul, maybe `DEBUG5` is ok.
But how to adjust log level of hawq? @paul-guo- And `DEBUG5` log is not
writen in log file now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1079: HAWQ-1256. Enhance libcurl connection to ...
Github user paul-guo- commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1079#discussion_r95507638 --- Diff: src/backend/tcop/postgres.c --- @@ -5314,6 +5336,24 @@ PostgresMain(int argc, char *argv[], const char *username) return 1; /* keep compiler quiet */ } +static void +curl_finalize(int code, Datum arg __attribute__((unused))) --- End diff -- Use __MAYBE_UNUSED in postgres.h --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1079: HAWQ-1256. Enhance libcurl connection to ...
Github user paul-guo- commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1079#discussion_r95507656
--- Diff: src/backend/tcop/postgres.c ---
@@ -4626,6 +4630,24 @@ PostgresMain(int argc, char *argv[], const char
*username)
if (!ignore_till_sync)
send_ready_for_query = true;/* initially, or after error */
+ /* for enable ranger*/
+ if (enable_ranger && !curl_context.hasInited)
+ {
+ memset(&curl_context, 0, sizeof(curl_context_t));
+ curl_global_init(CURL_GLOBAL_ALL);
+ /* init the curl session */
+ curl_context.curl_handle = curl_easy_init();
+ if (curl_context.curl_handle == NULL) {
+ /* cleanup curl stuff */
+ /* no need to cleanup curl_handle since it's null. just
cleanup curl global.*/
+ curl_global_cleanup();
+ }
+ curl_context.hasInited = true;
+ curl_context.response.buffer = palloc0(CURL_RES_BUFFER_SIZE);
+ curl_context.response.buffer_size = CURL_RES_BUFFER_SIZE;
+ elog(LOG, "when enable ranger, init global struct for
privileges check.");
--- End diff --
Syntax of the comments and the log (for all the patches). And, this comment
does more look like an debug level comment, please note logs are not just for
dev, also for users.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1079: HAWQ-1256. Enhance libcurl connection to ...
Github user paul-guo- commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1079#discussion_r95507152
--- Diff: src/backend/tcop/postgres.c ---
@@ -5314,6 +5336,24 @@ PostgresMain(int argc, char *argv[], const char
*username)
return 1; /* keep compiler quiet
*/
}
+static void
+curl_finalize(int code, Datum arg __attribute__((unused)))
+{
+ if (curl_context.hasInited)
+ {
+ if (curl_context.response.buffer != NULL) {
+ pfree(curl_context.response.buffer);
+ }
+ /* cleanup curl stuff */
+ if (curl_context.curl_handle) {
+ curl_easy_cleanup(curl_context.curl_handle);
+ }
+ /* we're done with libcurl, so clean it up */
+ curl_global_cleanup();
+ curl_context.hasInited = false;
+ elog(LOG, "finalize the global struct for curl handle
context.");
--- End diff --
Do we need LOG level here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #1079: HAWQ-1256. Enhance libcurl connection to Ranger ...
Github user stanlyxiang commented on the issue: https://github.com/apache/incubator-hawq/pull/1079 cc @linwen @interma @zhangh43 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1079: HAWQ-1256. Enhance libcurl connection to ...
GitHub user stanlyxiang opened a pull request: https://github.com/apache/incubator-hawq/pull/1079 HAWQ-1256. Enhance libcurl connection to Ranger Plugin Service, keep ⦠â¦it as a long-live connection in session level. You can merge this pull request into a Git repository by running: $ git pull https://github.com/stanlyxiang/incubator-hawq HAWQ-1256 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1079.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1079 commit b08a047bd628b58b09574d9af6b905bf6edbff13 Author: stanlyxiang Date: 2017-01-10T03:08:01Z HAWQ-1256. Enhance libcurl connection to Ranger Plugin Service, keep it as a long-live connection in session level --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (HAWQ-1252) PL/Java doc page - validate and enhance info
[ https://issues.apache.org/jira/browse/HAWQ-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15816785#comment-15816785 ] ASF GitHub Bot commented on HAWQ-1252: -- Github user dyozie commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/83#discussion_r95497019 --- Diff: markdown/plext/using_pljava.html.md.erb --- @@ -2,205 +2,222 @@ title: Using PL/Java --- -This section contains an overview of the HAWQ PL/Java language. +This section provides an overview of the HAWQ PL/Java procedural language extension. ## About PL/Java -With the HAWQ PL/Java extension, you can write Java methods using your favorite Java IDE and install the JAR files that implement the methods in your HAWQ cluster. +With the PL/Java extension, you can write Java methods using your favorite Java IDE and invoke the methods from PostgreSQL user-defined functions (UDFs). -**Note**: If building HAWQ from source, you must specify PL/Java as a build option when compiling HAWQ. To use PL/Java in a HAWQ deployment, you must explicitly enable the PL/Java extension in all desired databases. +The HAWQ PL/Java package is based on the open source PL/Java 1.4.0 and provides the following features: -The HAWQ PL/Java package is based on the open source PL/Java 1.4.0. HAWQ PL/Java provides the following features. - -- Ability to execute PL/Java functions with Java 1.6 or 1.7. -- Standardized utilities (modeled after the SQL 2003 proposal) to install and maintain Java code in the database. -- Standardized mappings of parameters and result. Complex types as well as sets are supported. -- An embedded, high performance, JDBC driver utilizing the internal HAWQ Database SPI routines. +- PL/Java function execution with Java 1.6 or 1.7. +- Standardized mappings of Java and PostgreSQL parameters and results. Complex types as well as sets are supported. +- Two HAWQ Java languages: + - `java` - Tusted PL/Java language + - `javau` - Untrusted PL/Java language (superusers only) +- An embedded, high performance, JDBC driver utilizing the internal HAWQ Database Server Programming Interface (SPI). - Metadata support for the JDBC driver. Both `DatabaseMetaData` and `ResultSetMetaData` are included. -- The ability to return a `ResultSet` from a query as an alternative to building a ResultSet row by row. +- A standard way of passing parameters and return values. Complex types and sets are passed using the standard JDBC `ResultSet` class. +- The ability to return a `ResultSet` from a query as an alternative to building a `ResultSet` row by row. - Full support for savepoints and exception handling. -- The ability to use IN, INOUT, and OUT parameters. -- Two separate HAWQ languages: - - pljava, TRUSTED PL/Java language - - pljavau, UNTRUSTED PL/Java language - Transaction and Savepoint listeners enabling code execution when a transaction or savepoint is committed or rolled back. - Integration with GNU GCJ on selected platforms. -A function in SQL will appoint a static method in a Java class. In order for the function to execute, the appointed class must available on the class path specified by the HAWQ server configuration parameter `pljava_classpath`. The PL/Java extension adds a set of functions that helps to install and maintain the Java classes. Classes are stored in normal Java archives, JAR files. A JAR file can optionally contain a deployment descriptor that in turn contains SQL commands to be executed when the JAR is deployed or undeployed. The functions are modeled after the standards proposed for SQL 2003. - -PL/Java implements a standard way of passing parameters and return values. Complex types and sets are passed using the standard JDBC ResultSet class. - -A JDBC driver is included in PL/Java. This driver calls HAWQ internal SPI routines. The driver is essential since it is common for functions to make calls back to the database to fetch data. When PL/Java functions fetch data, they must use the same transactional boundaries that are used by the main function that entered PL/Java execution context. +PL/Java is optimized for performance. The Java virtual machine executes within the same process as the backend, minimizing call overhead. PL/Java brings the power of Java to the database itself, enabling the database-intensive business logic to execute as close to the actual data as possible. -PL/Java is optimized for performance. The Java virtual machine executes within the same process as the backend to minimize call overhead. PL/Java is designed with the objective to enable the power of Java to the database itself so that database intensive business logic can execute a
[jira] [Commented] (HAWQ-1252) PL/Java doc page - validate and enhance info
[ https://issues.apache.org/jira/browse/HAWQ-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15816782#comment-15816782 ] ASF GitHub Bot commented on HAWQ-1252: -- Github user dyozie commented on a diff in the pull request: https://github.com/apache/incubator-hawq-docs/pull/83#discussion_r95496941 --- Diff: markdown/plext/using_pljava.html.md.erb --- @@ -2,205 +2,222 @@ title: Using PL/Java --- -This section contains an overview of the HAWQ PL/Java language. +This section provides an overview of the HAWQ PL/Java procedural language extension. ## About PL/Java -With the HAWQ PL/Java extension, you can write Java methods using your favorite Java IDE and install the JAR files that implement the methods in your HAWQ cluster. +With the PL/Java extension, you can write Java methods using your favorite Java IDE and invoke the methods from PostgreSQL user-defined functions (UDFs). -**Note**: If building HAWQ from source, you must specify PL/Java as a build option when compiling HAWQ. To use PL/Java in a HAWQ deployment, you must explicitly enable the PL/Java extension in all desired databases. +The HAWQ PL/Java package is based on the open source PL/Java 1.4.0 and provides the following features: -The HAWQ PL/Java package is based on the open source PL/Java 1.4.0. HAWQ PL/Java provides the following features. - -- Ability to execute PL/Java functions with Java 1.6 or 1.7. -- Standardized utilities (modeled after the SQL 2003 proposal) to install and maintain Java code in the database. -- Standardized mappings of parameters and result. Complex types as well as sets are supported. -- An embedded, high performance, JDBC driver utilizing the internal HAWQ Database SPI routines. +- PL/Java function execution with Java 1.6 or 1.7. +- Standardized mappings of Java and PostgreSQL parameters and results. Complex types as well as sets are supported. +- Two HAWQ Java languages: + - `java` - Tusted PL/Java language + - `javau` - Untrusted PL/Java language (superusers only) +- An embedded, high performance, JDBC driver utilizing the internal HAWQ Database Server Programming Interface (SPI). - Metadata support for the JDBC driver. Both `DatabaseMetaData` and `ResultSetMetaData` are included. -- The ability to return a `ResultSet` from a query as an alternative to building a ResultSet row by row. +- A standard way of passing parameters and return values. Complex types and sets are passed using the standard JDBC `ResultSet` class. +- The ability to return a `ResultSet` from a query as an alternative to building a `ResultSet` row by row. - Full support for savepoints and exception handling. -- The ability to use IN, INOUT, and OUT parameters. -- Two separate HAWQ languages: - - pljava, TRUSTED PL/Java language - - pljavau, UNTRUSTED PL/Java language - Transaction and Savepoint listeners enabling code execution when a transaction or savepoint is committed or rolled back. - Integration with GNU GCJ on selected platforms. -A function in SQL will appoint a static method in a Java class. In order for the function to execute, the appointed class must available on the class path specified by the HAWQ server configuration parameter `pljava_classpath`. The PL/Java extension adds a set of functions that helps to install and maintain the Java classes. Classes are stored in normal Java archives, JAR files. A JAR file can optionally contain a deployment descriptor that in turn contains SQL commands to be executed when the JAR is deployed or undeployed. The functions are modeled after the standards proposed for SQL 2003. - -PL/Java implements a standard way of passing parameters and return values. Complex types and sets are passed using the standard JDBC ResultSet class. - -A JDBC driver is included in PL/Java. This driver calls HAWQ internal SPI routines. The driver is essential since it is common for functions to make calls back to the database to fetch data. When PL/Java functions fetch data, they must use the same transactional boundaries that are used by the main function that entered PL/Java execution context. +PL/Java is optimized for performance. The Java virtual machine executes within the same process as the backend, minimizing call overhead. PL/Java brings the power of Java to the database itself, enabling the database-intensive business logic to execute as close to the actual data as possible. -PL/Java is optimized for performance. The Java virtual machine executes within the same process as the backend to minimize call overhead. PL/Java is designed with the objective to enable the power of Java to the database itself so that database intensive business logic can execute a
[jira] [Resolved] (HAWQ-1157) Make consistent docs link to PostgreSQL 8.2
[ https://issues.apache.org/jira/browse/HAWQ-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jane Beckman resolved HAWQ-1157. Resolution: Fixed Fix Version/s: 2.1.0.0-incubating > Make consistent docs link to PostgreSQL 8.2 > --- > > Key: HAWQ-1157 > URL: https://issues.apache.org/jira/browse/HAWQ-1157 > Project: Apache HAWQ > Issue Type: Task > Components: Documentation >Reporter: Jane Beckman >Assignee: David Yozie > Fix For: 2.1.0.0-incubating, 2.2.0.0-incubating > > > Some links to Postgresql link to other versions of PostgreSQL than 8.2. HAWQ > is based on PostgreSQL 8.2.15. One section mentions PostgreSQL 9.0 > specifically. These references should standardize on 8.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-1108) Add JDBC PXF Plugin
[ https://issues.apache.org/jira/browse/HAWQ-1108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15816344#comment-15816344 ] Shivram Mani commented on HAWQ-1108: [~michael.andre.pearce] i understand the frustration in the delay. The merge certification would've been a whole lot easier if the automation test code based is also open sourced. [~vVineet] at this point the JDBC Plugin code has passed the Code review. I am ok with merging this to master and qualifying this as beta and over time we can improve on the automation tests for this new profile. > Add JDBC PXF Plugin > --- > > Key: HAWQ-1108 > URL: https://issues.apache.org/jira/browse/HAWQ-1108 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF >Reporter: Michael Andre Pearce (IG) >Assignee: Devin Jia > > On the back of the work in : > https://issues.apache.org/jira/browse/HAWQ-779 > We would like to add to Hawq Plugins a JDBC implementation. > There are currently two noted implementations in the openly available in > GitHub. > 1) https://github.com/kojec/pxf-field/tree/master/jdbc-pxf-ext > 2) https://github.com/inspur-insight/pxf-plugin/tree/master/pxf-jdbc > The latter (2) is an improved version of the former (1) and also what > HAWQ-779 changes were to support. > [~jiadx] would you be happy to contribute the source as apache 2 license open > source? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HAWQ-1108) Add JDBC PXF Plugin
[ https://issues.apache.org/jira/browse/HAWQ-1108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15816309#comment-15816309 ] Michael Andre Pearce (IG) edited comment on HAWQ-1108 at 1/10/17 9:54 PM: -- [~shivram] [~GodenYao] Hi guys this seems that this has gone stale, can we get this added/merged so we don't loose the traction? Would be a real shame if this goes to the way side. was (Author: michael.andre.pearce): [~shivram] [~GodenYao] Hi guys this seems that this has gone stale, can we get this added/merged so we don't loose the traction? > Add JDBC PXF Plugin > --- > > Key: HAWQ-1108 > URL: https://issues.apache.org/jira/browse/HAWQ-1108 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF >Reporter: Michael Andre Pearce (IG) >Assignee: Devin Jia > > On the back of the work in : > https://issues.apache.org/jira/browse/HAWQ-779 > We would like to add to Hawq Plugins a JDBC implementation. > There are currently two noted implementations in the openly available in > GitHub. > 1) https://github.com/kojec/pxf-field/tree/master/jdbc-pxf-ext > 2) https://github.com/inspur-insight/pxf-plugin/tree/master/pxf-jdbc > The latter (2) is an improved version of the former (1) and also what > HAWQ-779 changes were to support. > [~jiadx] would you be happy to contribute the source as apache 2 license open > source? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-1108) Add JDBC PXF Plugin
[ https://issues.apache.org/jira/browse/HAWQ-1108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15816309#comment-15816309 ] Michael Andre Pearce (IG) commented on HAWQ-1108: - [~shivram] [~GodenYao] Hi guys this seems that this has gone stale, can we get this added/merged so we don't loose the traction? > Add JDBC PXF Plugin > --- > > Key: HAWQ-1108 > URL: https://issues.apache.org/jira/browse/HAWQ-1108 > Project: Apache HAWQ > Issue Type: New Feature > Components: PXF >Reporter: Michael Andre Pearce (IG) >Assignee: Devin Jia > > On the back of the work in : > https://issues.apache.org/jira/browse/HAWQ-779 > We would like to add to Hawq Plugins a JDBC implementation. > There are currently two noted implementations in the openly available in > GitHub. > 1) https://github.com/kojec/pxf-field/tree/master/jdbc-pxf-ext > 2) https://github.com/inspur-insight/pxf-plugin/tree/master/pxf-jdbc > The latter (2) is an improved version of the former (1) and also what > HAWQ-779 changes were to support. > [~jiadx] would you be happy to contribute the source as apache 2 license open > source? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1265) Support Complex Hive Schema's
[ https://issues.apache.org/jira/browse/HAWQ-1265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Andre Pearce (IG) updated HAWQ-1265: Description: Currently if in hive you have Avro or other formats, where the schema is complex, you cannot currently query the fields in the complex object via hcatalog/pxf integration. In terms of Avro Schema - records, enums and complex arrays, do not work. Hive fully supports this complex/object notation, as does many of the other SQL tools in the Hadoop eco-system (Spark, Impala, Drill). These all seem to support the same styled solution of using dots for complex object/path negotiation: SELECT schema.table.fieldA.nestedRecordFieldS FROM myavrotable; was: Currently if in hive you have Avro or other formats, where the schema is complex, you cannot currently query the fields in the complex object via hcatalog/pxf integration. In terms of Avro Schema - records, enums and complex arrays, do not work. Hive fully supports this complex/object notation, as does many of the other SQL tools in the Hadoop eco-system (Spark, Impala, Drill). These all seem to support the same styled solution of using dots for complex object/path negation: SELECT schema.table.fieldA.nestedRecordFieldS FROM myavrotable; > Support Complex Hive Schema's > - > > Key: HAWQ-1265 > URL: https://issues.apache.org/jira/browse/HAWQ-1265 > Project: Apache HAWQ > Issue Type: Improvement > Components: Hcatalog, PXF >Reporter: Michael Andre Pearce (IG) >Assignee: Ed Espino > > Currently if in hive you have Avro or other formats, where the schema is > complex, you cannot currently query the fields in the complex object via > hcatalog/pxf integration. > In terms of Avro Schema - records, enums and complex arrays, do not work. > Hive fully supports this complex/object notation, as does many of the other > SQL tools in the Hadoop eco-system (Spark, Impala, Drill). > These all seem to support the same styled solution of using dots for complex > object/path negotiation: > SELECT schema.table.fieldA.nestedRecordFieldS FROM myavrotable; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HAWQ-1265) Support Complex Hive Schema's
Michael Andre Pearce (IG) created HAWQ-1265: --- Summary: Support Complex Hive Schema's Key: HAWQ-1265 URL: https://issues.apache.org/jira/browse/HAWQ-1265 Project: Apache HAWQ Issue Type: Improvement Components: Hcatalog, PXF Reporter: Michael Andre Pearce (IG) Assignee: Ed Espino Currently if in hive you have Avro or other formats, where the schema is complex, you cannot currently query the fields in the complex object via hcatalog/pxf integration. In terms of Avro Schema - records, enums and complex arrays, do not work. Hive fully supports this complex/object notation, as does many of the other SQL tools in the Hadoop eco-system (Spark, Impala, Drill). These all seem to support the same styled solution of using dots for complex object/path negation: SELECT schema.table.fieldA.nestedRecordFieldS FROM myavrotable; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-1251) Add CREATE CAST/CONVERSION pages to documentation
[ https://issues.apache.org/jira/browse/HAWQ-1251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15815923#comment-15815923 ] ASF GitHub Bot commented on HAWQ-1251: -- GitHub user janebeckman opened a pull request: https://github.com/apache/incubator-hawq-docs/pull/85 Feature/hawq-1251 New pages for CREATE/DROP CAST/CONVERSION. https://issues.apache.org/jira/browse/HAWQ-1251 You can merge this pull request into a Git repository by running: $ git pull https://github.com/janebeckman/incubator-hawq-docs feature/HAWQ-1251 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq-docs/pull/85.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #85 commit 11455b2940304b68374ee8c1e57c2a9ce30d6c83 Author: Jane Beckman Date: 2017-01-09T22:21:32Z New SQL pages commit 0fa21174885169bd2dde8e3da7b66f69f800b132 Author: Jane Beckman Date: 2017-01-09T22:31:14Z Navs for new SQL commit d9a65569505e8017e2403193d86f918e22d35af7 Author: Jane Beckman Date: 2017-01-10T19:06:30Z Format and grammar improvements > Add CREATE CAST/CONVERSION pages to documentation > - > > Key: HAWQ-1251 > URL: https://issues.apache.org/jira/browse/HAWQ-1251 > Project: Apache HAWQ > Issue Type: Improvement > Components: Documentation >Reporter: Jane Beckman >Assignee: David Yozie > > The pg_cast table, created by CREATE CAST is already in the documentation, > but not the command. Testing shows CREATE CAST seems to be supported in HAWQ. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HAWQ-1259) Remove gprecoverseg from documentation
[ https://issues.apache.org/jira/browse/HAWQ-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jane Beckman resolved HAWQ-1259. Resolution: Fixed Fix Version/s: 2.0.0.0-incubating > Remove gprecoverseg from documentation > -- > > Key: HAWQ-1259 > URL: https://issues.apache.org/jira/browse/HAWQ-1259 > Project: Apache HAWQ > Issue Type: Bug > Components: Documentation >Reporter: Jane Beckman >Assignee: David Yozie > Fix For: 2.0.0.0-incubating > > > The gprecoverseg utility is not supported in HAWQ 2.x. References to it have > already been removed as of the HAWQ 2.1 release. References should be removed > from earlier releases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1248) Merge Dockerfiles for HAWQ Dev into HAWQ code base
[ https://issues.apache.org/jira/browse/HAWQ-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruilong Huo updated HAWQ-1248: -- Component/s: Build > Merge Dockerfiles for HAWQ Dev into HAWQ code base > -- > > Key: HAWQ-1248 > URL: https://issues.apache.org/jira/browse/HAWQ-1248 > Project: Apache HAWQ > Issue Type: Improvement > Components: Build >Reporter: Richard Guo >Assignee: Richard Guo > > HAWQ team is building HAWQ dev docker images to provide an out-of-box > way for developers to setup build and test environment for HAWQ. > Dockerfiles are supposed to merge into HAWQ code base for easy maintenance. > File this issue to track the merging task. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1248) Merge Dockerfiles for HAWQ Dev into HAWQ code base
[ https://issues.apache.org/jira/browse/HAWQ-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruilong Huo updated HAWQ-1248: -- Affects Version/s: 2.0.0.0-incubating > Merge Dockerfiles for HAWQ Dev into HAWQ code base > -- > > Key: HAWQ-1248 > URL: https://issues.apache.org/jira/browse/HAWQ-1248 > Project: Apache HAWQ > Issue Type: Improvement > Components: Build >Affects Versions: 2.0.0.0-incubating >Reporter: Richard Guo >Assignee: Richard Guo > > HAWQ team is building HAWQ dev docker images to provide an out-of-box > way for developers to setup build and test environment for HAWQ. > Dockerfiles are supposed to merge into HAWQ code base for easy maintenance. > File this issue to track the merging task. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HAWQ-1264) Add PGP KEYS files for code signing purposes.
[ https://issues.apache.org/jira/browse/HAWQ-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ed Espino updated HAWQ-1264: Fix Version/s: (was: 2.1.0.0-incubating) 2.2.0.0-incubating > Add PGP KEYS files for code signing purposes. > - > > Key: HAWQ-1264 > URL: https://issues.apache.org/jira/browse/HAWQ-1264 > Project: Apache HAWQ > Issue Type: New Feature >Reporter: Ed Espino >Assignee: Ed Espino > Fix For: 2.2.0.0-incubating > > > Add a KEYS file to the repository. The main purpose is for code signing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-hawq pull request #1078: HAWQ-1258. segment resource manager does ...
Github user paul-guo- commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1078#discussion_r95314993 --- Diff: src/backend/utils/misc/guc.c --- @@ -8200,7 +8200,7 @@ static struct config_string ConfigureNamesString[] = NULL }, &standby_addr_host, - "localhost", NULL, NULL + "none", NULL, NULL --- End diff -- Not sure whether I understand correctly, but could not we modify the code like this? "", NULL, NULL if (standby_addr_host[0] == '\0') --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
