[jira] [Commented] (HAWQ-1154) ERROR: Cannot support create protocol statement yet
[ https://issues.apache.org/jira/browse/HAWQ-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15756563#comment-15756563 ] Ruilong Huo commented on HAWQ-1154: --- The creation of protocol is not supported in hawq. Though it is supported in utility mode to pre-define some built in protocol. > ERROR: Cannot support create protocol statement yet > > > Key: HAWQ-1154 > URL: https://issues.apache.org/jira/browse/HAWQ-1154 > Project: Apache HAWQ > Issue Type: Bug >Reporter: Pratheesh Nair >Assignee: Lei Chang > > Hawq doesn't support creating custom protocols . > Example : > CREATE TRUSTED PROTOCOL myprot( > writefunc='mywrite', > readfunc='myread', > validatorfunc='myvalidate'); > ERROR: Cannot support create protocol statement yet -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-1223) hawq can not get resource when run with yarn
[ https://issues.apache.org/jira/browse/HAWQ-1223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15756488#comment-15756488 ] Ruilong Huo commented on HAWQ-1223: --- [~huxiao64], thanks for raising question in apache hawq community. As [~xsheng] mentioned, it is recommended to subscribe hawq dev and user email group so that the discussion can be done there. If it turn out to be an issue with hawq, then jira is created for further RCA and fix. > hawq can not get resource when run with yarn > > > Key: HAWQ-1223 > URL: https://issues.apache.org/jira/browse/HAWQ-1223 > Project: Apache HAWQ > Issue Type: Bug > Components: libyarn >Reporter: Xiao Hu >Assignee: Xiang Sheng > Fix For: 2.0.1.0-incubating > > > when I ran hawq with yarn(2.5.0) which is in the hadoop cluster, HAWQ could > not acquire resource from source manager but the resource is enough. > The information is below: > wangtest=# insert into t values(1); > ERROR: failed to acquire resource from resource manager, resource request is > timed out due to no available cluster (pquery.c:804) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (HAWQ-1214) Remove resource_parameters
[ https://issues.apache.org/jira/browse/HAWQ-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Guo closed HAWQ-1214. -- Resolution: Fixed Fix Version/s: 2.0.1.0-incubating > Remove resource_parameters > -- > > Key: HAWQ-1214 > URL: https://issues.apache.org/jira/browse/HAWQ-1214 > Project: Apache HAWQ > Issue Type: Bug >Reporter: Paul Guo >Assignee: Paul Guo > Fix For: 2.0.1.0-incubating > > > The reasons are: > 1) It is not used anymore. > 2) Have seen crash due to this. The scenario is that: > > The datalocality memory context on which resource_parameters is palloc-ed is > reset before calculate_planner_segment_num() finishes, thus later access to > resource_parameters could lead to crash or wrong results. A code path example > which could lead to segfault is: > ProcessUtility()->PerformCursorOpen()->_copyPlannedStmt()->_copyQueryResourceParameters() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-1223) hawq can not get resource when run with yarn
[ https://issues.apache.org/jira/browse/HAWQ-1223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15756128#comment-15756128 ] Xiang Sheng commented on HAWQ-1223: --- Thansk Xiao. Could you write down what's wrong with your config ? This would help others that meet the same problem with you. And we encourage you to subscribe our dev / user mail list ( http://hawq.incubator.apache.org/#mailing-lists ) . Post your problem to the lists can get more quickly and clearly help, maybe related people didn't see the JIRA and reply in time. > hawq can not get resource when run with yarn > > > Key: HAWQ-1223 > URL: https://issues.apache.org/jira/browse/HAWQ-1223 > Project: Apache HAWQ > Issue Type: Bug > Components: libyarn >Reporter: Xiao Hu >Assignee: Xiang Sheng > Fix For: 2.0.1.0-incubating > > > when I ran hawq with yarn(2.5.0) which is in the hadoop cluster, HAWQ could > not acquire resource from source manager but the resource is enough. > The information is below: > wangtest=# insert into t values(1); > ERROR: failed to acquire resource from resource manager, resource request is > timed out due to no available cluster (pquery.c:804) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-hawq pull request #1044: HAWQ-1178. Enhance Fragmenter api to retu...
Github user shivzone commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1044#discussion_r92900050
--- Diff:
pxf/pxf-service/src/main/java/org/apache/hawq/pxf/service/ProfileFactory.java
---
@@ -0,0 +1,46 @@
+package org.apache.hawq.pxf.service;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hadoop.hive.ql.io.orc.OrcInputFormat;
+//import org.apache.hadoop.hive.ql.io.RCFileInputFormat;
+import org.apache.hadoop.mapred.InputFormat;
+//import org.apache.hadoop.mapred.TextInputFormat;
+
+public class ProfileFactory {
+
+private static final String HIVE_TEXT_PROFILE = "HiveText";
+private static final String HIVE_RC_PROFILE = "HiveRC";
+private static final String HIVE_ORC_PROFILE = "HiveORC";
+
+public static String get(InputFormat inputFormat) throws Exception {
+String profileName = null;
+//if (inputFormat instanceof TextInputFormat) {
--- End diff --
If we do plan to rollout the hcatalog based lookup feature, we would need
to deal with that info as well. We can myabe create a separate PR/Jira for
that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #972: HAWQ-1108 Add JDBC PXF Plugin
Github user ig-michaelpearce commented on the issue: https://github.com/apache/incubator-hawq/pull/972 @shivzone do other pxf plugins have this? I cannot see this. Could this be merged and then later address in a follow on JIRA? Seems this has stalled. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (HAWQ-1225) Document the catalog table pg_aoseg.pgaoseg$relid
Jane Beckman created HAWQ-1225: -- Summary: Document the catalog table pg_aoseg.pgaoseg$relid Key: HAWQ-1225 URL: https://issues.apache.org/jira/browse/HAWQ-1225 Project: Apache HAWQ Issue Type: Improvement Components: Documentation Reporter: Jane Beckman Assignee: David Yozie Fix For: 2.0.1.0-incubating The pg_aoseg.pgaoseg$relid is new and related to the hawq register --force option for append-only tables. There is also pg_aoseg.pgpaqseg$relid, a table for parquet tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HAWQ-1217) Clean up Catalog Table Reference section
[ https://issues.apache.org/jira/browse/HAWQ-1217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jane Beckman resolved HAWQ-1217. Resolution: Fixed > Clean up Catalog Table Reference section > > > Key: HAWQ-1217 > URL: https://issues.apache.org/jira/browse/HAWQ-1217 > Project: Apache HAWQ > Issue Type: Bug > Components: Documentation >Reporter: Jane Beckman >Assignee: David Yozie > Fix For: 2.0.1.0-incubating > > > The Catalog Table Reference section has some instances of problematic formats > (spacing issues, typos, etc.), plus inconsistent use of formats. These should > be corrected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-hawq pull request #1052: HAWQ-1004. Implement calling Ranger REST ...
Github user denalex commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1052#discussion_r92857693
--- Diff: src/backend/libpq/rangerrest.c ---
@@ -0,0 +1,551 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*-
+ *
+ * rangerrest.c
+ * routines to interact with Ranger REST API
+ *
+ *-
+ */
+#include "postgres.h"
+
+#include
+
+#include "utils/acl.h"
+#include "utils/guc.h"
+#include "utils/rangerrest.h"
+
+/*
+ * Internal buffer for libcurl context
+ */
+typedef struct curl_context_t
+{
+CURL* curl_handle;
+
+char curl_error_buffer[CURL_ERROR_SIZE];
+
+int curl_still_running;
+
+struct
+{
+char* buffer;
+int size;
+} response;
+
+char* last_http_reponse;
+} curl_context_t;
+typedef curl_context_t* CURL_HANDLE;
+
+RangerACLResult parse_ranger_response(char* buffer)
+{
+Assert(buffer != NULL);
+if (strlen(buffer) == 0)
+return RANGERCHECK_UNKNOWN;
+
+elog(LOG, "read from Ranger Restful API: %s", buffer);
+
+struct json_object *response = json_tokener_parse(buffer);
+struct json_object *accessObj = json_object_object_get(response,
"access");
+
+int arraylen = json_object_array_length(accessObj);
+elog(LOG, "Array Length: %dn",arraylen);
+
+json_object * jvalue;
+json_object * jallow;
+json_bool result;
+// here should return which table's acl check failed in future.
+for (int i=0; i< arraylen; i++){
+ jvalue = json_object_array_get_idx(accessObj, i);
+ jallow = json_object_object_get(jvalue, "allowed");
+ result = json_object_get_boolean(jallow);
+ if(result != 1){
+return RANGERCHECK_NO_PRIV;
+ }
+}
+return RANGERCHECK_OK;
+
+}
+
+/*
+ * A mapping from AclObjectKind to string
+ */
+char* AclObjectKindStr[] =
+{
+"table", /* pg_class */
+"sequence", /* pg_sequence */
+"database", /* pg_database */
+"function", /* pg_proc */
+"operator", /* pg_operator */
+"type", /* pg_type */
+"language", /* pg_language */
+"namespace", /* pg_namespace */
+"oplass",/* pg_opclass */
+"conversion",/* pg_conversion */
+"tablespace",/* pg_tablespace */
+"filespace", /* pg_filespace */
+"filesystem",/* pg_filesystem */
+"fdw", /* pg_foreign_data_wrapper */
+"foreign_server",/* pg_foreign_server */
+"protocol", /* pg_extprotocol */
+"none" /* MUST BE LAST */
+};
+
+/*
+ * args: List of RangerRequestJsonArgs
+ */
+json_object *create_ranger_request_json_batch(List *args)
+{
+ json_object *juser = NULL;
+ json_object *jaccess = json_object_new_array();
+ json_object *jrequest = json_object_new_object();
+ char *user = NULL;
+ ListCell *arg;
+
+ foreach(arg, args)
+ {
+RangerRequestJsonArgs *arg_ptr = (RangerRequestJsonArgs *) lfirst(arg);
+if (user == NULL)
+{
+ user = arg_ptr->user;
+ juser = json_object_new_string(user);
+}
+AclObjectKind kind = arg_ptr->kind;
+char* object = arg_ptr->object;
+char* how = arg_ptr->how;
--- End diff --
we are not supporting "how" in RPS
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at
[jira] [Assigned] (HAWQ-1223) hawq can not get resource when run with yarn
[ https://issues.apache.org/jira/browse/HAWQ-1223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiang Sheng reassigned HAWQ-1223: - Assignee: Xiang Sheng (was: Ed Espino) > hawq can not get resource when run with yarn > > > Key: HAWQ-1223 > URL: https://issues.apache.org/jira/browse/HAWQ-1223 > Project: Apache HAWQ > Issue Type: Bug > Components: libyarn >Reporter: Xiao Hu >Assignee: Xiang Sheng > Fix For: 2.0.1.0-incubating > > > when I ran hawq with yarn(2.5.0) which is in the hadoop cluster, HAWQ could > not acquire resource from source manager but the resource is enough. > The information is below: > wangtest=# insert into t values(1); > ERROR: failed to acquire resource from resource manager, resource request is > timed out due to no available cluster (pquery.c:804) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HAWQ-1223) hawq can not get resource when run with yarn
[ https://issues.apache.org/jira/browse/HAWQ-1223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15754478#comment-15754478 ] Xiao Hu commented on HAWQ-1223: --- Thanks, Xiang, I have checked my system.I found it was a problem with my yarn, not HAWQ. Following your tips, I fixed it successfully. Sometimes there are some bugs on hdfs or yarn, not HAWQ, I must make sure the position of the problem. > hawq can not get resource when run with yarn > > > Key: HAWQ-1223 > URL: https://issues.apache.org/jira/browse/HAWQ-1223 > Project: Apache HAWQ > Issue Type: Bug > Components: libyarn >Reporter: Xiao Hu >Assignee: Ed Espino > Fix For: 2.0.1.0-incubating > > > when I ran hawq with yarn(2.5.0) which is in the hadoop cluster, HAWQ could > not acquire resource from source manager but the resource is enough. > The information is below: > wangtest=# insert into t values(1); > ERROR: failed to acquire resource from resource manager, resource request is > timed out due to no available cluster (pquery.c:804) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (HAWQ-1001) Implement HAWQ basic user ACL check through Ranger
[ https://issues.apache.org/jira/browse/HAWQ-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hubert Zhang closed HAWQ-1001. -- Resolution: Fixed > Implement HAWQ basic user ACL check through Ranger > -- > > Key: HAWQ-1001 > URL: https://issues.apache.org/jira/browse/HAWQ-1001 > Project: Apache HAWQ > Issue Type: Sub-task > Components: Core >Reporter: Lili Ma >Assignee: Hubert Zhang > Fix For: backlog > > > When a user run some query, HAWQ can connect to Ranger to judge whether the > user has the privilege to do that. > For each object with unique oid, send one request to Ranger -- This message was sent by Atlassian JIRA (v6.3.4#6332)
