date:20160826

[jira] [Commented] (HAWQ-1019) clarify database application interfaces discussion

2016-08-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/HAWQ-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440281#comment-15440281
 ] 

ASF GitHub Bot commented on HAWQ-1019:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-hawq-docs/pull/3


> clarify database application interfaces discussion
> --
>
> Key: HAWQ-1019
> URL: https://issues.apache.org/jira/browse/HAWQ-1019
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Lisa Owen
>Assignee: Lei Chang
>Priority: Minor
> Fix For: 2.0.1.0-incubating
>
>
> discussion of drivers for database application interfaces needs to be 
> clarified.
> relevant incubator-hawq-docs file:  
> clientaccess/g-database-application-interfaces.html.md.erb 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-1019) clarify database application interfaces discussion

2016-08-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/HAWQ-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440251#comment-15440251
 ] 

ASF GitHub Bot commented on HAWQ-1019:
--

GitHub user lisakowen opened a pull request:

https://github.com/apache/incubator-hawq-docs/pull/3

misc doc updates clarifying APIs

updates to clarify database application interfaces.  fixes HAWQ-1019

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lisakowen/incubator-hawq-docs 
feature/dbappif-fixes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq-docs/pull/3.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3


commit bec5e3e1855bed4f0f8a5ef2a41b94c3c92f17fc
Author: Lisa Owen 
Date:   2016-08-26T23:59:00Z

misc doc updates clarifying APIs




> clarify database application interfaces discussion
> --
>
> Key: HAWQ-1019
> URL: https://issues.apache.org/jira/browse/HAWQ-1019
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Lisa Owen
>Assignee: Lei Chang
>Priority: Minor
> Fix For: 2.0.1.0-incubating
>
>
> discussion of drivers for database application interfaces needs to be 
> clarified.
> relevant incubator-hawq-docs file:  
> clientaccess/g-database-application-interfaces.html.md.erb 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-1027) Book configuration directory needs to be outside of content repo

2016-08-26 Thread David Yozie (JIRA)


[ 
https://issues.apache.org/jira/browse/HAWQ-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440228#comment-15440228
 ] 

David Yozie commented on HAWQ-1027:
---

A Pivotal Tracker story has been created for this Issue: 
https://www.pivotaltracker.com/story/show/129183747

> Book configuration directory needs to be outside of content repo
> 
>
> Key: HAWQ-1027
> URL: https://issues.apache.org/jira/browse/HAWQ-1027
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: David Yozie
>Assignee: David Yozie
>
> The incubator-hawq-docs repo includes a sample book configuration (hawq-docs) 
> for producing HTML.  Unfortunately, this configuration directory causes 
> problems with the latest middleman that will be used in an upcoming 
> bookbinder release, so it needs to be moved.  
> We can probably get away with putting it in a separate branch for now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HAWQ-1027) Book configuration directory needs to be outside of content repo

2016-08-26 Thread David Yozie (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Yozie reassigned HAWQ-1027:
-

Assignee: David Yozie  (was: Lei Chang)

> Book configuration directory needs to be outside of content repo
> 
>
> Key: HAWQ-1027
> URL: https://issues.apache.org/jira/browse/HAWQ-1027
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: David Yozie
>Assignee: David Yozie
>
> The incubator-hawq-docs repo includes a sample book configuration (hawq-docs) 
> for producing HTML.  Unfortunately, this configuration directory causes 
> problems with the latest middleman that will be used in an upcoming 
> bookbinder release, so it needs to be moved.  
> We can probably get away with putting it in a separate branch for now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-1027) Book configuration directory needs to be outside of content repo

2016-08-26 Thread David Yozie (JIRA)

David Yozie created HAWQ-1027:
-

 Summary: Book configuration directory needs to be outside of 
content repo
 Key: HAWQ-1027
 URL: https://issues.apache.org/jira/browse/HAWQ-1027
 Project: Apache HAWQ
  Issue Type: Improvement
  Components: Documentation
Reporter: David Yozie
Assignee: Lei Chang


The incubator-hawq-docs repo includes a sample book configuration (hawq-docs) 
for producing HTML.  Unfortunately, this configuration directory causes 
problems with the latest middleman that will be used in an upcoming bookbinder 
release, so it needs to be moved.  

We can probably get away with putting it in a separate branch for now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...

2016-08-26 Thread hsyuan

Github user hsyuan commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/837#discussion_r76479540
  
--- Diff: 
pxf/pxf-hbase/src/main/java/org/apache/hawq/pxf/plugins/hbase/HBaseFilterBuilder.java
 ---
@@ -165,6 +165,14 @@ private Filter 
handleSimpleOperations(FilterParser.Operation opId,
 ByteArrayComparable comparator = 
getComparator(hbaseColumn.columnTypeCode(),
 constant.constant());
 
+if(operatorsMap.get(opId) == null){
+//HBase not support HDOP_LIKE, use 'NOT NULL' Comarator
--- End diff --

HBase does not support


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...

2016-08-26 Thread GodenYao

Github user GodenYao commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/837#discussion_r76469613
  
--- Diff: 
pxf/pxf-hbase/src/main/java/org/apache/hawq/pxf/plugins/hbase/HBaseFilterBuilder.java
 ---
@@ -165,6 +165,14 @@ private Filter 
handleSimpleOperations(FilterParser.Operation opId,
 ByteArrayComparable comparator = 
getComparator(hbaseColumn.columnTypeCode(),
 constant.constant());
 
+if(operatorsMap.get(opId) == null){
+//HBase not support HDOP_LIKE, use 'NOT NULL' Comarator
--- End diff --

comparator


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (HAWQ-1021) Need to log for some local_ssh function calls.

2016-08-26 Thread Goden Yao (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Goden Yao updated HAWQ-1021:

Fix Version/s: 2.0.1.0-incubating

> Need to log for some local_ssh function calls.
> --
>
> Key: HAWQ-1021
> URL: https://issues.apache.org/jira/browse/HAWQ-1021
> Project: Apache HAWQ
>  Issue Type: Bug
>Reporter: Paul Guo
>Assignee: hongwu
> Fix For: 2.0.1.0-incubating
>
>
> HAWQ management tools calls a lot of local_ssh() function to run some 
> external commands. The function is defined in hawqpylib/hawqlib.py. Many 
> callers does not set logger so we do not know any details about the command 
> running process. This is annoying when users/developers fail to run some 
> related commands and want to know the root cause quickly.
> Besides, there is two definitions of local_ssh(). Although they are not in 
> the same namespace, it is kind of annoying. We need to rename either one or 
> even both.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-1025) Check the consistency of AO/Parquet_FileLocations.Files.size attribute in extracted yaml file and the actual file size in HDFS.

2016-08-26 Thread Kyle R Dunn (JIRA)


[ 
https://issues.apache.org/jira/browse/HAWQ-1025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439308#comment-15439308
 ] 

Kyle R Dunn commented on HAWQ-1025:
---

This functionality again relates to the discussion in HAWQ-1011. In order to 
take advantage of the register from YAML feature in the context of why it was 
requested, we will need to allow the actual file size to be different from the 
catalog size attribute, at least until a proper recovery is performed. These 
are critical aspects to the initial purpose of this feature.

> Check the consistency of AO/Parquet_FileLocations.Files.size attribute in 
> extracted yaml file and the actual file size in HDFS.
> ---
>
> Key: HAWQ-1025
> URL: https://issues.apache.org/jira/browse/HAWQ-1025
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Command Line Tools
>Affects Versions: 2.0.1.0-incubating
>Reporter: hongwu
>Assignee: hongwu
> Fix For: 2.0.1.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger

2016-08-26 Thread Don Bosco Durai (JIRA)


[ 
https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438636#comment-15438636
 ] 

Don Bosco Durai commented on HAWQ-256:
--

Where is the latest API definition? From the JSON in the above comments, I 
think we have to make sure we have the following:
1. Groups for the user (either we send in the API or we can do the group lookup 
from the Ranger PDP (Policy Decision Point) Server itself.
2. Send IP of the client
3. Entire user query (if possible). I have seen users asking for entire query 
along with the audit record. We can truncate to a max predefined size.

I have copied [~sneethiraj], [~madhan.neethiraj] and [~kulkabhay] to give their 
comments also.

Thanks



> Integrate Security with Apache Ranger
> -
>
> Key: HAWQ-256
> URL: https://issues.apache.org/jira/browse/HAWQ-256
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: PXF, Security
>Reporter: Michael Andre Pearce (IG)
>Assignee: Lili Ma
> Fix For: backlog
>
> Attachments: HAWQRangerSupportDesign.pdf
>
>
> Integrate security with Apache Ranger for a unified Hadoop security solution. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HAWQ-256) Integrate Security with Apache Ranger

2016-08-26 Thread Lili Ma (JIRA)


[ 
https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438530#comment-15438530
 ] 

Lili Ma edited comment on HAWQ-256 at 8/26/16 7:56 AM:
---

Agree on the Grant/Revoke should be disabled if ranger is enabled.

In this way, the with grant option will not be considered.

Another thing is the owner management. Besides normal ACL, HAWQ has a 
definition of owner. The owner of the object can do any operation.  And for the 
owner part,  "Grant parent role to member role" and "reassign" are two SQL 
commands for owner control. I think we should move owner control to Ranger, to 
enable a fully Ranger-centralized access control. Your thoughts? 
[~vVineet][~bosco]


was (Author: lilima):
Agree on the Grant/Revoke should be disabled is ranger is enabled.

In this way, the with grant option will not be considered.

Another thing is the owner management. Besides normal ACL, HAWQ has a 
definition of owner. The owner of the object can do any operation.  And for the 
owner part,  "Grant parent role to member role" and "reassign" are two SQL 
commands for owner control. I think we should move owner control to Ranger, to 
enable a fully Ranger-centralized access control. Your thoughts? 
[~vVineet][~bosco]

> Integrate Security with Apache Ranger
> -
>
> Key: HAWQ-256
> URL: https://issues.apache.org/jira/browse/HAWQ-256
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: PXF, Security
>Reporter: Michael Andre Pearce (IG)
>Assignee: Lili Ma
> Fix For: backlog
>
> Attachments: HAWQRangerSupportDesign.pdf
>
>
> Integrate security with Apache Ranger for a unified Hadoop security solution. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HAWQ-1026) HAWQ sync user information from LDAP

2016-08-26 Thread Lili Ma (JIRA)

Lili Ma created HAWQ-1026:
-

 Summary: HAWQ sync user information from LDAP
 Key: HAWQ-1026
 URL: https://issues.apache.org/jira/browse/HAWQ-1026
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: Security
Reporter: Lili Ma
Assignee: Lei Chang


HAWQ sync user information from LDAP, so that HAWQ user doesn't need to 
manually create role in HAWQ for all users in LDAP.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq issue #839: HAWQ-983. Fix bug that minirepro generates wrong ...

2016-08-26 Thread hsyuan

Github user hsyuan commented on the issue:

https://github.com/apache/incubator-hawq/pull/839
  
Pushed to master:

https://github.com/apache/incubator-hawq/commit/19124b7da032b61d9215520483b67a109246e363

https://github.com/apache/incubator-hawq/commit/08a1c6c2af705df850b1b9bbd6188d0cd2036b42


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #839: HAWQ-983. Fix bug that minirepro generates...

2016-08-26 Thread hsyuan

Github user hsyuan closed the pull request at:

https://github.com/apache/incubator-hawq/pull/839


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq issue #838: HAWQ-934. Populate canSetTag of PlannedStmt from ...

2016-08-26 Thread hsyuan

Github user hsyuan commented on the issue:

https://github.com/apache/incubator-hawq/pull/838
  
pushed to master:

https://github.com/apache/incubator-hawq/commit/a17647be029a42c85e659cfc7fe253708a036bd7


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #838: HAWQ-934. Populate canSetTag of PlannedStm...

2016-08-26 Thread hsyuan

Github user hsyuan closed the pull request at:

https://github.com/apache/incubator-hawq/pull/838


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger

2016-08-26 Thread Don Bosco Durai (JIRA)


[ 
https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438560#comment-15438560
 ] 

Don Bosco Durai commented on HAWQ-256:
--

I agree. We can disable grant/revoke on the HAWQ side it Ranger is enabled. I 
feel, those using Ranger will prefer to manage the policies from one place.

Also, even if we support GRANT/REVOKE from the HAWQ SQL command, we shouldn't 
expect that the behavior will be same when Ranger is enabled. E.g. Currently, 
when you give delegated admin privilege to any user (e.g. user 1) for the 
resource, then user1 can give any access to other users (e.g. user2) for that 
resource regardless what permission user1 has. This addresses use case where 
you don't want Admin to read/write, but be able to manage permissions for 
others. 

> Integrate Security with Apache Ranger
> -
>
> Key: HAWQ-256
> URL: https://issues.apache.org/jira/browse/HAWQ-256
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: PXF, Security
>Reporter: Michael Andre Pearce (IG)
>Assignee: Lili Ma
> Fix For: backlog
>
> Attachments: HAWQRangerSupportDesign.pdf
>
>
> Integrate security with Apache Ranger for a unified Hadoop security solution. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HAWQ-968) Incorrect free in url_close

2016-08-26 Thread hongwu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hongwu reassigned HAWQ-968:
---

Assignee: hongwu  (was: Lei Chang)

> Incorrect free in url_close
> ---
>
> Key: HAWQ-968
> URL: https://issues.apache.org/jira/browse/HAWQ-968
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: External Tables
>Affects Versions: backlog
>Reporter: hongwu
>Assignee: hongwu
> Fix For: backlog
>
>
> There is potential double free risk in url_close of url.c



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HAWQ-638) gpload bug using pip installed PyGreSQL

2016-08-26 Thread hongwu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hongwu reassigned HAWQ-638:
---

Assignee: hongwu  (was: Lei Chang)

> gpload bug using pip installed PyGreSQL
> ---
>
> Key: HAWQ-638
> URL: https://issues.apache.org/jira/browse/HAWQ-638
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Command Line Tools
>Reporter: hongwu
>Assignee: hongwu
> Fix For: 2.0.0.0-incubating
>
>
> Since greenplum's gpload is based on private patch upon PyGreSQL for internal 
> usage, and gpload.py is copied from greenplum incompletely, it will generate 
> error while using gpload tools.
> Details: 
> self.db.notices() depends on the implementation of pg.DB.notices, which was 
> implemented internal in greenplum, it is wrong to use this attribute in 
> gpload tool of hawq.
> Reference:
> https://github.com/apache/incubator-hawq/blob/master/tools/bin/gpload.py#L704
> https://github.com/greenplum-db/gpdb/blob/master/gpMgmt/bin/pythonSrc/PyGreSQL-4.0/pgmodule.c#L2929



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HAWQ-647) configure compilation flag bug

2016-08-26 Thread hongwu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hongwu reassigned HAWQ-647:
---

Assignee: hongwu  (was: Lei Chang)

> configure compilation flag bug
> --
>
> Key: HAWQ-647
> URL: https://issues.apache.org/jira/browse/HAWQ-647
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Build
>Reporter: hongwu
>Assignee: hongwu
> Fix For: 2.0.0.0-incubating
>
>
> It is dangerous to use -O3 compilation flag in debug mode with -g together...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HAWQ-707) Remove google test dependency from libhdfs3 and libyarn folder

2016-08-26 Thread hongwu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hongwu reassigned HAWQ-707:
---

Assignee: hongwu  (was: Lei Chang)

> Remove google test dependency from libhdfs3 and libyarn folder
> --
>
> Key: HAWQ-707
> URL: https://issues.apache.org/jira/browse/HAWQ-707
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Build
>Reporter: hongwu
>Assignee: hongwu
> Fix For: 2.0.0.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HAWQ-734) refactor feature test build

2016-08-26 Thread hongwu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hongwu reassigned HAWQ-734:
---

Assignee: hongwu  (was: Jiali Yao)

> refactor feature test build
> ---
>
> Key: HAWQ-734
> URL: https://issues.apache.org/jira/browse/HAWQ-734
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Tests
>Reporter: hongwu
>Assignee: hongwu
> Fix For: 2.0.0.0-incubating
>
>
> Refactor feature test build for general makefile which could avoid conflict 
> pushing pull requests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HAWQ-754) Open googletest code inside hawq for test usage

2016-08-26 Thread hongwu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hongwu reassigned HAWQ-754:
---

Assignee: hongwu  (was: Lei Chang)

> Open googletest code inside hawq for test usage
> ---
>
> Key: HAWQ-754
> URL: https://issues.apache.org/jira/browse/HAWQ-754
> Project: Apache HAWQ
>  Issue Type: Test
>  Components: Build
>Reporter: hongwu
>Assignee: hongwu
> Fix For: 2.0.0.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HAWQ-721) New feature test skeleton

2016-08-26 Thread hongwu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hongwu reassigned HAWQ-721:
---

Assignee: hongwu  (was: Lei Chang)

> New feature test skeleton
> -
>
> Key: HAWQ-721
> URL: https://issues.apache.org/jira/browse/HAWQ-721
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: Build
>Reporter: hongwu
>Assignee: hongwu
> Fix For: 2.0.0.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HAWQ-735) Import thrift-0.9.1

2016-08-26 Thread hongwu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hongwu reassigned HAWQ-735:
---

Assignee: hongwu  (was: Lei Chang)

> Import thrift-0.9.1
> ---
>
> Key: HAWQ-735
> URL: https://issues.apache.org/jira/browse/HAWQ-735
> Project: Apache HAWQ
>  Issue Type: Improvement
>  Components: Build
>Reporter: hongwu
>Assignee: hongwu
> Fix For: 2.0.0.0-incubating
>
>
> Since apt-get do not have `libthrift-dev` source package, we need to import 
> thrift inside hawq and supply a flag: --with-thrift.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HAWQ-772) Hotfix for HAWQ-760

2016-08-26 Thread hongwu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hongwu reassigned HAWQ-772:
---

Assignee: hongwu  (was: Lei Chang)

> Hotfix for HAWQ-760
> ---
>
> Key: HAWQ-772
> URL: https://issues.apache.org/jira/browse/HAWQ-772
> Project: Apache HAWQ
>  Issue Type: Bug
>  Components: Build
>Reporter: hongwu
>Assignee: hongwu
> Fix For: 2.0.0.0-incubating
>
>
> HAWQ-760 code is based on old code base, which conflicts with HAWQ-768, the 
> new merged commit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HAWQ-757) Hotfix feature-test building bug.

2016-08-26 Thread hongwu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hongwu reassigned HAWQ-757:
---

Assignee: hongwu  (was: Lei Chang)

> Hotfix feature-test building bug.
> -
>
> Key: HAWQ-757
> URL: https://issues.apache.org/jira/browse/HAWQ-757
> Project: Apache HAWQ
>  Issue Type: Test
>  Components: Build
>Reporter: hongwu
>Assignee: hongwu
> Fix For: 2.0.0.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-1023) Incorrect usage of java.lang.String.replaceAll

2016-08-26 Thread hongwu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hongwu updated HAWQ-1023:
-
Fix Version/s: backlog

> Incorrect usage of java.lang.String.replaceAll
> --
>
> Key: HAWQ-1023
> URL: https://issues.apache.org/jira/browse/HAWQ-1023
> Project: Apache HAWQ
>  Issue Type: Bug
>Reporter: hongwu
>Assignee: Lei Chang
> Fix For: backlog
>
>
> Incorrect usage for java.lang.String.replaceAll generate useless calls:
> https://github.com/apache/incubator-hawq/blob/master/contrib/hawq-hadoop/hawq-mapreduce-common/src/main/java/com/pivotal/hawq/mapreduce/datatype/HAWQPath.java#L51
> https://github.com/apache/incubator-hawq/blob/master/contrib/hawq-hadoop/hawq-mapreduce-common/src/main/java/com/pivotal/hawq/mapreduce/datatype/HAWQPoint.java#L45



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HAWQ-1023) Incorrect usage of java.lang.String.replaceAll

2016-08-26 Thread hongwu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hongwu updated HAWQ-1023:
-
Priority: Minor  (was: Major)

> Incorrect usage of java.lang.String.replaceAll
> --
>
> Key: HAWQ-1023
> URL: https://issues.apache.org/jira/browse/HAWQ-1023
> Project: Apache HAWQ
>  Issue Type: Bug
>Reporter: hongwu
>Assignee: Lei Chang
>Priority: Minor
> Fix For: backlog
>
>
> Incorrect usage for java.lang.String.replaceAll generate useless calls:
> https://github.com/apache/incubator-hawq/blob/master/contrib/hawq-hadoop/hawq-mapreduce-common/src/main/java/com/pivotal/hawq/mapreduce/datatype/HAWQPath.java#L51
> https://github.com/apache/incubator-hawq/blob/master/contrib/hawq-hadoop/hawq-mapreduce-common/src/main/java/com/pivotal/hawq/mapreduce/datatype/HAWQPoint.java#L45



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HAWQ-1023) Incorrect usage of java.lang.String.replaceAll

2016-08-26 Thread hongwu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HAWQ-1023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hongwu reassigned HAWQ-1023:


Assignee: hongwu  (was: Lei Chang)

> Incorrect usage of java.lang.String.replaceAll
> --
>
> Key: HAWQ-1023
> URL: https://issues.apache.org/jira/browse/HAWQ-1023
> Project: Apache HAWQ
>  Issue Type: Bug
>Reporter: hongwu
>Assignee: hongwu
>Priority: Minor
> Fix For: backlog
>
>
> Incorrect usage for java.lang.String.replaceAll generate useless calls:
> https://github.com/apache/incubator-hawq/blob/master/contrib/hawq-hadoop/hawq-mapreduce-common/src/main/java/com/pivotal/hawq/mapreduce/datatype/HAWQPath.java#L51
> https://github.com/apache/incubator-hawq/blob/master/contrib/hawq-hadoop/hawq-mapreduce-common/src/main/java/com/pivotal/hawq/mapreduce/datatype/HAWQPoint.java#L45



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...

2016-08-26 Thread hsyuan

Github user hsyuan commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/837#discussion_r76370526
  
--- Diff: 
pxf/pxf-api/src/test/java/org/apache/hawq/pxf/api/FilterParserTest.java ---
@@ -215,6 +215,10 @@ public void parseColumnOnLeft() throws Exception {
 filter = "a1c2o7";
 op = Operation.HDOP_AND;
 runParseOneOperation("this filter was build from HDOP_AND", 
filter, op);
+
+filter = "a1c2o8";
+op = Operation.HDOP_LIKE;
+runParseOneOperation("this filter was build from HDOP_LIKE", 
filter, op);
--- End diff --

build -> built


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #845: HAWQ-999. Treat hash table as random when ...

2016-08-26 Thread hsyuan

Github user hsyuan commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/845#discussion_r76369938
  
--- Diff: src/backend/cdb/cdbdatalocality.c ---
@@ -3791,12 +3791,27 @@ run_allocation_algorithm(SplitAllocResult *result, 
List *virtual_segments, Query
targetPolicy = GpPolicyFetch(CurrentMemoryContext, myrelid);
bool isRelationHash = is_relation_hash(targetPolicy);
 
+   int fileCountInRelation = list_length(rel_data->files);
+   bool FileCountBucketNumMismatch = false;
+   if (targetPolicy->bucketnum > 0) {
+ FileCountBucketNumMismatch = fileCountInRelation %
--- End diff --

why not `fileCountBucketNumMismatch = fileCountInRelation % 
targetPolicy->bucketnum != 0`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #845: HAWQ-999. Treat hash table as random when ...

2016-08-26 Thread hsyuan

Github user hsyuan commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/845#discussion_r76369662
  
--- Diff: src/backend/utils/misc/guc.c ---
@@ -3176,6 +3177,17 @@ static struct config_bool ConfigureNamesBool[] =
},
 
{
+   {"allow_file_count_bucket_num_mismatch", PGC_POSTMASTER, 
CLIENT_CONN_LOCALE,
+ gettext_noop("allow hash table to be treated as random when 
file count and"
--- End diff --

allow file count and bucket number mismatch by viewing hash distributed 
table as randomly distributed table


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger

2016-08-26 Thread Lili Ma (JIRA)


[ 
https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438530#comment-15438530
 ] 

Lili Ma commented on HAWQ-256:
--

Agree on the Grant/Revoke should be disabled is ranger is enabled.

In this way, the with grant option will not be considered.

Another thing is the owner management. Besides normal ACL, HAWQ has a 
definition of owner. The owner of the object can do any operation.  And for the 
owner part,  "Grant parent role to member role" and "reassign" are two SQL 
commands for owner control. I think we should move owner control to Ranger, to 
enable a fully Ranger-centralized access control. Your thoughts? 
[~vVineet][~bosco]

> Integrate Security with Apache Ranger
> -
>
> Key: HAWQ-256
> URL: https://issues.apache.org/jira/browse/HAWQ-256
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: PXF, Security
>Reporter: Michael Andre Pearce (IG)
>Assignee: Lili Ma
> Fix For: backlog
>
> Attachments: HAWQRangerSupportDesign.pdf
>
>
> Integrate security with Apache Ranger for a unified Hadoop security solution. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HAWQ-256) Integrate Security with Apache Ranger

2016-08-26 Thread Lili Ma (JIRA)


[ 
https://issues.apache.org/jira/browse/HAWQ-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438531#comment-15438531
 ] 

Lili Ma commented on HAWQ-256:
--

[~bosco] Do you have any feedback on the API definition from Hortonworks side? 
Thanks 

> Integrate Security with Apache Ranger
> -
>
> Key: HAWQ-256
> URL: https://issues.apache.org/jira/browse/HAWQ-256
> Project: Apache HAWQ
>  Issue Type: New Feature
>  Components: PXF, Security
>Reporter: Michael Andre Pearce (IG)
>Assignee: Lili Ma
> Fix For: backlog
>
> Attachments: HAWQRangerSupportDesign.pdf
>
>
> Integrate security with Apache Ranger for a unified Hadoop security solution. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-hawq pull request #845: HAWQ-999. Treat hash table as random when ...

2016-08-26 Thread hsyuan

Github user hsyuan commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/845#discussion_r76369259
  
--- Diff: src/backend/cdb/cdbdatalocality.c ---
@@ -3791,12 +3791,27 @@ run_allocation_algorithm(SplitAllocResult *result, 
List *virtual_segments, Query
targetPolicy = GpPolicyFetch(CurrentMemoryContext, myrelid);
bool isRelationHash = is_relation_hash(targetPolicy);
 
+   int fileCountInRelation = list_length(rel_data->files);
+   bool FileCountBucketNumMismatch = false;
+   if (targetPolicy->bucketnum > 0) {
+ FileCountBucketNumMismatch = fileCountInRelation %
+   targetPolicy->bucketnum == 0 ? false : true;
+   }
+   if (FileCountBucketNumMismatch && 
!allow_file_count_bucket_num_mismatch) {
+ elog(ERROR, "file count %d in catalog is not in proportion to 
the bucket "
+ "number %d of hash table with oid=%u, some data may be 
lost, if you "
--- End diff --

number %d of distribution policy for table with oid = %u, some data may be 
missing or invalid.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #845: HAWQ-999. Treat hash table as random when ...

2016-08-26 Thread hsyuan

Github user hsyuan commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/845#discussion_r76369524
  
--- Diff: src/backend/utils/misc/guc.c ---
@@ -450,6 +450,7 @@ char   *Debug_dtm_action_protocol_str;
 
 /* Enable check for compatibility of encoding and locale in createdb */
 bool   gp_encoding_check_locale_compatibility;
+bool  allow_file_count_bucket_num_mismatch;
--- End diff --

indent


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #845: HAWQ-999. Treat hash table as random when ...

2016-08-26 Thread hsyuan

Github user hsyuan commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/845#discussion_r76369429
  
--- Diff: src/backend/cdb/cdbdatalocality.c ---
@@ -3791,12 +3791,27 @@ run_allocation_algorithm(SplitAllocResult *result, 
List *virtual_segments, Query
targetPolicy = GpPolicyFetch(CurrentMemoryContext, myrelid);
bool isRelationHash = is_relation_hash(targetPolicy);
 
+   int fileCountInRelation = list_length(rel_data->files);
+   bool FileCountBucketNumMismatch = false;
+   if (targetPolicy->bucketnum > 0) {
+ FileCountBucketNumMismatch = fileCountInRelation %
+   targetPolicy->bucketnum == 0 ? false : true;
+   }
+   if (FileCountBucketNumMismatch && 
!allow_file_count_bucket_num_mismatch) {
+ elog(ERROR, "file count %d in catalog is not in proportion to 
the bucket "
+ "number %d of hash table with oid=%u, some data may be 
lost, if you "
+ "still want to continue the query by considering the 
table as random, set GUC "
--- End diff --

You can force running the query by viewing the table as randomly 
distributed table. set GUC allow_file_count_bucket_num_mismatch on to enable 
this feature.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #845: HAWQ-999. Treat hash table as random when ...

2016-08-26 Thread hsyuan

Github user hsyuan commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/845#discussion_r76368733
  
--- Diff: src/backend/cdb/cdbdatalocality.c ---
@@ -3791,12 +3791,27 @@ run_allocation_algorithm(SplitAllocResult *result, 
List *virtual_segments, Query
targetPolicy = GpPolicyFetch(CurrentMemoryContext, myrelid);
bool isRelationHash = is_relation_hash(targetPolicy);
 
+   int fileCountInRelation = list_length(rel_data->files);
+   bool FileCountBucketNumMismatch = false;
--- End diff --

why is this var capital leading? `FileCountBucketNumMismatch`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #837: HAWQ-779 support pxf filter pushdwon at th...

2016-08-26 Thread jiadexin

Github user jiadexin commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/837#discussion_r76368428
  
--- Diff: src/backend/optimizer/plan/createplan.c ---
@@ -1146,7 +1146,7 @@ static char** create_pxf_plan(char **segdb_file_map, 
RelOptInfo *rel, int total_
Relation relation = RelationIdGetRelation(planner_rt_fetch(scan_relid, 
ctx->root)->relid);
segdb_work_map = map_hddata_2gp_segments(uri_str, 

 total_segs, segs_participating,
-   
 relation, NULL);
+   
 relation, ctx->root->parse->jointree->quals);
--- End diff --

ok.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #845: HAWQ-999. Treat hash table as random when ...

2016-08-26 Thread hsyuan

Github user hsyuan commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/845#discussion_r76368378
  
--- Diff: src/backend/cdb/cdbdatalocality.c ---
@@ -3822,8 +3837,13 @@ run_allocation_algorithm(SplitAllocResult *result, 
List *virtual_segments, Query
uint64_t before_run_allocate_hash_or_random = 
gettime_microsec();
/*allocate hash relation*/
if (isRelationHash) {
-   if (context->keep_hash && 
assignment_context.virtual_segment_num
-   == targetPolicy->bucketnum) {
+ /*
+  * if file count of the table is not equal to or multiple of
+  * bucket number, we should process it as random table.
+  */
+   if (context->keep_hash
+   && assignment_context.virtual_segment_num== 
targetPolicy->bucketnum
--- End diff --

space before ==


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #861: HAWQ-1001. Implement HAWQ user ACL check t...

2016-08-26 Thread hsyuan

Github user hsyuan commented on a diff in the pull request:

https://github.com/apache/incubator-hawq/pull/861#discussion_r76368047
  
--- Diff: src/backend/utils/misc/guc.c ---
@@ -4323,6 +4324,16 @@ static struct config_bool ConfigureNamesBool[] =
},
 
{
+{"enable_ranger", PGC_POSTMASTER, CONN_AUTH_SETTINGS,
+ gettext_noop("support to using ranger to manage hawq privilege."),
--- End diff --

Enable Apache Ranger for HAWQ privilege management.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

42 matches

Mail list logo