[GitHub] incubator-hawq issue #720: HAWQ-823. Initial Amazon S3 External Table Suppor...

[Jack-LiuXiong]

Github user Jack-LiuXiong commented on the issue:

https://github.com/apache/incubator-hawq/pull/720
  
Hello, is there any version restriction for this practice?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1281: Update pxf classpath file to make it gene...

[shivzone]

Github user shivzone closed the pull request at:

https://github.com/apache/incubator-hawq/pull/1281


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hawq pull request #1281: Update pxf classpath file to make it gene...

[shivzone]

GitHub user shivzone opened a pull request:

https://github.com/apache/incubator-hawq/pull/1281

Update pxf classpath file to make it generic



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shivzone/incubator-hawq patch-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hawq/pull/1281.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1281


commit 8b16623d6c10925eebb444c1a36e45a838d467c3
Author: Shivram Mani 
Date:   2017-08-28T18:50:15Z

Update pxf classpath file to make it generic




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (HAWQ-1520) gpcheckhdfs should skip hdfs trash directory

[Hongxu Ma (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143497#comment-16143497
 ] 

Hongxu Ma commented on HAWQ-1520:
-

about hdfs trash:
https://www.cloudera.com/documentation/enterprise/5-8-x/topics/cm_mc_config_trash.html

> gpcheckhdfs should skip hdfs trash directory
> 
>
> Key: HAWQ-1520
> URL: https://issues.apache.org/jira/browse/HAWQ-1520
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Command Line Tools
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: 2.3.0.0-incubating
>
>
> When enable hdfs trash feature, there is a *Trash* directory under the 
> encryption zone.
> Example:
> {code}
> [gpadmin@test1 hawq]$ sudo -u hdfs hdfs crypto -listZones
> /hawq/hawq-1503886333  tdekey
> [gpadmin@test1 hawq]$ hdfs dfs -ls /hawq/hawq-1503886333/
> Found 1 items
> drwxrwxrwt   - hdfs hdfs  0 2017-08-27 23:59 
> /hawq/hawq-1503886333/.Trash
> {code}
> But gpcheckhdfs scrpit doesn't consider it(/hawq/hawq-1503886333) as an empty 
> directory.
> We should fix it to skip this trash directory.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HAWQ-1520) gpcheckhdfs should skip hdfs trash directory

[Hongxu Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hongxu Ma reassigned HAWQ-1520:
---

Assignee: Hongxu Ma  (was: Radar Lei)

> gpcheckhdfs should skip hdfs trash directory
> 
>
> Key: HAWQ-1520
> URL: https://issues.apache.org/jira/browse/HAWQ-1520
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Command Line Tools
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: 2.3.0.0-incubating
>
>
> When enable hdfs trash feature, there is a *Trash* directory under the 
> encryption zone.
> Example:
> {code}
> [gpadmin@test1 hawq]$ sudo -u hdfs hdfs crypto -listZones
> /hawq/hawq-1503886333  tdekey
> [gpadmin@test1 hawq]$ hdfs dfs -ls /hawq/hawq-1503886333/
> Found 1 items
> drwxrwxrwt   - hdfs hdfs  0 2017-08-27 23:59 
> /hawq/hawq-1503886333/.Trash
> {code}
> But gpcheckhdfs scrpit doesn't consider it(/hawq/hawq-1503886333) as an empty 
> directory.
> We should fix it to skip this trash directory.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HAWQ-1520) gpcheckhdfs should skip hdfs trash directory

[Hongxu Ma (JIRA)]

Hongxu Ma created HAWQ-1520:
---

 Summary: gpcheckhdfs should skip hdfs trash directory
 Key: HAWQ-1520
 URL: https://issues.apache.org/jira/browse/HAWQ-1520
 Project: Apache HAWQ
  Issue Type: Sub-task
  Components: Command Line Tools
Reporter: Hongxu Ma
Assignee: Radar Lei
 Fix For: 2.3.0.0-incubating


When enable hdfs trash feature, there is a *Trash* directory under the 
encryption zone.
Example:

{code}
[gpadmin@test1 hawq]$ sudo -u hdfs hdfs crypto -listZones
/hawq/hawq-1503886333  tdekey
[gpadmin@test1 hawq]$ hdfs dfs -ls /hawq/hawq-1503886333/
Found 1 items
drwxrwxrwt   - hdfs hdfs  0 2017-08-27 23:59 
/hawq/hawq-1503886333/.Trash
{code}

But gpcheckhdfs scrpit doesn't consider it(/hawq/hawq-1503886333) as an empty 
directory.
We should fix it to skip this trash directory.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HAWQ-1510) Add TDE-related functionality into hawq command line tools

[Hongxu Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hongxu Ma updated HAWQ-1510:

Description: 
1, hawq init
the only way to enable tde in hawq:
user should give a key name(already created by hadoop key command) parameter 
when execuate the init command, it makes the whole hawq_default directory as an 
encryption zone.

note:
* cannot support transfer the existed(and non-empty) hawq_default directory 
into an encryption zone.
* create encryption zone need hdfs *superuser privilege*, so if hawq user and 
hdfs superuser is not the same one, you should create the encryption zone on 
hawq directory manually before running hawq-init script, example:
{code}
hdfs crypto -createZone -keyName key_demo -path /hawq_default/
{code}

command:
{code}
hawq init cluster --tde_keyname key_demo
{code}

-2, hawq state-
-show the encryption zone info if user enable tde in hawq.-

3, hawq register 
cannot register file in different encryption zones / un-encryption zones.

4, hawq extract
give user a warning of the table data is stored in encryption zone if user 
enable tde in hawq.


  was:
1, hawq init
the only way to enable tde in hawq:
user should give a key name(already created by hadoop key command) parameter 
when execuate the init command, it makes the whole hawq_default directory as an 
encryption zone.

note:
* cannot support transfer the existed(and non-empty) hawq_default directory 
into an encryption zone.
* create encryption zone need hdfs *superuser privilege*, so if hawq user and 
hdfs superuser is not the same one, you should create the encryption zone on 
hawq directory manually before running hawq-init script.

command:
{code}
hawq init cluster --tde_keyname key_demo
{code}

-2, hawq state-
-show the encryption zone info if user enable tde in hawq.-

3, hawq register 
cannot register file in different encryption zones / un-encryption zones.

4, hawq extract
give user a warning of the table data is stored in encryption zone if user 
enable tde in hawq.



> Add TDE-related functionality into hawq command line tools
> --
>
> Key: HAWQ-1510
> URL: https://issues.apache.org/jira/browse/HAWQ-1510
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Command Line Tools
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: 2.3.0.0-incubating
>
>
> 1, hawq init
> the only way to enable tde in hawq:
> user should give a key name(already created by hadoop key command) parameter 
> when execuate the init command, it makes the whole hawq_default directory as 
> an encryption zone.
> note:
> * cannot support transfer the existed(and non-empty) hawq_default directory 
> into an encryption zone.
> * create encryption zone need hdfs *superuser privilege*, so if hawq user and 
> hdfs superuser is not the same one, you should create the encryption zone on 
> hawq directory manually before running hawq-init script, example:
> {code}
> hdfs crypto -createZone -keyName key_demo -path /hawq_default/
> {code}
> command:
> {code}
> hawq init cluster --tde_keyname key_demo
> {code}
> -2, hawq state-
> -show the encryption zone info if user enable tde in hawq.-
> 3, hawq register 
> cannot register file in different encryption zones / un-encryption zones.
> 4, hawq extract
> give user a warning of the table data is stored in encryption zone if user 
> enable tde in hawq.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HAWQ-1510) Add TDE-related functionality into hawq command line tools

[Hongxu Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hongxu Ma updated HAWQ-1510:

Description: 
1, hawq init
the only way to enable tde in hawq:
user should give a key name(already created by hadoop key command) parameter 
when execuate the init command, it makes the whole hawq_default directory as an 
encryption zone.

note:
* cannot support transfer the existed(and non-empty) hawq_default directory 
into an encryption zone.
* if hawq user and hdfs superuser is not the same one, you should create the 
encryption zone on hawq directory manually before running hawq-init script.

command:
{code}
hawq init cluster --tde_keyname key_demo
{code}

-2, hawq state-
-show the encryption zone info if user enable tde in hawq.-

3, hawq register 
cannot register file in different encryption zones / un-encryption zones.

4, hawq extract
give user a warning of the table data is stored in encryption zone if user 
enable tde in hawq.


  was:
1, hawq init
the only way to enable tde in hawq:
user should give a key name(already created by hadoop key command) parameter 
when execuate the init command, it makes the whole hawq_default directory as an 
encryption zone.

note:
* cannot support transfer the existed(and non-empty) hawq_default directory 
into an encryption zone.
* 

command:
{code}
hawq init cluster --tde_keyname key_demo
{code}

-2, hawq state-
-show the encryption zone info if user enable tde in hawq.-

3, hawq register 
cannot register file in different encryption zones / un-encryption zones.

4, hawq extract
give user a warning of the table data is stored in encryption zone if user 
enable tde in hawq.



> Add TDE-related functionality into hawq command line tools
> --
>
> Key: HAWQ-1510
> URL: https://issues.apache.org/jira/browse/HAWQ-1510
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Command Line Tools
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: 2.3.0.0-incubating
>
>
> 1, hawq init
> the only way to enable tde in hawq:
> user should give a key name(already created by hadoop key command) parameter 
> when execuate the init command, it makes the whole hawq_default directory as 
> an encryption zone.
> note:
> * cannot support transfer the existed(and non-empty) hawq_default directory 
> into an encryption zone.
> * if hawq user and hdfs superuser is not the same one, you should create the 
> encryption zone on hawq directory manually before running hawq-init script.
> command:
> {code}
> hawq init cluster --tde_keyname key_demo
> {code}
> -2, hawq state-
> -show the encryption zone info if user enable tde in hawq.-
> 3, hawq register 
> cannot register file in different encryption zones / un-encryption zones.
> 4, hawq extract
> give user a warning of the table data is stored in encryption zone if user 
> enable tde in hawq.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HAWQ-1510) Add TDE-related functionality into hawq command line tools

[Hongxu Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hongxu Ma updated HAWQ-1510:

Description: 
1, hawq init
the only way to enable tde in hawq:
user should give a key name(already created by hadoop key command) parameter 
when execuate the init command, it makes the whole hawq_default directory as an 
encryption zone.

note:
* cannot support transfer the existed(and non-empty) hawq_default directory 
into an encryption zone.
* create encryption zone need hdfs *superuser privilege*, so if hawq user and 
hdfs superuser is not the same one, you should create the encryption zone on 
hawq directory manually before running hawq-init script.

command:
{code}
hawq init cluster --tde_keyname key_demo
{code}

-2, hawq state-
-show the encryption zone info if user enable tde in hawq.-

3, hawq register 
cannot register file in different encryption zones / un-encryption zones.

4, hawq extract
give user a warning of the table data is stored in encryption zone if user 
enable tde in hawq.


  was:
1, hawq init
the only way to enable tde in hawq:
user should give a key name(already created by hadoop key command) parameter 
when execuate the init command, it makes the whole hawq_default directory as an 
encryption zone.

note:
* cannot support transfer the existed(and non-empty) hawq_default directory 
into an encryption zone.
* if hawq user and hdfs superuser is not the same one, you should create the 
encryption zone on hawq directory manually before running hawq-init script.

command:
{code}
hawq init cluster --tde_keyname key_demo
{code}

-2, hawq state-
-show the encryption zone info if user enable tde in hawq.-

3, hawq register 
cannot register file in different encryption zones / un-encryption zones.

4, hawq extract
give user a warning of the table data is stored in encryption zone if user 
enable tde in hawq.



> Add TDE-related functionality into hawq command line tools
> --
>
> Key: HAWQ-1510
> URL: https://issues.apache.org/jira/browse/HAWQ-1510
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Command Line Tools
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: 2.3.0.0-incubating
>
>
> 1, hawq init
> the only way to enable tde in hawq:
> user should give a key name(already created by hadoop key command) parameter 
> when execuate the init command, it makes the whole hawq_default directory as 
> an encryption zone.
> note:
> * cannot support transfer the existed(and non-empty) hawq_default directory 
> into an encryption zone.
> * create encryption zone need hdfs *superuser privilege*, so if hawq user and 
> hdfs superuser is not the same one, you should create the encryption zone on 
> hawq directory manually before running hawq-init script.
> command:
> {code}
> hawq init cluster --tde_keyname key_demo
> {code}
> -2, hawq state-
> -show the encryption zone info if user enable tde in hawq.-
> 3, hawq register 
> cannot register file in different encryption zones / un-encryption zones.
> 4, hawq extract
> give user a warning of the table data is stored in encryption zone if user 
> enable tde in hawq.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HAWQ-1510) Add TDE-related functionality into hawq command line tools

[Hongxu Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HAWQ-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hongxu Ma updated HAWQ-1510:

Description: 
1, hawq init
the only way to enable tde in hawq:
user should give a key name(already created by hadoop key command) parameter 
when execuate the init command, it makes the whole hawq_default directory as an 
encryption zone.

note:
* cannot support transfer the existed(and non-empty) hawq_default directory 
into an encryption zone.
* 

command:
{code}
hawq init cluster --tde_keyname key_demo
{code}

-2, hawq state-
-show the encryption zone info if user enable tde in hawq.-

3, hawq register 
cannot register file in different encryption zones / un-encryption zones.

4, hawq extract
give user a warning of the table data is stored in encryption zone if user 
enable tde in hawq.


  was:
1, hawq init
the only way to enable tde in hawq:
user should give a key name(already created by hadoop key command) parameter 
when execuate the init command, it makes the whole hawq_default directory as an 
encryption zone.

note:
cannot support transfer the existed(and non-empty) hawq_default directory into 
an encryption zone.

command:
{code}
hawq init cluster --tde_keyname key_demo
{code}

-2, hawq state-
-show the encryption zone info if user enable tde in hawq.-

3, hawq register 
cannot register file in different encryption zones / un-encryption zones.

4, hawq extract
give user a warning of the table data is stored in encryption zone if user 
enable tde in hawq.



> Add TDE-related functionality into hawq command line tools
> --
>
> Key: HAWQ-1510
> URL: https://issues.apache.org/jira/browse/HAWQ-1510
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Command Line Tools
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: 2.3.0.0-incubating
>
>
> 1, hawq init
> the only way to enable tde in hawq:
> user should give a key name(already created by hadoop key command) parameter 
> when execuate the init command, it makes the whole hawq_default directory as 
> an encryption zone.
> note:
> * cannot support transfer the existed(and non-empty) hawq_default directory 
> into an encryption zone.
> * 
> command:
> {code}
> hawq init cluster --tde_keyname key_demo
> {code}
> -2, hawq state-
> -show the encryption zone info if user enable tde in hawq.-
> 3, hawq register 
> cannot register file in different encryption zones / un-encryption zones.
> 4, hawq extract
> give user a warning of the table data is stored in encryption zone if user 
> enable tde in hawq.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HAWQ-1510) Add TDE-related functionality into hawq command line tools

[Hongxu Ma (JIRA)]

[ 
https://issues.apache.org/jira/browse/HAWQ-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143442#comment-16143442
 ] 

Hongxu Ma commented on HAWQ-1510:
-

Note: 
create encryption zone need hdfs **superuser privilege**.
so if hawq user and hdfs superuser is not the same one, you should create the 
encryption zone on hawq directory manually before running hawq-init script.

> Add TDE-related functionality into hawq command line tools
> --
>
> Key: HAWQ-1510
> URL: https://issues.apache.org/jira/browse/HAWQ-1510
> Project: Apache HAWQ
>  Issue Type: Sub-task
>  Components: Command Line Tools
>Reporter: Hongxu Ma
>Assignee: Hongxu Ma
> Fix For: 2.3.0.0-incubating
>
>
> 1, hawq init
> the only way to enable tde in hawq:
> user should give a key name(already created by hadoop key command) parameter 
> when execuate the init command, it makes the whole hawq_default directory as 
> an encryption zone.
> note:
> cannot support transfer the existed(and non-empty) hawq_default directory 
> into an encryption zone.
> command:
> {code}
> hawq init cluster --tde_keyname key_demo
> {code}
> -2, hawq state-
> -show the encryption zone info if user enable tde in hawq.-
> 3, hawq register 
> cannot register file in different encryption zones / un-encryption zones.
> 4, hawq extract
> give user a warning of the table data is stored in encryption zone if user 
> enable tde in hawq.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)