[jira] [Updated] (HADOOP-15012) Add readahead, dropbehind, and unbuffer to StreamCapabilities

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen updated HADOOP-15012:
---
Hadoop Flags: Reviewed
  Status: Patch Available  (was: Reopened)

> Add readahead, dropbehind, and unbuffer to StreamCapabilities
> -
>
> Key: HADOOP-15012
> URL: https://issues.apache.org/jira/browse/HADOOP-15012
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs
>Affects Versions: 2.9.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Fix For: 3.1.0
>
> Attachments: HADOOP-15012.branch-2.01.patch
>
>
> A split from HADOOP-14872 to track changes that enhance StreamCapabilities 
> class with READAHEAD, DROPBEHIND, and UNBUFFER capability.
> Discussions and code reviews are done in HADOOP-14872.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15012) Add readahead, dropbehind, and unbuffer to StreamCapabilities

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen updated HADOOP-15012:
---
Attachment: HADOOP-15012.branch-2.01.patch

Reopening to attach a branch-2 patch

> Add readahead, dropbehind, and unbuffer to StreamCapabilities
> -
>
> Key: HADOOP-15012
> URL: https://issues.apache.org/jira/browse/HADOOP-15012
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs
>Affects Versions: 2.9.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Fix For: 3.1.0
>
> Attachments: HADOOP-15012.branch-2.01.patch
>
>
> A split from HADOOP-14872 to track changes that enhance StreamCapabilities 
> class with READAHEAD, DROPBEHIND, and UNBUFFER capability.
> Discussions and code reviews are done in HADOOP-14872.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-15012) Add readahead, dropbehind, and unbuffer to StreamCapabilities

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen reopened HADOOP-15012:


> Add readahead, dropbehind, and unbuffer to StreamCapabilities
> -
>
> Key: HADOOP-15012
> URL: https://issues.apache.org/jira/browse/HADOOP-15012
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs
>Affects Versions: 2.9.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Fix For: 3.1.0
>
> Attachments: HADOOP-15012.branch-2.01.patch
>
>
> A split from HADOOP-14872 to track changes that enhance StreamCapabilities 
> class with READAHEAD, DROPBEHIND, and UNBUFFER capability.
> Discussions and code reviews are done in HADOOP-14872.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281377#comment-16281377
 ] 

genericqa commented on HADOOP-15056:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
 2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m  
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 41s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
 2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m  
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m 55s{color} | {color:green} patch has no errors when building and testing our 
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m 
36s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m  2s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
35s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}181m 18s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
|   | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15056 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12900936/HADOOP-15056.005.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux b47b99329021 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 40b0045e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| 

[jira] [Updated] (HADOOP-15096) start-build-env.sh can create a docker image that fills up disk

[Addison Higham (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Addison Higham updated HADOOP-15096:

Description: 
start-build-env.sh has the potential to build an image that can fill up root 
disks by exploding a sparse file.

In my case, the right ingredients are:
Ubuntu 17.04
Docker 17.09.0
AUFS storage driver
userId and groupid with a high number

This happens when building the hadoop-build-${USER_ID} image, specifically in 
the 


{noformat}
RUN useradd -g ${GROUP_ID} -u ${USER_ID} -k /root -m ${USER_NAME}
{noformat}


command.

The reason for this:
/var/log/lastlog is a sparse file that pre-reserves based on highest seen UID 
and GID, in my case, those numbers are very high (above 1 billion). Locally, 
this result in a sparse file that reports as 443 GB. However, under docker and 
specifically AUFS, it appears that his file *isn't* sparse and it tries to 
allocate the whole file.

If you start this script and walk away to wait for it to finish, you come back 
to a computer with a completely full disk.

Luckily, the fix is quite easy, simply add the `-l` option to useradd which 
won't create those files

  was:
start-build-env.sh has the potential to build an image that can fill up root 
disks by exploding a sparse file.

In my case, the right ingredients are:
Ubuntu 17.04
Docker 17.09.0
AUFS storage driver
userId and groupid with a high number

This happens when building the hadoop-build-${USER_ID} image, specifically in 
the 

RUN useradd -g ${GROUP_ID} -u ${USER_ID} -k /root -m ${USER_NAME}

command.

The reason for this:
/var/log/lastlog is a sparse file that pre-reserves based on highest seen UID 
and GID, in my case, those numbers are very high (above 1 billion). Locally, 
this result in a sparse file that reports as 443 GB. However, under docker and 
specifically AUFS, it appears that his file *isn't* sparse and it tries to 
allocate the whole file.

If you start this script and walk away to wait for it to finish, you come back 
to a computer with a completely full disk.

Luckily, the fix is quite easy, simply add the `-l` option to useradd which 
won't create those files


> start-build-env.sh can create a docker image that fills up disk
> ---
>
> Key: HADOOP-15096
> URL: https://issues.apache.org/jira/browse/HADOOP-15096
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: build
>Affects Versions: 3.1.0
>Reporter: Addison Higham
>
> start-build-env.sh has the potential to build an image that can fill up root 
> disks by exploding a sparse file.
> In my case, the right ingredients are:
> Ubuntu 17.04
> Docker 17.09.0
> AUFS storage driver
> userId and groupid with a high number
> This happens when building the hadoop-build-${USER_ID} image, specifically in 
> the 
> {noformat}
> RUN useradd -g ${GROUP_ID} -u ${USER_ID} -k /root -m ${USER_NAME}
> {noformat}
> command.
> The reason for this:
> /var/log/lastlog is a sparse file that pre-reserves based on highest seen UID 
> and GID, in my case, those numbers are very high (above 1 billion). Locally, 
> this result in a sparse file that reports as 443 GB. However, under docker 
> and specifically AUFS, it appears that his file *isn't* sparse and it tries 
> to allocate the whole file.
> If you start this script and walk away to wait for it to finish, you come 
> back to a computer with a completely full disk.
> Luckily, the fix is quite easy, simply add the `-l` option to useradd which 
> won't create those files



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15096) start-build-env.sh can create a docker image that fills up disk

[Addison Higham (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Addison Higham updated HADOOP-15096:

Description: 
start-build-env.sh has the potential to build an image that can fill up root 
disks by exploding a sparse file.

In my case, the right ingredients are:
Ubuntu 17.04
Docker 17.09.0
AUFS storage driver
userId and groupid with a high number

This happens when building the hadoop-build-${USER_ID} image, specifically in 
the 

{code:none}
RUN useradd -g ${GROUP_ID} -u ${USER_ID} -k /root -m ${USER_NAME}
{code}

command.

The reason for this:
/var/log/lastlog is a sparse file that pre-reserves based on highest seen UID 
and GID, in my case, those numbers are very high (above 1 billion). Locally, 
this result in a sparse file that reports as 443 GB. However, under docker and 
specifically AUFS, it appears that his file *isn't* sparse and it tries to 
allocate the whole file.

If you start this script and walk away to wait for it to finish, you come back 
to a computer with a completely full disk.

Luckily, the fix is quite easy, simply add the `-l` option to useradd which 
won't create those files

  was:
start-build-env.sh has the potential to build an image that can fill up root 
disks by exploding a sparse file.

In my case, the right ingredients are:
Ubuntu 17.04
Docker 17.09.0
AUFS storage driver
userId and groupid with a high number

This happens when building the hadoop-build-${USER_ID} image, specifically in 
the 

{code:bash}
RUN useradd -g ${GROUP_ID} -u ${USER_ID} -k /root -m ${USER_NAME}
{code}

command.

The reason for this:
/var/log/lastlog is a sparse file that pre-reserves based on highest seen UID 
and GID, in my case, those numbers are very high (above 1 billion). Locally, 
this result in a sparse file that reports as 443 GB. However, under docker and 
specifically AUFS, it appears that his file *isn't* sparse and it tries to 
allocate the whole file.

If you start this script and walk away to wait for it to finish, you come back 
to a computer with a completely full disk.

Luckily, the fix is quite easy, simply add the `-l` option to useradd which 
won't create those files


> start-build-env.sh can create a docker image that fills up disk
> ---
>
> Key: HADOOP-15096
> URL: https://issues.apache.org/jira/browse/HADOOP-15096
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: build
>Affects Versions: 3.1.0
>Reporter: Addison Higham
>
> start-build-env.sh has the potential to build an image that can fill up root 
> disks by exploding a sparse file.
> In my case, the right ingredients are:
> Ubuntu 17.04
> Docker 17.09.0
> AUFS storage driver
> userId and groupid with a high number
> This happens when building the hadoop-build-${USER_ID} image, specifically in 
> the 
> {code:none}
> RUN useradd -g ${GROUP_ID} -u ${USER_ID} -k /root -m ${USER_NAME}
> {code}
> command.
> The reason for this:
> /var/log/lastlog is a sparse file that pre-reserves based on highest seen UID 
> and GID, in my case, those numbers are very high (above 1 billion). Locally, 
> this result in a sparse file that reports as 443 GB. However, under docker 
> and specifically AUFS, it appears that his file *isn't* sparse and it tries 
> to allocate the whole file.
> If you start this script and walk away to wait for it to finish, you come 
> back to a computer with a completely full disk.
> Luckily, the fix is quite easy, simply add the `-l` option to useradd which 
> won't create those files



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15096) start-build-env.sh can create a docker image that fills up disk

[Addison Higham (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Addison Higham updated HADOOP-15096:

Description: 
start-build-env.sh has the potential to build an image that can fill up root 
disks by exploding a sparse file.

In my case, the right ingredients are:
Ubuntu 17.04
Docker 17.09.0
AUFS storage driver
userId and groupid with a high number

This happens when building the hadoop-build-${USER_ID} image, specifically in 
the 

RUN useradd -g ${GROUP_ID} -u ${USER_ID} -k /root -m ${USER_NAME}

command.

The reason for this:
/var/log/lastlog is a sparse file that pre-reserves based on highest seen UID 
and GID, in my case, those numbers are very high (above 1 billion). Locally, 
this result in a sparse file that reports as 443 GB. However, under docker and 
specifically AUFS, it appears that his file *isn't* sparse and it tries to 
allocate the whole file.

If you start this script and walk away to wait for it to finish, you come back 
to a computer with a completely full disk.

Luckily, the fix is quite easy, simply add the `-l` option to useradd which 
won't create those files

  was:
start-build-env.sh has the potential to build an image that can fill up root 
disks by exploding a sparse file.

In my case, the right ingredients are:
Ubuntu 17.04
Docker 17.09.0
AUFS storage driver
userId and groupid with a high number

This happens when building the hadoop-build-${USER_ID} image, specifically in 
the 

{code:none}
RUN useradd -g ${GROUP_ID} -u ${USER_ID} -k /root -m ${USER_NAME}
{code}

command.

The reason for this:
/var/log/lastlog is a sparse file that pre-reserves based on highest seen UID 
and GID, in my case, those numbers are very high (above 1 billion). Locally, 
this result in a sparse file that reports as 443 GB. However, under docker and 
specifically AUFS, it appears that his file *isn't* sparse and it tries to 
allocate the whole file.

If you start this script and walk away to wait for it to finish, you come back 
to a computer with a completely full disk.

Luckily, the fix is quite easy, simply add the `-l` option to useradd which 
won't create those files


> start-build-env.sh can create a docker image that fills up disk
> ---
>
> Key: HADOOP-15096
> URL: https://issues.apache.org/jira/browse/HADOOP-15096
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: build
>Affects Versions: 3.1.0
>Reporter: Addison Higham
>
> start-build-env.sh has the potential to build an image that can fill up root 
> disks by exploding a sparse file.
> In my case, the right ingredients are:
> Ubuntu 17.04
> Docker 17.09.0
> AUFS storage driver
> userId and groupid with a high number
> This happens when building the hadoop-build-${USER_ID} image, specifically in 
> the 
> RUN useradd -g ${GROUP_ID} -u ${USER_ID} -k /root -m ${USER_NAME}
> command.
> The reason for this:
> /var/log/lastlog is a sparse file that pre-reserves based on highest seen UID 
> and GID, in my case, those numbers are very high (above 1 billion). Locally, 
> this result in a sparse file that reports as 443 GB. However, under docker 
> and specifically AUFS, it appears that his file *isn't* sparse and it tries 
> to allocate the whole file.
> If you start this script and walk away to wait for it to finish, you come 
> back to a computer with a completely full disk.
> Luckily, the fix is quite easy, simply add the `-l` option to useradd which 
> won't create those files



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15096) start-build-env.sh can create a docker image that fills up disk

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281372#comment-16281372
 ] 

ASF GitHub Bot commented on HADOOP-15096:
-

GitHub user addisonj opened a pull request:

https://github.com/apache/hadoop/pull/311

[HADOOP-15096] Don't create user with lastlog

This fixes a problem where in certain cases, the useradd command can create 
a very large diff that can blow up the host disk size.

The reason for this is that lastlog is a sparse file, but AUFS under docker 
apparently doesn't deal with those well and creates a very large file.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/addisonj/hadoop patch-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hadoop/pull/311.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #311


commit f5ffd36e78be66d9b075050f948d8af67cf53b56
Author: Addison Higham 
Date:   2017-12-07T06:07:27Z

[HADOOP-15096] Don't create user with lastlog

This fixes a problem where in certain cases, the useradd command can create 
a very large diff that can blow up the host disk size.

The reason for this is that lastlog is a sparse file, but AUFS under docker 
apparently doesn't deal with those well and creates a very large file.




> start-build-env.sh can create a docker image that fills up disk
> ---
>
> Key: HADOOP-15096
> URL: https://issues.apache.org/jira/browse/HADOOP-15096
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: build
>Affects Versions: 3.1.0
>Reporter: Addison Higham
>
> start-build-env.sh has the potential to build an image that can fill up root 
> disks by exploding a sparse file.
> In my case, the right ingredients are:
> Ubuntu 17.04
> Docker 17.09.0
> AUFS storage driver
> userId and groupid with a high number
> This happens when building the hadoop-build-${USER_ID} image, specifically in 
> the 
> {code}
> RUN useradd -g ${GROUP_ID} -u ${USER_ID} -k /root -m ${USER_NAME}
> {code}
> command.
> The reason for this:
> /var/log/lastlog is a sparse file that pre-reserves based on highest seen UID 
> and GID, in my case, those numbers are very high (above 1 billion). Locally, 
> this result in a sparse file that reports as 443 GB. However, under docker 
> and specifically AUFS, it appears that his file *isn't* sparse and it tries 
> to allocate the whole file.
> If you start this script and walk away to wait for it to finish, you come 
> back to a computer with a completely full disk.
> Luckily, the fix is quite easy, simply add the `-l` option to useradd which 
> won't create those files



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15096) start-build-env.sh can create a docker image that fills up disk

[Addison Higham (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Addison Higham updated HADOOP-15096:

Description: 
start-build-env.sh has the potential to build an image that can fill up root 
disks by exploding a sparse file.

In my case, the right ingredients are:
Ubuntu 17.04
Docker 17.09.0
AUFS storage driver
userId and groupid with a high number

This happens when building the hadoop-build-${USER_ID} image, specifically in 
the 

{code:bash}
RUN useradd -g ${GROUP_ID} -u ${USER_ID} -k /root -m ${USER_NAME}
{code}

command.

The reason for this:
/var/log/lastlog is a sparse file that pre-reserves based on highest seen UID 
and GID, in my case, those numbers are very high (above 1 billion). Locally, 
this result in a sparse file that reports as 443 GB. However, under docker and 
specifically AUFS, it appears that his file *isn't* sparse and it tries to 
allocate the whole file.

If you start this script and walk away to wait for it to finish, you come back 
to a computer with a completely full disk.

Luckily, the fix is quite easy, simply add the `-l` option to useradd which 
won't create those files

  was:
start-build-env.sh has the potential to build an image that can fill up root 
disks by exploding a sparse file.

In my case, the right ingredients are:
Ubuntu 17.04
Docker 17.09.0
AUFS storage driver
userId and groupid with a high number

This happens when building the hadoop-build-${USER_ID} image, specifically in 
the 

{code}
RUN useradd -g ${GROUP_ID} -u ${USER_ID} -k /root -m ${USER_NAME}
{code}

command.

The reason for this:
/var/log/lastlog is a sparse file that pre-reserves based on highest seen UID 
and GID, in my case, those numbers are very high (above 1 billion). Locally, 
this result in a sparse file that reports as 443 GB. However, under docker and 
specifically AUFS, it appears that his file *isn't* sparse and it tries to 
allocate the whole file.

If you start this script and walk away to wait for it to finish, you come back 
to a computer with a completely full disk.

Luckily, the fix is quite easy, simply add the `-l` option to useradd which 
won't create those files


> start-build-env.sh can create a docker image that fills up disk
> ---
>
> Key: HADOOP-15096
> URL: https://issues.apache.org/jira/browse/HADOOP-15096
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: build
>Affects Versions: 3.1.0
>Reporter: Addison Higham
>
> start-build-env.sh has the potential to build an image that can fill up root 
> disks by exploding a sparse file.
> In my case, the right ingredients are:
> Ubuntu 17.04
> Docker 17.09.0
> AUFS storage driver
> userId and groupid with a high number
> This happens when building the hadoop-build-${USER_ID} image, specifically in 
> the 
> {code:bash}
> RUN useradd -g ${GROUP_ID} -u ${USER_ID} -k /root -m ${USER_NAME}
> {code}
> command.
> The reason for this:
> /var/log/lastlog is a sparse file that pre-reserves based on highest seen UID 
> and GID, in my case, those numbers are very high (above 1 billion). Locally, 
> this result in a sparse file that reports as 443 GB. However, under docker 
> and specifically AUFS, it appears that his file *isn't* sparse and it tries 
> to allocate the whole file.
> If you start this script and walk away to wait for it to finish, you come 
> back to a computer with a completely full disk.
> Luckily, the fix is quite easy, simply add the `-l` option to useradd which 
> won't create those files



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15096) start-build-env.sh can create a docker image that fills up disk

[Addison Higham (JIRA)]

Addison Higham created HADOOP-15096:
---

 Summary: start-build-env.sh can create a docker image that fills 
up disk
 Key: HADOOP-15096
 URL: https://issues.apache.org/jira/browse/HADOOP-15096
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Affects Versions: 3.1.0
Reporter: Addison Higham


start-build-env.sh has the potential to build an image that can fill up root 
disks by exploding a sparse file.

In my case, the right ingredients are:
Ubuntu 17.04
Docker 17.09.0
AUFS storage driver
userId and groupid with a high number

This happens when building the hadoop-build-${USER_ID} image, specifically in 
the 

{code}
RUN useradd -g ${GROUP_ID} -u ${USER_ID} -k /root -m ${USER_NAME}
{code}

command.

The reason for this:
/var/log/lastlog is a sparse file that pre-reserves based on highest seen UID 
and GID, in my case, those numbers are very high (above 1 billion). Locally, 
this result in a sparse file that reports as 443 GB. However, under docker and 
specifically AUFS, it appears that his file *isn't* sparse and it tries to 
allocate the whole file.

If you start this script and walk away to wait for it to finish, you come back 
to a computer with a completely full disk.

Luckily, the fix is quite easy, simply add the `-l` option to useradd which 
won't create those files



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13126) Add Brotli compression codec

[Carlo Alberto Ferraris (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281358#comment-16281358
 ] 

Carlo Alberto Ferraris commented on HADOOP-13126:
-

[~ajisakaa] or [~steve_l] maybe?

> Add Brotli compression codec
> 
>
> Key: HADOOP-13126
> URL: https://issues.apache.org/jira/browse/HADOOP-13126
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: io
>Affects Versions: 2.7.2
>Reporter: Ryan Blue
>Assignee: Ryan Blue
> Attachments: HADOOP-13126.1.patch, HADOOP-13126.2.patch, 
> HADOOP-13126.3.patch, HADOOP-13126.4.patch, HADOOP-13126.5.patch
>
>
> I've been testing [Brotli|https://github.com/google/brotli/], a new 
> compression library based on LZ77 from Google. Google's [brotli 
> benchmarks|https://cran.r-project.org/web/packages/brotli/vignettes/brotli-2015-09-22.pdf]
>  look really good and we're also seeing a significant improvement in 
> compression size, compression speed, or both.
> {code:title=Brotli preliminary test results}
> [blue@work Downloads]$ time parquet from test.parquet -o test.snappy.parquet 
> --compression-codec snappy --overwrite  
> real1m17.106s
> user1m30.804s
> sys 0m4.404s
> [blue@work Downloads]$ time parquet from test.parquet -o test.br.parquet 
> --compression-codec brotli --overwrite 
> real1m16.640s
> user1m24.244s
> sys 0m6.412s
> [blue@work Downloads]$ time parquet from test.parquet -o test.gz.parquet 
> --compression-codec gzip --overwrite
> real3m39.496s
> user3m48.736s
> sys 0m3.880s
> [blue@work Downloads]$ ls -l
> -rw-r--r-- 1 blue blue 1068821936 May 10 11:06 test.br.parquet
> -rw-r--r-- 1 blue blue 1421601880 May 10 11:10 test.gz.parquet
> -rw-r--r-- 1 blue blue 2265950833 May 10 10:30 test.snappy.parquet
> {code}
> Brotli, at quality 1, is as fast as snappy and ends up smaller than gzip-9. 
> Another test resulted in a slightly larger Brotli file than gzip produced, 
> but Brotli was 4x faster. I'd like to get this compression codec into Hadoop.
> [Brotli is licensed with the MIT 
> license|https://github.com/google/brotli/blob/master/LICENSE], and the [JNI 
> library jbrotli is 
> ALv2|https://github.com/MeteoGroup/jbrotli/blob/master/LICENSE].



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Assigned] (HADOOP-9129) ViewFs does not validate internal names in the mount table

[Hanisha Koneru (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hanisha Koneru reassigned HADOOP-9129:
--

Assignee: Hanisha Koneru

> ViewFs does not validate internal names in the mount table
> --
>
> Key: HADOOP-9129
> URL: https://issues.apache.org/jira/browse/HADOOP-9129
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: viewfs
>Affects Versions: 3.0.0-alpha1
>Reporter: Chris Nauroth
>Assignee: Hanisha Koneru
>
> Currently, there is no explicit validation of {{ViewFs}} internal names in 
> the mount table during initialization.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances

[Xiao Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281331#comment-16281331
 ] 

Xiao Chen commented on HADOOP-14445:


Looking around this again, I don't think the conf override solution can solve 
the following problem:
For pure kms operations like {{hadoop key list}}, we rely on the kp conf to 
list from the correct kms. Even with this fix, we'd still need the conf. It's 
not reasonable to let client separate their configurations for different 
purpose of the hadoop rpc / apis. 

Checking the code, I think old clients / renewer should always work even with 
the new token. IMO we need to have a fall back logic to the configurations, 
rather than override. Specifically, when {new clients cannot find the new token 
format, new renewer cannot create kp from the token format}, they'll use the 
config as a last measure. Error handling is still tricky since we'd ideally 
want to tell from the exception or client log on what exactly was attempted and 
failed.

Other than the above I think this really should work.

Some minor comments:
- I'd mark {{DelegationTokenAuthenticatedURL#getDelegationTokenService}} as 
{{InterfaceAudience.Private}}.
- Add comments on the KMSCP override of the above method to explain why.
- I'm scared of {{SecurityUtil#useIpForTokenService}}, which is used to 
buildTokenService. I don't see any problems but would love to see a test 
covering it.
- Can we extract the unit-test-only {{KMSUtil}} code to a test-only method, and 
make it only possible to execute from unit test? Maybe via some injector or 
better approaches.


> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Rushabh S Shah
> Attachments: HADOOP-14445-branch-2.8.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do 
> not share delegation tokens. (a client uses KMS address/port as the key for 
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation 
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another 
> KMS instance, by checking the shared secret used to sign the delegation 
> token. To do this, all KMS instances must be able to retrieve the shared 
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share 
> delegation tokens.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15059) 3.0 deployment cannot work with old version MR tar ball which breaks rolling upgrade

[Ray Chiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281195#comment-16281195
 ] 

Ray Chiang commented on HADOOP-15059:
-

I pushed the 005 patch through the build system I have access to.  It's the 
same as the one that Vijaya Krishna Kalluru Subbarao did for the 3.0.0-beta1 
RC0 vote.  It's run through the build and tests cleanly (minus some flaky 
tests).

I'll see about running the tests tonight.

> 3.0 deployment cannot work with old version MR tar ball which breaks rolling 
> upgrade
> 
>
> Key: HADOOP-15059
> URL: https://issues.apache.org/jira/browse/HADOOP-15059
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Reporter: Junping Du
>Assignee: Jason Lowe
>Priority: Blocker
> Attachments: HADOOP-15059.001.patch, HADOOP-15059.002.patch, 
> HADOOP-15059.003.patch, HADOOP-15059.004.patch, HADOOP-15059.005.patch, 
> HADOOP-15059.006.patch
>
>
> I tried to deploy 3.0 cluster with 2.9 MR tar ball. The MR job is failed 
> because following error:
> {noformat}
> 2017-11-21 12:42:50,911 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for 
> application appattempt_1511295641738_0003_01
> 2017-11-21 12:42:51,070 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
> Unable to load native-hadoop library for your platform... using builtin-java 
> classes where applicable
> 2017-11-21 12:42:51,118 FATAL [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.lang.RuntimeException: Unable to determine current user
>   at 
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:254)
>   at 
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:220)
>   at 
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:212)
>   at 
> org.apache.hadoop.conf.Configuration.addResource(Configuration.java:888)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1638)
> Caused by: java.io.IOException: Exception reading 
> /tmp/nm-local-dir/usercache/jdu/appcache/application_1511295641738_0003/container_e03_1511295641738_0003_01_01/container_tokens
>   at 
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:208)
>   at 
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:907)
>   at 
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:820)
>   at 
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:689)
>   at 
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:252)
>   ... 4 more
> Caused by: java.io.IOException: Unknown version 1 in token storage.
>   at 
> org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:226)
>   at 
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:205)
>   ... 8 more
> 2017-11-21 12:42:51,122 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting 
> with status 1: java.lang.RuntimeException: Unable to determine current user
> {noformat}
> I think it is due to token incompatiblity change between 2.9 and 3.0. As we 
> claim "rolling upgrade" is supported in Hadoop 3, we should fix this before 
> we ship 3.0 otherwise all MR running applications will get stuck during/after 
> upgrade.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14959) DelegationTokenAuthenticator.authenticate() to wrap network exceptions

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281186#comment-16281186
 ] 

genericqa commented on HADOOP-14959:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m 
47s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m  
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 14s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
12s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m 
22s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 39s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch 
generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 25s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m 
19s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
35s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}109m 35s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14959 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12900968/HADOOP-14959.001.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 07d0a4a3be6d 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 40b0045e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/13797/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/13797/testReport/ |
| Max. process+thread count | 1361 (vs. ulimit of 5000) |
| modules | C: hadoop-common-project/hadoop-common U: 

[jira] [Commented] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281185#comment-16281185
 ] 

genericqa commented on HADOOP-15056:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m  
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 16s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m  
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m  
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m  
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
8m 39s{color} | {color:green} patch has no errors when building and testing our 
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m 
27s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}165m 47s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
45s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}269m 37s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestErasureCodingPolicies |
|   | hadoop.hdfs.TestDatanodeDeath |
|   | hadoop.hdfs.TestMaintenanceState |
|   | hadoop.hdfs.TestAppendSnapshotTruncate |
|   | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
|   | 
hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForContentSummary |
|   | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
|   | hadoop.hdfs.server.namenode.TestAuditLoggerWithCommands |
|   | hadoop.hdfs.server.namenode.TestFileTruncate |
|   | hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy |
|   | hadoop.hdfs.server.namenode.TestFsck |
|   | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040 |
|   | hadoop.hdfs.server.namenode.snapshot.TestSnapRootDescendantDiff |
|   | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
|   | 

[jira] [Commented] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure

[Xiao Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281172#comment-16281172
 ] 

Xiao Chen commented on HADOOP-15056:


Thanks for promptly revving [~jackbearden]! Looks good to me overall.

A few nits:
- This change looks unnecessary, let's not touch it here.
{code}
   if (in instanceof StreamCapabilities
   && ((StreamCapabilities) in).hasCapability(
-  StreamCapabilities.UNBUFFER)) {
+  StreamCapabilities.UNBUFFER)) {
{code}
- Sorry I wasn't clear. I was hoping the message of 
{{UnsupportedOperationException}} still contains inner stream's class name. The 
rest of the text looks great. 
- Can we define the message as a constant string variable in 
{{StreamCapabilitiesPolicy}}, and verify it in the test?

It seems pre-commit is still running at 
https://builds.apache.org/job/PreCommit-HADOOP-Build/13795, okay to address 
this after it's back.

> Fix TestUnbuffer#testUnbufferException failure
> --
>
> Key: HADOOP-15056
> URL: https://issues.apache.org/jira/browse/HADOOP-15056
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 2.9.0
>Reporter: Jack Bearden
>Assignee: Jack Bearden
>Priority: Minor
> Attachments: HADOOP-15056.001.patch, HADOOP-15056.002.patch, 
> HADOOP-15056.003.patch, HADOOP-15056.004.patch, HADOOP-15056.005.patch
>
>
> Hello! I am a new contributor and actually contributing to open source for 
> the very first time. :) 
> I pulled down Hadoop today and when running the tests I encountered a failure 
> with the TestUnbuffer#testUnbufferException test.
> The unbuffer code has recently gone through some changes and I believe this 
> test case may have been overlooked. Using today's git commit 
> (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, 
> there is an expect mock for an exception UnsupportedOperationException that 
> is no longer being thrown. 
> It would appear that a test like this would be valuable so my initial 
> proposed patch did not remove it. Instead, I removed the conditions that were 
> guarding the cast from being able to fire -- as was the previous behavior. 
> Now when we encounter an object that doesn't have the UNBUFFERED 
> StreamCapability, it will throw an error as it did prior to the recent 
> changes. 
> Please review and let me know what you think! :D



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-12897) KerberosAuthenticator.authenticate to include URL on IO failures

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-12897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281106#comment-16281106
 ] 

genericqa commented on HADOOP-12897:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m  
2s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 32m 
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
18m 45s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 21m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
21s{color} | {color:green} hadoop-common-project/hadoop-auth: The patch 
generated 0 new + 27 unchanged - 1 fixed = 27 total (was 28) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m 50s{color} | {color:green} patch has no errors when building and testing our 
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
54s{color} | {color:green} hadoop-auth in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
32s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}126m 37s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-12897 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12900950/HADOOP-12897.001.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 020ee44b02e9 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 
15:49:21 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 40b0045e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/13796/testReport/ |
| Max. process+thread count | 359 (vs. ulimit of 5000) |
| modules | C: hadoop-common-project/hadoop-auth U: 
hadoop-common-project/hadoop-auth |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/13796/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT   

[jira] [Comment Edited] (HADOOP-14914) Change to a safely casting long to int.

[Ajay Kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280842#comment-16280842
 ] 

Ajay Kumar edited comment on HADOOP-14914 at 12/6/17 11:09 PM:
---

[~yufeigu], thanks for kicking Jenkins run.
TestUnbuffer and TestContainerLaunch are failing without the patch as well. 
Other two passed locally.


was (Author: ajayydv):
[~yufeigu], thanks for submitting the patch.
TestUnbuffer and TestContainerLaunch are failing without the patch as well. 
Other two passed locally.

> Change to a safely casting long to int. 
> 
>
> Key: HADOOP-14914
> URL: https://issues.apache.org/jira/browse/HADOOP-14914
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>Assignee: Ajay Kumar
> Attachments: HADOOP-14914.001.patch, HADOOP-14914.002.patch
>
>
> There are bunches of casting long to int like this:
> {code}
> long l = 123
> int i = (int) l;
> {code}
> This is not a safe cast. if l is greater than Integer.MAX_VALUE, i would be 
> negative, which is an unexpected behavior. We probably at least want to throw 
> an exception in that case. I suggest to use {{Math.toIntExact(longValue)}} to 
> replace them, which throws an exception if the value overflows an int. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14959) DelegationTokenAuthenticator.authenticate() to wrap network exceptions

[Ajay Kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281063#comment-16281063
 ] 

Ajay Kumar commented on HADOOP-14959:
-

[~ste...@apache.org], Could you please have a look at attached patch.

> DelegationTokenAuthenticator.authenticate() to wrap network exceptions
> --
>
> Key: HADOOP-14959
> URL: https://issues.apache.org/jira/browse/HADOOP-14959
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: net, security
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Ajay Kumar
>Priority: Minor
> Attachments: HADOOP-14959.001.patch
>
>
> network errors raised in {{DelegationTokenAuthenticator.authenticate()}} 
> aren't being wrapped, so only return the usual limited-value java.net error 
> text. using {{NetUtils.wrapException()}} can address that



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14959) DelegationTokenAuthenticator.authenticate() to wrap network exceptions

[Ajay Kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ajay Kumar updated HADOOP-14959:

Status: Patch Available  (was: Open)

> DelegationTokenAuthenticator.authenticate() to wrap network exceptions
> --
>
> Key: HADOOP-14959
> URL: https://issues.apache.org/jira/browse/HADOOP-14959
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: net, security
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Ajay Kumar
>Priority: Minor
> Attachments: HADOOP-14959.001.patch
>
>
> network errors raised in {{DelegationTokenAuthenticator.authenticate()}} 
> aren't being wrapped, so only return the usual limited-value java.net error 
> text. using {{NetUtils.wrapException()}} can address that



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14959) DelegationTokenAuthenticator.authenticate() to wrap network exceptions

[Ajay Kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ajay Kumar updated HADOOP-14959:

Attachment: HADOOP-14959.001.patch

> DelegationTokenAuthenticator.authenticate() to wrap network exceptions
> --
>
> Key: HADOOP-14959
> URL: https://issues.apache.org/jira/browse/HADOOP-14959
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: net, security
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Ajay Kumar
>Priority: Minor
> Attachments: HADOOP-14959.001.patch
>
>
> network errors raised in {{DelegationTokenAuthenticator.authenticate()}} 
> aren't being wrapped, so only return the usual limited-value java.net error 
> text. using {{NetUtils.wrapException()}} can address that



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Assigned] (HADOOP-14959) DelegationTokenAuthenticator.authenticate() to wrap network exceptions

[Ajay Kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ajay Kumar reassigned HADOOP-14959:
---

Assignee: Ajay Kumar

> DelegationTokenAuthenticator.authenticate() to wrap network exceptions
> --
>
> Key: HADOOP-14959
> URL: https://issues.apache.org/jira/browse/HADOOP-14959
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: net, security
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Ajay Kumar
>Priority: Minor
>
> network errors raised in {{DelegationTokenAuthenticator.authenticate()}} 
> aren't being wrapped, so only return the usual limited-value java.net error 
> text. using {{NetUtils.wrapException()}} can address that



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-12897) KerberosAuthenticator.authenticate to include URL on IO failures

[Ajay Kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ajay Kumar updated HADOOP-12897:

Status: Patch Available  (was: Open)

> KerberosAuthenticator.authenticate to include URL on IO failures
> 
>
> Key: HADOOP-12897
> URL: https://issues.apache.org/jira/browse/HADOOP-12897
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-12897.001.patch
>
>
> If {{KerberosAuthenticator.authenticate}} can't connect to the endpoint, you 
> get a stack trace, but without the URL it is trying to talk to.
> That is: it doesn't have any equivalent of the {{NetUtils.wrapException}} 
> handler —which can't be called here as its not in the {{hadoop-auth}} module



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-12897) KerberosAuthenticator.authenticate to include URL on IO failures

[Ajay Kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-12897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ajay Kumar updated HADOOP-12897:

Attachment: HADOOP-12897.001.patch

[~yuanbo],[~steve_l], Adding patch for review. Let me know if moving 
{{KerberosAuthenticator.wrapExceptionWithMessage}} to a separate util class 
under {{hadoop-auth}} make more sense.

> KerberosAuthenticator.authenticate to include URL on IO failures
> 
>
> Key: HADOOP-12897
> URL: https://issues.apache.org/jira/browse/HADOOP-12897
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-12897.001.patch
>
>
> If {{KerberosAuthenticator.authenticate}} can't connect to the endpoint, you 
> get a stack trace, but without the URL it is trying to talk to.
> That is: it doesn't have any equivalent of the {{NetUtils.wrapException}} 
> handler —which can't be called here as its not in the {{hadoop-auth}} module



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15059) 3.0 deployment cannot work with old version MR tar ball which breaks rolling upgrade

[Vinod Kumar Vavilapalli (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280875#comment-16280875
 ] 

Vinod Kumar Vavilapalli commented on HADOOP-15059:
--

That makes sense. Then I can commit the previous patch. [~djp], can you give 
005 patch a spin? 

> 3.0 deployment cannot work with old version MR tar ball which breaks rolling 
> upgrade
> 
>
> Key: HADOOP-15059
> URL: https://issues.apache.org/jira/browse/HADOOP-15059
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Reporter: Junping Du
>Assignee: Jason Lowe
>Priority: Blocker
> Attachments: HADOOP-15059.001.patch, HADOOP-15059.002.patch, 
> HADOOP-15059.003.patch, HADOOP-15059.004.patch, HADOOP-15059.005.patch, 
> HADOOP-15059.006.patch
>
>
> I tried to deploy 3.0 cluster with 2.9 MR tar ball. The MR job is failed 
> because following error:
> {noformat}
> 2017-11-21 12:42:50,911 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for 
> application appattempt_1511295641738_0003_01
> 2017-11-21 12:42:51,070 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
> Unable to load native-hadoop library for your platform... using builtin-java 
> classes where applicable
> 2017-11-21 12:42:51,118 FATAL [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.lang.RuntimeException: Unable to determine current user
>   at 
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:254)
>   at 
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:220)
>   at 
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:212)
>   at 
> org.apache.hadoop.conf.Configuration.addResource(Configuration.java:888)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1638)
> Caused by: java.io.IOException: Exception reading 
> /tmp/nm-local-dir/usercache/jdu/appcache/application_1511295641738_0003/container_e03_1511295641738_0003_01_01/container_tokens
>   at 
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:208)
>   at 
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:907)
>   at 
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:820)
>   at 
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:689)
>   at 
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:252)
>   ... 4 more
> Caused by: java.io.IOException: Unknown version 1 in token storage.
>   at 
> org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:226)
>   at 
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:205)
>   ... 8 more
> 2017-11-21 12:42:51,122 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting 
> with status 1: java.lang.RuntimeException: Unable to determine current user
> {noformat}
> I think it is due to token incompatiblity change between 2.9 and 3.0. As we 
> claim "rolling upgrade" is supported in Hadoop 3, we should fix this before 
> we ship 3.0 otherwise all MR running applications will get stuck during/after 
> upgrade.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15059) 3.0 deployment cannot work with old version MR tar ball which breaks rolling upgrade

[Ray Chiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ray Chiang updated HADOOP-15059:

Summary: 3.0 deployment cannot work with old version MR tar ball which 
breaks rolling upgrade  (was: 3.0 deployment cannot work with old version MR 
tar ball which break rolling upgrade)

> 3.0 deployment cannot work with old version MR tar ball which breaks rolling 
> upgrade
> 
>
> Key: HADOOP-15059
> URL: https://issues.apache.org/jira/browse/HADOOP-15059
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Reporter: Junping Du
>Assignee: Jason Lowe
>Priority: Blocker
> Attachments: HADOOP-15059.001.patch, HADOOP-15059.002.patch, 
> HADOOP-15059.003.patch, HADOOP-15059.004.patch, HADOOP-15059.005.patch, 
> HADOOP-15059.006.patch
>
>
> I tried to deploy 3.0 cluster with 2.9 MR tar ball. The MR job is failed 
> because following error:
> {noformat}
> 2017-11-21 12:42:50,911 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for 
> application appattempt_1511295641738_0003_01
> 2017-11-21 12:42:51,070 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
> Unable to load native-hadoop library for your platform... using builtin-java 
> classes where applicable
> 2017-11-21 12:42:51,118 FATAL [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.lang.RuntimeException: Unable to determine current user
>   at 
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:254)
>   at 
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:220)
>   at 
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:212)
>   at 
> org.apache.hadoop.conf.Configuration.addResource(Configuration.java:888)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1638)
> Caused by: java.io.IOException: Exception reading 
> /tmp/nm-local-dir/usercache/jdu/appcache/application_1511295641738_0003/container_e03_1511295641738_0003_01_01/container_tokens
>   at 
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:208)
>   at 
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:907)
>   at 
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:820)
>   at 
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:689)
>   at 
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:252)
>   ... 4 more
> Caused by: java.io.IOException: Unknown version 1 in token storage.
>   at 
> org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:226)
>   at 
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:205)
>   ... 8 more
> 2017-11-21 12:42:51,122 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting 
> with status 1: java.lang.RuntimeException: Unable to determine current user
> {noformat}
> I think it is due to token incompatiblity change between 2.9 and 3.0. As we 
> claim "rolling upgrade" is supported in Hadoop 3, we should fix this before 
> we ship 3.0 otherwise all MR running applications will get stuck during/after 
> upgrade.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14914) Change to a safely casting long to int.

[Ajay Kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280842#comment-16280842
 ] 

Ajay Kumar commented on HADOOP-14914:
-

[~yufeigu], thanks for submitting the patch.
TestUnbuffer and TestContainerLaunch are failing without the patch as well. 
Other two passed locally.

> Change to a safely casting long to int. 
> 
>
> Key: HADOOP-14914
> URL: https://issues.apache.org/jira/browse/HADOOP-14914
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>Assignee: Ajay Kumar
> Attachments: HADOOP-14914.001.patch, HADOOP-14914.002.patch
>
>
> There are bunches of casting long to int like this:
> {code}
> long l = 123
> int i = (int) l;
> {code}
> This is not a safe cast. if l is greater than Integer.MAX_VALUE, i would be 
> negative, which is an unexpected behavior. We probably at least want to throw 
> an exception in that case. I suggest to use {{Math.toIntExact(longValue)}} to 
> replace them, which throws an exception if the value overflows an int. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15059) 3.0 deployment cannot work with old version MR tar ball which break rolling upgrade

[Jason Lowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280843#comment-16280843
 ] 

Jason Lowe commented on HADOOP-15059:
-

Sorry for the delay in replying, been busy/traveling.

Inner enums are already static, so declaring them static is redundant.  I see 
checkstyle reported the same.

I also haven't manually tested the rolling upgrade case Junping originall 
reported with this latest patch.  It _should_ solve the problem, but in the 
interest of full disclosure I have not had time to test that scenario.


> 3.0 deployment cannot work with old version MR tar ball which break rolling 
> upgrade
> ---
>
> Key: HADOOP-15059
> URL: https://issues.apache.org/jira/browse/HADOOP-15059
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Reporter: Junping Du
>Assignee: Jason Lowe
>Priority: Blocker
> Attachments: HADOOP-15059.001.patch, HADOOP-15059.002.patch, 
> HADOOP-15059.003.patch, HADOOP-15059.004.patch, HADOOP-15059.005.patch, 
> HADOOP-15059.006.patch
>
>
> I tried to deploy 3.0 cluster with 2.9 MR tar ball. The MR job is failed 
> because following error:
> {noformat}
> 2017-11-21 12:42:50,911 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for 
> application appattempt_1511295641738_0003_01
> 2017-11-21 12:42:51,070 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
> Unable to load native-hadoop library for your platform... using builtin-java 
> classes where applicable
> 2017-11-21 12:42:51,118 FATAL [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.lang.RuntimeException: Unable to determine current user
>   at 
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:254)
>   at 
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:220)
>   at 
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:212)
>   at 
> org.apache.hadoop.conf.Configuration.addResource(Configuration.java:888)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1638)
> Caused by: java.io.IOException: Exception reading 
> /tmp/nm-local-dir/usercache/jdu/appcache/application_1511295641738_0003/container_e03_1511295641738_0003_01_01/container_tokens
>   at 
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:208)
>   at 
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:907)
>   at 
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:820)
>   at 
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:689)
>   at 
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:252)
>   ... 4 more
> Caused by: java.io.IOException: Unknown version 1 in token storage.
>   at 
> org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:226)
>   at 
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:205)
>   ... 8 more
> 2017-11-21 12:42:51,122 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting 
> with status 1: java.lang.RuntimeException: Unable to determine current user
> {noformat}
> I think it is due to token incompatiblity change between 2.9 and 3.0. As we 
> claim "rolling upgrade" is supported in Hadoop 3, we should fix this before 
> we ship 3.0 otherwise all MR running applications will get stuck during/after 
> upgrade.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15059) 3.0 deployment cannot work with old version MR tar ball which break rolling upgrade

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280826#comment-16280826
 ] 

genericqa commented on HADOOP-15059:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
 1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m  
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 44s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m 
12s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
2m  1s{color} | {color:orange} root: The patch generated 3 new + 20 unchanged - 
4 fixed = 23 total (was 24) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m  3s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m 
37s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 58s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
35s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}180m 14s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
|   | hadoop.fs.TestUnbuffer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15059 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12900897/HADOOP-15059.006.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 23335f39401a 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 40b0045e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| 

[jira] [Commented] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure

[Jack Bearden (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280817#comment-16280817
 ] 

Jack Bearden commented on HADOOP-15056:
---

Really great feedback, thanks for reviewing the code [~xiaochen]. I really 
appreciate that you took the time to give me specific feedback on how I could 
improve. My responses are below:

{quote}Can we modify the test to something like this? [code removed]{quote}
That test snippet you provided was actually really helpful and I implemented it 
in patch #5 as best I could.

{quote}About the message, while it good to have this information when a stream 
doesn't support unbuffer, WARN may be intrusive for some cases. For example, I 
think it's not unreasonable for an application to write some general codes to 
unbuffer regardless, but after this they'll see WARN. It's true this is the 
only log for that logger, but they still will catch this and have to change the 
log level. DEBUG may be a better option, where people can opt-in.{quote}
You are totally right. I was thinking WARN was it's own log level for some 
reason -- DEBUG is the appropriate call here. I have modified the log statement 
in patch #5.

{quote}Can we change the text of the UnsupportedOperationException? Maybe 
something like claims to have unbuffer capabilty but does not to implement 
CanUnbuffer ?{quote}
I have updated the error message.


 [~jzhuge] [~xiaochen]  Please review and let me know if anything was missed. 
Thank you again

> Fix TestUnbuffer#testUnbufferException failure
> --
>
> Key: HADOOP-15056
> URL: https://issues.apache.org/jira/browse/HADOOP-15056
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 2.9.0
>Reporter: Jack Bearden
>Assignee: Jack Bearden
>Priority: Minor
> Attachments: HADOOP-15056.001.patch, HADOOP-15056.002.patch, 
> HADOOP-15056.003.patch, HADOOP-15056.004.patch, HADOOP-15056.005.patch
>
>
> Hello! I am a new contributor and actually contributing to open source for 
> the very first time. :) 
> I pulled down Hadoop today and when running the tests I encountered a failure 
> with the TestUnbuffer#testUnbufferException test.
> The unbuffer code has recently gone through some changes and I believe this 
> test case may have been overlooked. Using today's git commit 
> (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, 
> there is an expect mock for an exception UnsupportedOperationException that 
> is no longer being thrown. 
> It would appear that a test like this would be valuable so my initial 
> proposed patch did not remove it. Instead, I removed the conditions that were 
> guarding the cast from being able to fire -- as was the previous behavior. 
> Now when we encounter an object that doesn't have the UNBUFFERED 
> StreamCapability, it will throw an error as it did prior to the recent 
> changes. 
> Please review and let me know what you think! :D



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure

[Jack Bearden (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jack Bearden updated HADOOP-15056:
--
Attachment: HADOOP-15056.005.patch

> Fix TestUnbuffer#testUnbufferException failure
> --
>
> Key: HADOOP-15056
> URL: https://issues.apache.org/jira/browse/HADOOP-15056
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 2.9.0
>Reporter: Jack Bearden
>Assignee: Jack Bearden
>Priority: Minor
> Attachments: HADOOP-15056.001.patch, HADOOP-15056.002.patch, 
> HADOOP-15056.003.patch, HADOOP-15056.004.patch, HADOOP-15056.005.patch
>
>
> Hello! I am a new contributor and actually contributing to open source for 
> the very first time. :) 
> I pulled down Hadoop today and when running the tests I encountered a failure 
> with the TestUnbuffer#testUnbufferException test.
> The unbuffer code has recently gone through some changes and I believe this 
> test case may have been overlooked. Using today's git commit 
> (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, 
> there is an expect mock for an exception UnsupportedOperationException that 
> is no longer being thrown. 
> It would appear that a test like this would be valuable so my initial 
> proposed patch did not remove it. Instead, I removed the conditions that were 
> guarding the cast from being able to fire -- as was the previous behavior. 
> Now when we encounter an object that doesn't have the UNBUFFERED 
> StreamCapability, it will throw an error as it did prior to the recent 
> changes. 
> Please review and let me know what you think! :D



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15080) Cat-X dependency on org.json via derived json-lib

[Sean Mackrory (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280732#comment-16280732
 ] 

Sean Mackrory commented on HADOOP-15080:


{quote}I agree. It's like saying "we need gcc to build the native code on 
linux". We do, we just don't redistribute it.{quote}

I'm still uncomfortable with the comparison to gcc, which is what I don't think 
is being understood on LEGAL-349: they're both category X licenses, but the GPL 
restricts distribution, and json-lib is not approved because it restricts 
usage. I feel like if this library was GPL it would actually be fair game for 
use as a test dependency. But it's academic now - it sounds like we're all in 
agreement on the right paths to pursue both immediately and long-term.

> Cat-X dependency on org.json via derived json-lib
> -
>
> Key: HADOOP-15080
> URL: https://issues.apache.org/jira/browse/HADOOP-15080
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/oss
>Affects Versions: 3.0.0-beta1
>Reporter: Chris Douglas
>Priority: Blocker
> Attachments: HADOOP-15080-branch-3.0.0.001.patch
>
>
> The OSS SDK has a dependency on json-lib. In LEGAL-245, the org.json library 
> (from which json-lib may be derived) is released under a 
> [category-x|https://www.apache.org/legal/resolved.html#json] license.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15080) Cat-X dependency on org.json via derived json-lib

[Steve Loughran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280724#comment-16280724
 ] 

Steve Loughran commented on HADOOP-15080:
-

bq.  if Hadoop itself can be developed, built and tested without json-lib being 
on the machine, it's a very big stretch to argue that the ASF is "using" it by 
any definition

I agree. It's like saying "we need gcc to build the native code on linux". We 
do, we just don't redistribute it.

Interesting point about that explicit references though. Doing a grep for gcc 
in our source, while we don't call it out as the required compiler, we 
certainly have explicit support for it being used throughout the native libs.




> Cat-X dependency on org.json via derived json-lib
> -
>
> Key: HADOOP-15080
> URL: https://issues.apache.org/jira/browse/HADOOP-15080
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/oss
>Affects Versions: 3.0.0-beta1
>Reporter: Chris Douglas
>Priority: Blocker
> Attachments: HADOOP-15080-branch-3.0.0.001.patch
>
>
> The OSS SDK has a dependency on json-lib. In LEGAL-245, the org.json library 
> (from which json-lib may be derived) is released under a 
> [category-x|https://www.apache.org/legal/resolved.html#json] license.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14403) Should document HttpFileSystem which reads from HTTP/HTTPS

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280632#comment-16280632
 ] 

genericqa commented on HADOOP-14403:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
28m 26s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 42s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m  4s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14403 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12867177/HADOOP-14403.001.patch
 |
| Optional Tests |  asflicense  mvnsite  |
| uname | Linux d79142347347 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 40b0045e |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 326 (vs. ulimit of 5000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/13794/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Should document HttpFileSystem which reads from HTTP/HTTPS
> --
>
> Key: HADOOP-14403
> URL: https://issues.apache.org/jira/browse/HADOOP-14403
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HADOOP-14403.001.patch
>
>
> HADOOP-14383 adds {{HttpFileSystem}} which allows HDFS to read from a 
> http/https url. This is a very useful feature and should documented to ease 
> the usage. This JIRA tries to add a .md file to the doc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14445) Delegation tokens are not shared between KMS instances

[Wei-Chiu Chuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-14445:
-
Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption

> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Rushabh S Shah
> Attachments: HADOOP-14445-branch-2.8.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do 
> not share delegation tokens. (a client uses KMS address/port as the key for 
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation 
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another 
> KMS instance, by checking the shared secret used to sign the delegation 
> token. To do this, all KMS instances must be able to retrieve the shared 
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share 
> delegation tokens.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14441) LoadBalancingKMSClientProvider#addDelegationTokens should add delegation tokens from all KMS instances

[Wei-Chiu Chuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-14441:
-
Resolution: Duplicate
Status: Resolved  (was: Patch Available)

This jira is not moving forward and is a duplicate of HADOOP-14445. Let's 
concentrate the discussion at HADOOP-14445 instead.

> LoadBalancingKMSClientProvider#addDelegationTokens should add delegation 
> tokens from all KMS instances
> --
>
> Key: HADOOP-14441
> URL: https://issues.apache.org/jira/browse/HADOOP-14441
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.7.0
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14441.001.patch, HADOOP-14441.002.patch, 
> HADOOP-14441.003.patch, HADOOP-14441.004.patch
>
>
> LoadBalancingKMSClientProvider only gets delegation token from one KMS 
> instance, in a round-robin fashion. This is arguably a bug, as JavaDoc for 
> {{KeyProviderDelegationTokenExtension#addDelegationTokens}} states:
> {quote}
> /**
>  * The implementer of this class will take a renewer and add all
>  * delegation tokens associated with the renewer to the 
>  * Credentials object if it is not already present, 
> ...
> **/
> {quote}
> This bug doesn't pop up very often, because HDFS clients such as MapReduce 
> unintentionally calls {{FileSystem#addDelegationTokens}} multiple times.
> We have a custom client that accesses HDFS/KMS-HA using delegation token, and 
> we were puzzled why it always throws "Failed to find any Kerberos tgt" 
> exceptions talking to one KMS but not the other. Turns out that client 
> couldn't talk to the KMS because {{FileSystem#addDelegationTokens}} only gets 
> one KMS delegation token at a time.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14445) Delegation tokens are not shared between KMS instances

[Wei-Chiu Chuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-14445:
-
Component/s: (was: documentation)

> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Rushabh S Shah
> Attachments: HADOOP-14445-branch-2.8.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do 
> not share delegation tokens. (a client uses KMS address/port as the key for 
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation 
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another 
> KMS instance, by checking the shared secret used to sign the delegation 
> token. To do this, all KMS instances must be able to retrieve the shared 
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share 
> delegation tokens.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-10768) Optimize Hadoop RPC encryption performance

[Daryn Sharp (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-10768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280567#comment-16280567
 ] 

Daryn Sharp commented on HADOOP-10768:
--

I dug around a bit and learned you're right: JCE performance is still horrible, 
but improving in part due to your contributions.

bq. performance of AES-GCM(openssl) ~= AES-CTR + MD5
I respectfully refuse to believe this patch's native CTR with a serial HMAC 
computation in java (ie. {{Integrity#calculateHMAC}}) is even remotely 
comparably to a native GCM auth tag computation.  That method does so many 
allocations and array copies that cpu and gc overhead must be noticeably 
elevated?

> Optimize Hadoop RPC encryption performance
> --
>
> Key: HADOOP-10768
> URL: https://issues.apache.org/jira/browse/HADOOP-10768
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: performance, security
>Affects Versions: 3.0.0-alpha1
>Reporter: Yi Liu
>Assignee: Dapeng Sun
> Attachments: HADOOP-10768.001.patch, HADOOP-10768.002.patch, 
> HADOOP-10768.003.patch, HADOOP-10768.004.patch, HADOOP-10768.005.patch, 
> HADOOP-10768.006.patch, HADOOP-10768.007.patch, HADOOP-10768.008.patch, 
> Optimize Hadoop RPC encryption performance.pdf
>
>
> Hadoop RPC encryption is enabled by setting {{hadoop.rpc.protection}} to 
> "privacy". It utilized SASL {{GSSAPI}} and {{DIGEST-MD5}} mechanisms for 
> secure authentication and data protection. Even {{GSSAPI}} supports using 
> AES, but without AES-NI support by default, so the encryption is slow and 
> will become bottleneck.
> After discuss with [~atm], [~tucu00] and [~umamaheswararao], we can do the 
> same optimization as in HDFS-6606. Use AES-NI with more than *20x* speedup.
> On the other hand, RPC message is small, but RPC is frequent and there may be 
> lots of RPC calls in one connection, we needs to setup benchmark to see real 
> improvement and then make a trade-off. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14403) Should document HttpFileSystem which reads from HTTP/HTTPS

[Ajay Kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280560#comment-16280560
 ] 

Ajay Kumar commented on HADOOP-14403:
-

Hi [~vagarychen], Thanks for working on this. Patch looks good to me. Some 
minor comments.
L25 "the user only needs to {{provid}} the http/https url either 
programmatically"
L25 "Following are two simple {{exampels}}"
This may be my personal view and other may not agree but i think we can remove 
the last line. ({{"Then the code may proceed reading and processing the content 
of the file."}}) You have already provided code snipped to read from an URL, we 
can leave it to users what they want to do afterwards. 

> Should document HttpFileSystem which reads from HTTP/HTTPS
> --
>
> Key: HADOOP-14403
> URL: https://issues.apache.org/jira/browse/HADOOP-14403
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HADOOP-14403.001.patch
>
>
> HADOOP-14383 adds {{HttpFileSystem}} which allows HDFS to read from a 
> http/https url. This is a very useful feature and should documented to ease 
> the usage. This JIRA tries to add a .md file to the doc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15092) Proxy failures during NamenodeWebHdfsMethods are not logged

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280532#comment-16280532
 ] 

genericqa commented on HADOOP-15092:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 35s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m  0s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  9m 
33s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
32s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 39s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15092 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12900888/HADOOP-15092.2.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 797b8bebdf9b 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 40b0045e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/13792/testReport/ |
| Max. process+thread count | 1403 (vs. ulimit of 5000) |
| modules | C: hadoop-common-project/hadoop-common U: 
hadoop-common-project/hadoop-common |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/13792/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Proxy failures during NamenodeWebHdfsMethods are not logged 
> 
>
> Key: 

[jira] [Updated] (HADOOP-15059) 3.0 deployment cannot work with old version MR tar ball which break rolling upgrade

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli updated HADOOP-15059:
-
Attachment: HADOOP-15059.006.patch

The patch looks good to me.

[~daryn], don't get scared by exception handling!

Uploading a new patch that is same as that of Jason's but with SerializedFormat 
made static.

The test case failures reported here are unrelated. Though it is surprising how 
badly the unit tests are broken - I'll debug them independently and file bugs.

If Jenkins says okay, I'll commit this unless [~daryn] / [~jlowe] say no.

> 3.0 deployment cannot work with old version MR tar ball which break rolling 
> upgrade
> ---
>
> Key: HADOOP-15059
> URL: https://issues.apache.org/jira/browse/HADOOP-15059
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Reporter: Junping Du
>Assignee: Jason Lowe
>Priority: Blocker
> Attachments: HADOOP-15059.001.patch, HADOOP-15059.002.patch, 
> HADOOP-15059.003.patch, HADOOP-15059.004.patch, HADOOP-15059.005.patch, 
> HADOOP-15059.006.patch
>
>
> I tried to deploy 3.0 cluster with 2.9 MR tar ball. The MR job is failed 
> because following error:
> {noformat}
> 2017-11-21 12:42:50,911 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for 
> application appattempt_1511295641738_0003_01
> 2017-11-21 12:42:51,070 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
> Unable to load native-hadoop library for your platform... using builtin-java 
> classes where applicable
> 2017-11-21 12:42:51,118 FATAL [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.lang.RuntimeException: Unable to determine current user
>   at 
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:254)
>   at 
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:220)
>   at 
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:212)
>   at 
> org.apache.hadoop.conf.Configuration.addResource(Configuration.java:888)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1638)
> Caused by: java.io.IOException: Exception reading 
> /tmp/nm-local-dir/usercache/jdu/appcache/application_1511295641738_0003/container_e03_1511295641738_0003_01_01/container_tokens
>   at 
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:208)
>   at 
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:907)
>   at 
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:820)
>   at 
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:689)
>   at 
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:252)
>   ... 4 more
> Caused by: java.io.IOException: Unknown version 1 in token storage.
>   at 
> org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:226)
>   at 
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:205)
>   ... 8 more
> 2017-11-21 12:42:51,122 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting 
> with status 1: java.lang.RuntimeException: Unable to determine current user
> {noformat}
> I think it is due to token incompatiblity change between 2.9 and 3.0. As we 
> claim "rolling upgrade" is supported in Hadoop 3, we should fix this before 
> we ship 3.0 otherwise all MR running applications will get stuck during/after 
> upgrade.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15059) 3.0 deployment cannot work with old version MR tar ball which break rolling upgrade

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli updated HADOOP-15059:
-
Hadoop Flags: Reviewed
  Status: Patch Available  (was: Open)

> 3.0 deployment cannot work with old version MR tar ball which break rolling 
> upgrade
> ---
>
> Key: HADOOP-15059
> URL: https://issues.apache.org/jira/browse/HADOOP-15059
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Reporter: Junping Du
>Assignee: Jason Lowe
>Priority: Blocker
> Attachments: HADOOP-15059.001.patch, HADOOP-15059.002.patch, 
> HADOOP-15059.003.patch, HADOOP-15059.004.patch, HADOOP-15059.005.patch, 
> HADOOP-15059.006.patch
>
>
> I tried to deploy 3.0 cluster with 2.9 MR tar ball. The MR job is failed 
> because following error:
> {noformat}
> 2017-11-21 12:42:50,911 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for 
> application appattempt_1511295641738_0003_01
> 2017-11-21 12:42:51,070 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
> Unable to load native-hadoop library for your platform... using builtin-java 
> classes where applicable
> 2017-11-21 12:42:51,118 FATAL [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.lang.RuntimeException: Unable to determine current user
>   at 
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:254)
>   at 
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:220)
>   at 
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:212)
>   at 
> org.apache.hadoop.conf.Configuration.addResource(Configuration.java:888)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1638)
> Caused by: java.io.IOException: Exception reading 
> /tmp/nm-local-dir/usercache/jdu/appcache/application_1511295641738_0003/container_e03_1511295641738_0003_01_01/container_tokens
>   at 
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:208)
>   at 
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:907)
>   at 
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:820)
>   at 
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:689)
>   at 
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:252)
>   ... 4 more
> Caused by: java.io.IOException: Unknown version 1 in token storage.
>   at 
> org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:226)
>   at 
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:205)
>   ... 8 more
> 2017-11-21 12:42:51,122 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting 
> with status 1: java.lang.RuntimeException: Unable to determine current user
> {noformat}
> I think it is due to token incompatiblity change between 2.9 and 3.0. As we 
> claim "rolling upgrade" is supported in Hadoop 3, we should fix this before 
> we ship 3.0 otherwise all MR running applications will get stuck during/after 
> upgrade.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15059) 3.0 deployment cannot work with old version MR tar ball which break rolling upgrade

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli updated HADOOP-15059:
-
Status: Open  (was: Patch Available)

> 3.0 deployment cannot work with old version MR tar ball which break rolling 
> upgrade
> ---
>
> Key: HADOOP-15059
> URL: https://issues.apache.org/jira/browse/HADOOP-15059
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Reporter: Junping Du
>Assignee: Jason Lowe
>Priority: Blocker
> Attachments: HADOOP-15059.001.patch, HADOOP-15059.002.patch, 
> HADOOP-15059.003.patch, HADOOP-15059.004.patch, HADOOP-15059.005.patch
>
>
> I tried to deploy 3.0 cluster with 2.9 MR tar ball. The MR job is failed 
> because following error:
> {noformat}
> 2017-11-21 12:42:50,911 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for 
> application appattempt_1511295641738_0003_01
> 2017-11-21 12:42:51,070 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
> Unable to load native-hadoop library for your platform... using builtin-java 
> classes where applicable
> 2017-11-21 12:42:51,118 FATAL [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.lang.RuntimeException: Unable to determine current user
>   at 
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:254)
>   at 
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:220)
>   at 
> org.apache.hadoop.conf.Configuration$Resource.(Configuration.java:212)
>   at 
> org.apache.hadoop.conf.Configuration.addResource(Configuration.java:888)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1638)
> Caused by: java.io.IOException: Exception reading 
> /tmp/nm-local-dir/usercache/jdu/appcache/application_1511295641738_0003/container_e03_1511295641738_0003_01_01/container_tokens
>   at 
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:208)
>   at 
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:907)
>   at 
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:820)
>   at 
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:689)
>   at 
> org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:252)
>   ... 4 more
> Caused by: java.io.IOException: Unknown version 1 in token storage.
>   at 
> org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:226)
>   at 
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:205)
>   ... 8 more
> 2017-11-21 12:42:51,122 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting 
> with status 1: java.lang.RuntimeException: Unable to determine current user
> {noformat}
> I think it is due to token incompatiblity change between 2.9 and 3.0. As we 
> claim "rolling upgrade" is supported in Hadoop 3, we should fix this before 
> we ship 3.0 otherwise all MR running applications will get stuck during/after 
> upgrade.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13126) Add Brotli compression codec

[Ryan Blue (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280469#comment-16280469
 ] 

Ryan Blue commented on HADOOP-13126:


I'm happy to if there is interest from reviewers.

> Add Brotli compression codec
> 
>
> Key: HADOOP-13126
> URL: https://issues.apache.org/jira/browse/HADOOP-13126
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: io
>Affects Versions: 2.7.2
>Reporter: Ryan Blue
>Assignee: Ryan Blue
> Attachments: HADOOP-13126.1.patch, HADOOP-13126.2.patch, 
> HADOOP-13126.3.patch, HADOOP-13126.4.patch, HADOOP-13126.5.patch
>
>
> I've been testing [Brotli|https://github.com/google/brotli/], a new 
> compression library based on LZ77 from Google. Google's [brotli 
> benchmarks|https://cran.r-project.org/web/packages/brotli/vignettes/brotli-2015-09-22.pdf]
>  look really good and we're also seeing a significant improvement in 
> compression size, compression speed, or both.
> {code:title=Brotli preliminary test results}
> [blue@work Downloads]$ time parquet from test.parquet -o test.snappy.parquet 
> --compression-codec snappy --overwrite  
> real1m17.106s
> user1m30.804s
> sys 0m4.404s
> [blue@work Downloads]$ time parquet from test.parquet -o test.br.parquet 
> --compression-codec brotli --overwrite 
> real1m16.640s
> user1m24.244s
> sys 0m6.412s
> [blue@work Downloads]$ time parquet from test.parquet -o test.gz.parquet 
> --compression-codec gzip --overwrite
> real3m39.496s
> user3m48.736s
> sys 0m3.880s
> [blue@work Downloads]$ ls -l
> -rw-r--r-- 1 blue blue 1068821936 May 10 11:06 test.br.parquet
> -rw-r--r-- 1 blue blue 1421601880 May 10 11:10 test.gz.parquet
> -rw-r--r-- 1 blue blue 2265950833 May 10 10:30 test.snappy.parquet
> {code}
> Brotli, at quality 1, is as fast as snappy and ends up smaller than gzip-9. 
> Another test resulted in a slightly larger Brotli file than gzip produced, 
> but Brotli was 4x faster. I'd like to get this compression codec into Hadoop.
> [Brotli is licensed with the MIT 
> license|https://github.com/google/brotli/blob/master/LICENSE], and the [JNI 
> library jbrotli is 
> ALv2|https://github.com/MeteoGroup/jbrotli/blob/master/LICENSE].



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15092) Proxy failures during NamenodeWebHdfsMethods are not logged

[Prabhu Joseph (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prabhu Joseph updated HADOOP-15092:
---
Attachment: HADOOP-15092.2.patch

> Proxy failures during NamenodeWebHdfsMethods are not logged 
> 
>
> Key: HADOOP-15092
> URL: https://issues.apache.org/jira/browse/HADOOP-15092
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.3
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
> Attachments: HADOOP-15092.1.patch, HADOOP-15092.2.patch, 
> HADOOP-15092.patch
>
>
> When GetDelegationToken http request fails with proxy issue, there is no logs 
> from NameNode to indicate it's a proxy issue. Below is the only log. 
> IOExceptions are logged but AuthorizationException are not logged. It will be 
> helpful to log proxy failures from JspHelper getUGI().
> {code}
> 2017-12-05 13:05:02,045 INFO  namenode.GetDelegationTokenServlet 
> (GetDelegationTokenServlet.java:doGet(56)) - Request for token received with 
> no authentication from 172.26.93.73
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15080) Cat-X dependency on org.json via derived json-lib

[Sean Mackrory (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280340#comment-16280340
 ] 

Sean Mackrory commented on HADOOP-15080:


I think what you're suggesting sounds good: if Hadoop itself can be developed, 
built and tested without json-lib being on the machine, it's a very big stretch 
to argue that the ASF is "using" it by any definition. Aliyun is using it, so 
it's Aliyun's call and a separate issue. Moving to jersey-json sounds good, but 
isn't required to unblock the release if we can get an SDK that already has the 
correctly limited SDK, in my opinion. I don't think we got an unambiguous 
answer, but it sounds like declaring a test-only dependency on json-lib in 
hadoop-aliyunoss to prevent it from being distributed to end-users might still 
be considered "use" in the sense that the license refers to.

> Cat-X dependency on org.json via derived json-lib
> -
>
> Key: HADOOP-15080
> URL: https://issues.apache.org/jira/browse/HADOOP-15080
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/oss
>Affects Versions: 3.0.0-beta1
>Reporter: Chris Douglas
>Priority: Blocker
> Attachments: HADOOP-15080-branch-3.0.0.001.patch
>
>
> The OSS SDK has a dependency on json-lib. In LEGAL-245, the org.json library 
> (from which json-lib may be derived) is released under a 
> [category-x|https://www.apache.org/legal/resolved.html#json] license.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15092) Proxy failures during NamenodeWebHdfsMethods are not logged

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280325#comment-16280325
 ] 

genericqa commented on HADOOP-15092:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  9m 
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 
 8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 37s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m 49s{color} | {color:green} patch has no errors when building and testing our 
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m 
40s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
32s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 89m  8s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15092 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12900868/HADOOP-15092.1.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux e25af6af3839 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 40b0045e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/13791/testReport/ |
| Max. process+thread count | 1717 (vs. ulimit of 5000) |
| modules | C: hadoop-common-project/hadoop-common U: 
hadoop-common-project/hadoop-common |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/13791/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Proxy failures during NamenodeWebHdfsMethods are not logged 
> 
>
> Key: 

[jira] [Updated] (HADOOP-15092) Proxy failures during NamenodeWebHdfsMethods are not logged

[Prabhu Joseph (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prabhu Joseph updated HADOOP-15092:
---
Attachment: HADOOP-15092.1.patch

> Proxy failures during NamenodeWebHdfsMethods are not logged 
> 
>
> Key: HADOOP-15092
> URL: https://issues.apache.org/jira/browse/HADOOP-15092
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.3
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
> Attachments: HADOOP-15092.1.patch, HADOOP-15092.patch
>
>
> When GetDelegationToken http request fails with proxy issue, there is no logs 
> from NameNode to indicate it's a proxy issue. Below is the only log. 
> IOExceptions are logged but AuthorizationException are not logged. It will be 
> helpful to log proxy failures from JspHelper getUGI().
> {code}
> 2017-12-05 13:05:02,045 INFO  namenode.GetDelegationTokenServlet 
> (GetDelegationTokenServlet.java:doGet(56)) - Request for token received with 
> no authentication from 172.26.93.73
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15095) S3a committer factory to warn when default FileOutputFormat committer is created

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-15095:
---

 Summary: S3a committer factory to warn when default 
FileOutputFormat committer is created
 Key: HADOOP-15095
 URL: https://issues.apache.org/jira/browse/HADOOP-15095
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3
Reporter: Steve Loughran
Priority: Minor


The S3ACommitterFactory should warn when the classic FileOutputCommitter is 
used (i.e. the client is not configured to use a new one). Something like

"this committer is neither fast nor guaranteed to be correct. See $URL" where 
URL is a pointer to something (wiki? hadoop docs?).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13974) S3a CLI to support list/purge of pending multipart commits

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-13974:

Parent Issue: HADOOP-14831  (was: HADOOP-13786)

> S3a CLI to support list/purge of pending multipart commits
> --
>
> Key: HADOOP-13974
> URL: https://issues.apache.org/jira/browse/HADOOP-13974
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Aaron Fabbri
> Attachments: HADOOP-13974.001.patch, HADOOP-13974.002.patch, 
> HADOOP-13974.003.patch
>
>
> The S3A CLI will need to be able to list and delete pending multipart 
> commits. 
> We can do the cleanup already via fs.s3a properties. The CLI will let scripts 
> stat for outstanding data (have a different exit code) and permit batch jobs 
> to explicitly trigger cleanups.
> This will become critical with the multipart committer, as there's a 
> significantly higher likelihood of commits remaining outstanding.
> We may also want to be able to enumerate/cancel all pending commits in the FS 
> tree



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13974) S3a CLI to support list/purge of pending multipart commits

[Steve Loughran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280152#comment-16280152
 ] 

Steve Loughran commented on HADOOP-13974:
-

Moved directly under the Phase IV JIRA, so we don't lose this

> S3a CLI to support list/purge of pending multipart commits
> --
>
> Key: HADOOP-13974
> URL: https://issues.apache.org/jira/browse/HADOOP-13974
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Aaron Fabbri
> Attachments: HADOOP-13974.001.patch, HADOOP-13974.002.patch, 
> HADOOP-13974.003.patch
>
>
> The S3A CLI will need to be able to list and delete pending multipart 
> commits. 
> We can do the cleanup already via fs.s3a properties. The CLI will let scripts 
> stat for outstanding data (have a different exit code) and permit batch jobs 
> to explicitly trigger cleanups.
> This will become critical with the multipart committer, as there's a 
> significantly higher likelihood of commits remaining outstanding.
> We may also want to be able to enumerate/cancel all pending commits in the FS 
> tree



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-13967) S3ABlockOutputStream to support plugin point for different multipart strategies

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HADOOP-13967.
-
   Resolution: Duplicate
Fix Version/s: 3.1.0

> S3ABlockOutputStream to support plugin point for different multipart 
> strategies
> ---
>
> Key: HADOOP-13967
> URL: https://issues.apache.org/jira/browse/HADOOP-13967
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: 3.1.0
>
>
> For 0-rename commits, we need to delay the final commit of a multipart PUT, 
> instead saving the data needed to build that commit into the s3 bucket.
> This means changes to {{S3ABlockOutputStream}} so that it can support 
> different policies on how to do this, "classic" and "delayed commit".
> Having this self contained means we can test it in isolation of anything else.
> I'm ignoring the old output stream...we will switch to fast output whenever a 
> special destination path is encountered



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-13968) S3a FS to support "__magic" path for the special "unmaterialized" writes

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HADOOP-13968.
-
   Resolution: Duplicate
 Assignee: Steve Loughran
Fix Version/s: 3.1.0

> S3a FS to support "__magic" path for the special "unmaterialized" writes
> 
>
> Key: HADOOP-13968
> URL: https://issues.apache.org/jira/browse/HADOOP-13968
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: 3.1.0
>
>
> S3AFileSystem to add support for a special path, such as  
> {{.temp_pending_put/}} or similar, which, when used as the base of a path, 
> indicates that the file is actually to be saved to the parent dir, but only 
> via a delayed put commit operation.
> At the same time, we may need blocks on some normal fileIO ops under these 
> dirs, especially rename and delete, as this would cause serious problems 
> including data loss and large bills for pending data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-13969) S3A to support commit(path) operation, which commits all pending put commits in a path

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HADOOP-13969.
-
   Resolution: Duplicate
 Assignee: Steve Loughran
Fix Version/s: 3.1.0

> S3A to support commit(path) operation, which commits all pending put commits 
> in a path
> --
>
> Key: HADOOP-13969
> URL: https://issues.apache.org/jira/browse/HADOOP-13969
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: 3.1.0
>
>
> as well as creating and saving data with a pending-commit, s3a needs to add 
> the actual commit operation.
> this would scan a directory, take its pending commits, read them in and 
> execute them. 
> issue: what to do on failures?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15092) Proxy failures during NamenodeWebHdfsMethods are not logged

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280099#comment-16280099
 ] 

genericqa commented on HADOOP-15092:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 25s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 
46s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 37s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch 
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m 47s{color} | {color:green} patch has no errors when building and testing our 
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m 
37s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
33s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 86m 24s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15092 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12900845/HADOOP-15092.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux e683952c798f 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 40b0045e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/13790/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/13790/testReport/ |
| Max. process+thread count | 1466 (vs. ulimit of 5000) |
| modules | C: hadoop-common-project/hadoop-common U: 
hadoop-common-project/hadoop-common |
| 

[jira] [Commented] (HADOOP-14914) Change to a safely casting long to int.

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280080#comment-16280080
 ] 

genericqa commented on HADOOP-14914:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
 2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 17s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
 4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m 55s{color} | {color:green} patch has no errors when building and testing our 
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 86m 37s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 39s{color} 
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
33s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}185m 35s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics |
|   | hadoop.fs.TestUnbuffer |
|   | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
|   | 
hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14914 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12900736/HADOOP-14914.002.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 5998c26bcfaa 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 
11:55:51 UTC 2017 x86_64 

[jira] [Updated] (HADOOP-15094) FileSystem.getCanonicalUri() to be public

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-15094:

Affects Version/s: 2.7.4
 Priority: Minor  (was: Major)
  Summary: FileSystem.getCanonicalUri() to be public  (was: 
FileSystem)
  Component/s: fs

> FileSystem.getCanonicalUri() to be public
> -
>
> Key: HADOOP-15094
> URL: https://issues.apache.org/jira/browse/HADOOP-15094
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs
>Affects Versions: 2.7.4
>Reporter: Steve Loughran
>Priority: Minor
>
> Discussion around SPARK-22587 highlights how per-fs notions of a canonical 
> URI make it hard to determine if a file is on a specific filesystem, or, put 
> differently, if two filesystems are equivalent.
> You can't reliably use this.getUri == that.getUri as it doesn't handle FQDN 
> == unqualified DN, bit you can't do nslookup as HDFS HA doesn't use hosnames.
> If {{FileSystem.getCanonicalUri()}} were public, then this could be used to 
> compare things consistently.
> needs: filesystem.md coverage; contract test (two filesystem instances are 
> equal, different filesystems aren't). Or at least: this method never returns 
> null.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15094) FileSystem

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-15094:
---

 Summary: FileSystem
 Key: HADOOP-15094
 URL: https://issues.apache.org/jira/browse/HADOOP-15094
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Steve Loughran


Discussion around SPARK-22587 highlights how per-fs notions of a canonical URI 
make it hard to determine if a file is on a specific filesystem, or, put 
differently, if two filesystems are equivalent.

You can't reliably use this.getUri == that.getUri as it doesn't handle FQDN == 
unqualified DN, bit you can't do nslookup as HDFS HA doesn't use hosnames.

If {{FileSystem.getCanonicalUri()}} were public, then this could be used to 
compare things consistently.

needs: filesystem.md coverage; contract test (two filesystem instances are 
equal, different filesystems aren't). Or at least: this method never returns 
null.





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15092) Proxy failures during NamenodeWebHdfsMethods are not logged

[Prabhu Joseph (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prabhu Joseph updated HADOOP-15092:
---
Status: Patch Available  (was: Open)

> Proxy failures during NamenodeWebHdfsMethods are not logged 
> 
>
> Key: HADOOP-15092
> URL: https://issues.apache.org/jira/browse/HADOOP-15092
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.3
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
> Attachments: HADOOP-15092.patch
>
>
> When GetDelegationToken http request fails with proxy issue, there is no logs 
> from NameNode to indicate it's a proxy issue. Below is the only log. 
> IOExceptions are logged but AuthorizationException are not logged. It will be 
> helpful to log proxy failures from JspHelper getUGI().
> {code}
> 2017-12-05 13:05:02,045 INFO  namenode.GetDelegationTokenServlet 
> (GetDelegationTokenServlet.java:doGet(56)) - Request for token received with 
> no authentication from 172.26.93.73
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15092) Proxy failures during NamenodeWebHdfsMethods are not logged

[Prabhu Joseph (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prabhu Joseph updated HADOOP-15092:
---
Attachment: HADOOP-15092.patch

> Proxy failures during NamenodeWebHdfsMethods are not logged 
> 
>
> Key: HADOOP-15092
> URL: https://issues.apache.org/jira/browse/HADOOP-15092
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.3
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
> Attachments: HADOOP-15092.patch
>
>
> When GetDelegationToken http request fails with proxy issue, there is no logs 
> from NameNode to indicate it's a proxy issue. Below is the only log. 
> IOExceptions are logged but AuthorizationException are not logged. It will be 
> helpful to log proxy failures from JspHelper getUGI().
> {code}
> 2017-12-05 13:05:02,045 INFO  namenode.GetDelegationTokenServlet 
> (GetDelegationTokenServlet.java:doGet(56)) - Request for token received with 
> no authentication from 172.26.93.73
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15092) Proxy failures during NamenodeWebHdfsMethods are not logged

[Prabhu Joseph (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1628#comment-1628
 ] 

Prabhu Joseph commented on HADOOP-15092:


AuthorizationException does not provide stacktrace for security purposes. 
Logger only prints the stacktrace of the Throwable and does not log the message 
in case if stacktrace is empty. Attached a patch which returns the message 
alone as part of printStackTrace() of AuthorizationException class.

> Proxy failures during NamenodeWebHdfsMethods are not logged 
> 
>
> Key: HADOOP-15092
> URL: https://issues.apache.org/jira/browse/HADOOP-15092
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: security
>Affects Versions: 2.7.3
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>
> When GetDelegationToken http request fails with proxy issue, there is no logs 
> from NameNode to indicate it's a proxy issue. Below is the only log. 
> IOExceptions are logged but AuthorizationException are not logged. It will be 
> helpful to log proxy failures from JspHelper getUGI().
> {code}
> 2017-12-05 13:05:02,045 INFO  namenode.GetDelegationTokenServlet 
> (GetDelegationTokenServlet.java:doGet(56)) - Request for token received with 
> no authentication from 172.26.93.73
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-10768) Optimize Hadoop RPC encryption performance

[Dapeng Sun (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-10768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279929#comment-16279929
 ] 

Dapeng Sun edited comment on HADOOP-10768 at 12/6/17 9:46 AM:
--

Thank [~daryn] for your comments!

JCE Cipher may not a good choice from performance aspect:
* From java 7u40, Cipher uses native intrinsics. But the performance is not 
good for CTR mode: it have been fixed at JDK 9 
https://bugs.openjdk.java.net/browse/JDK-8143925, For performance reason, we 
should use HadoopCryptoCodec or Apache Commons Crypto.
* About AES-GCM, JDK 8 and above would support it, but the performance of JCE 
was very bad (~Half of Openssl),  Apache Commons Crypto support GCM via 
openssl, but it haven't release now, and the performance of AES-GCM(openssl) ~= 
AES-CTR + MD5

 I would do more investigation on QOP and key exchange, and reply the detail 
tomorrow.



was (Author: dapengsun):
Thank [~daryn] for your comments!

JCE Cipher may not a good choice from performance aspect:
* From java 7u40, Cipher supposedly uses native intrinsics. But the performance 
is not good for CTR mode: it have been fixed at JDK 9 
https://bugs.openjdk.java.net/browse/JDK-8143925, For performance reason, we 
should use HadoopCryptoCodec or Apache Commons Crypto.
* About AES-GCM, JDK 8 and above would support it, but the performance of JCE 
was very bad (~Half of Openssl),  Apache Commons Crypto support GCM via 
openssl, but it haven't release now, and the performance of AES-GCM(openssl) ~= 
AES-CTR + MD5

 I would do more investigation on QOP and key exchange, and reply the detail 
tomorrow.


> Optimize Hadoop RPC encryption performance
> --
>
> Key: HADOOP-10768
> URL: https://issues.apache.org/jira/browse/HADOOP-10768
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: performance, security
>Affects Versions: 3.0.0-alpha1
>Reporter: Yi Liu
>Assignee: Dapeng Sun
> Attachments: HADOOP-10768.001.patch, HADOOP-10768.002.patch, 
> HADOOP-10768.003.patch, HADOOP-10768.004.patch, HADOOP-10768.005.patch, 
> HADOOP-10768.006.patch, HADOOP-10768.007.patch, HADOOP-10768.008.patch, 
> Optimize Hadoop RPC encryption performance.pdf
>
>
> Hadoop RPC encryption is enabled by setting {{hadoop.rpc.protection}} to 
> "privacy". It utilized SASL {{GSSAPI}} and {{DIGEST-MD5}} mechanisms for 
> secure authentication and data protection. Even {{GSSAPI}} supports using 
> AES, but without AES-NI support by default, so the encryption is slow and 
> will become bottleneck.
> After discuss with [~atm], [~tucu00] and [~umamaheswararao], we can do the 
> same optimization as in HDFS-6606. Use AES-NI with more than *20x* speedup.
> On the other hand, RPC message is small, but RPC is frequent and there may be 
> lots of RPC calls in one connection, we needs to setup benchmark to see real 
> improvement and then make a trade-off. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-10768) Optimize Hadoop RPC encryption performance

[Dapeng Sun (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-10768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279929#comment-16279929
 ] 

Dapeng Sun commented on HADOOP-10768:
-

Thank [~daryn] for your comments!

JCE Cipher may not a good choice from performance aspect:
* From java 7u40, Cipher supposedly uses native intrinsics. But the performance 
is not good for CTR mode: it have been fixed at JDK 9 
https://bugs.openjdk.java.net/browse/JDK-8143925, For performance reason, we 
should use HadoopCryptoCodec or Apache Commons Crypto.
* About AES-GCM, JDK 8 and above would support it, but the performance of JCE 
was very bad (~Half of Openssl),  Apache Commons Crypto support GCM via 
openssl, but it haven't release now, and the performance of AES-GCM(openssl) ~= 
AES-CTR + MD5

 I would do more investigation on QOP and key exchange, and reply the detail 
tomorrow.


> Optimize Hadoop RPC encryption performance
> --
>
> Key: HADOOP-10768
> URL: https://issues.apache.org/jira/browse/HADOOP-10768
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: performance, security
>Affects Versions: 3.0.0-alpha1
>Reporter: Yi Liu
>Assignee: Dapeng Sun
> Attachments: HADOOP-10768.001.patch, HADOOP-10768.002.patch, 
> HADOOP-10768.003.patch, HADOOP-10768.004.patch, HADOOP-10768.005.patch, 
> HADOOP-10768.006.patch, HADOOP-10768.007.patch, HADOOP-10768.008.patch, 
> Optimize Hadoop RPC encryption performance.pdf
>
>
> Hadoop RPC encryption is enabled by setting {{hadoop.rpc.protection}} to 
> "privacy". It utilized SASL {{GSSAPI}} and {{DIGEST-MD5}} mechanisms for 
> secure authentication and data protection. Even {{GSSAPI}} supports using 
> AES, but without AES-NI support by default, so the encryption is slow and 
> will become bottleneck.
> After discuss with [~atm], [~tucu00] and [~umamaheswararao], we can do the 
> same optimization as in HDFS-6606. Use AES-NI with more than *20x* speedup.
> On the other hand, RPC message is small, but RPC is frequent and there may be 
> lots of RPC calls in one connection, we needs to setup benchmark to see real 
> improvement and then make a trade-off. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14914) Change to a safely casting long to int.

[Yufei Gu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279846#comment-16279846
 ] 

Yufei Gu commented on HADOOP-14914:
---

Thanks for the patch. Submit path to kick off the Jenkins.

> Change to a safely casting long to int. 
> 
>
> Key: HADOOP-14914
> URL: https://issues.apache.org/jira/browse/HADOOP-14914
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>Assignee: Ajay Kumar
> Attachments: HADOOP-14914.001.patch, HADOOP-14914.002.patch
>
>
> There are bunches of casting long to int like this:
> {code}
> long l = 123
> int i = (int) l;
> {code}
> This is not a safe cast. if l is greater than Integer.MAX_VALUE, i would be 
> negative, which is an unexpected behavior. We probably at least want to throw 
> an exception in that case. I suggest to use {{Math.toIntExact(longValue)}} to 
> replace them, which throws an exception if the value overflows an int. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14914) Change to a safely casting long to int.

[Yufei Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated HADOOP-14914:
--
Status: Patch Available  (was: Open)

> Change to a safely casting long to int. 
> 
>
> Key: HADOOP-14914
> URL: https://issues.apache.org/jira/browse/HADOOP-14914
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>Assignee: Ajay Kumar
> Attachments: HADOOP-14914.001.patch, HADOOP-14914.002.patch
>
>
> There are bunches of casting long to int like this:
> {code}
> long l = 123
> int i = (int) l;
> {code}
> This is not a safe cast. if l is greater than Integer.MAX_VALUE, i would be 
> negative, which is an unexpected behavior. We probably at least want to throw 
> an exception in that case. I suggest to use {{Math.toIntExact(longValue)}} to 
> replace them, which throws an exception if the value overflows an int. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org