[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL

2017-01-23 Thread Daryn Sharp (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834797#comment-15834797
 ] 

Daryn Sharp commented on HADOOP-13836:
--

I'll try to review/comment in the next few days.  The cited performance hit is 
rather concerning though.

> Securing Hadoop RPC using SSL
> -
>
> Key: HADOOP-13836
> URL: https://issues.apache.org/jira/browse/HADOOP-13836
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: ipc
>Reporter: kartheek muthyala
>Assignee: kartheek muthyala
> Attachments: HADOOP-13836.patch, HADOOP-13836-v2.patch, 
> HADOOP-13836-v3.patch, Secure IPC OSS Proposal-1.pdf, SecureIPC Performance 
> Analysis-OSS.pdf
>
>
> Today, RPC connections in Hadoop are encrypted using Simple Authentication & 
> Security Layer (SASL), with the Kerberos ticket based authentication or 
> Digest-md5 checksum based authentication protocols. This proposal is about 
> enhancing this cipher suite with SSL/TLS based encryption and authentication. 
> SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that 
> provides data security and integrity across two different end points in a 
> network. This protocol has made its way to a number of applications such as 
> web browsing, email, internet faxing, messaging, VOIP etc. And supporting 
> this cipher suite at the core of Hadoop would give a good synergy with the 
> applications on top and also bolster industry adoption of Hadoop.
> The Server and Client code in Hadoop IPC should support the following modes 
> of communication
> 1.Plain 
> 2. SASL encryption with an underlying authentication
> 3. SSL based encryption and authentication (x509 certificate)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores

2017-01-23 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834776#comment-15834776
 ] 

Hadoop QA commented on HADOOP-13946:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
17s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m  8s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13946 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12848924/HADOOP-13946-004.patch
 |
| Optional Tests |  asflicense  mvnsite  |
| uname | Linux bc5550b0b49e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 3fa0d54 |
| whitespace | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/11496/artifact/patchprocess/whitespace-eol.txt
 |
| modules | C: hadoop-common-project/hadoop-common U: 
hadoop-common-project/hadoop-common |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/11496/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Document how HDFS updates timestamps in the FS spec; compare with object 
> stores
> ---
>
> Key: HADOOP-13946
> URL: https://issues.apache.org/jira/browse/HADOOP-13946
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: documentation, fs
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13946-001.patch, HADOOP-13946-002.patch, 
> HADOOP-13946-003.patch, HADOOP-13946-004.patch
>
>
> SPARK-17159 shows that the behavior of when HDFS updates timestamps isn't 
> well documented. Document these in the FS spec.
> I'm not going to add tests for this, as it is so very dependent on FS 
> implementations, as in "POSIX filesystems may behave differently from HDFS". 
> If someone knows what happens there, their contribution is welcome.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-8143) Change distcp to have -pb on by default

2017-01-23 Thread Jason Lowe (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834761#comment-15834761
 ] 

Jason Lowe commented on HADOOP-8143:


My apologies, this got lost again, and the patch needs a refresh.

I'm OK with this going into Hadoop 3.x and leaning towards allowing it into 
2.x.  [~aw] do you have any thoughts on where this should be applied?

> Change distcp to have -pb on by default
> ---
>
> Key: HADOOP-8143
> URL: https://issues.apache.org/jira/browse/HADOOP-8143
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Dave Thompson
>Assignee: Mithun Radhakrishnan
>Priority: Minor
>  Labels: BB2015-05-TBR
> Attachments: HADOOP-8143.1.patch
>
>
> We should have the preserve blocksize (-pb) on in distcp by default.
> checksum which is on by default will always fail if blocksize is not the same.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser

2017-01-23 Thread Greg Senia (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834756#comment-15834756
 ] 

Greg Senia commented on HADOOP-13988:
-

[~xyao] and [~lmccay] here is log output from 02.patch

2017-01-23 10:29:17,424 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:doAs(1744)) - PrivilegedActionException as:knox 
(auth:TOKEN) 
cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
 Operation category READ is not supported in state standby
2017-01-23 10:29:17,424 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:doAs(1744)) - PrivilegedActionException as:knox 
(auth:TOKEN) 
cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
 Operation category READ is not supported in state standby
2017-01-23 10:29:17,426 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:logPrivilegedAction(1767)) - PrivilegedAction 
as:knox (auth:TOKEN) 
from:org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:758)
2017-01-23 10:29:17,426 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:logPrivilegedAction(1767)) - PrivilegedAction 
as:knox (auth:TOKEN) 
from:org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:758)
2017-01-23 10:29:17,437 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:logAllUserInfo(1774)) - UGI: gss2002 (auth:PROXY) 
via knox (auth:TOKEN)
2017-01-23 10:29:17,437 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:logAllUserInfo(1774)) - UGI: gss2002 (auth:PROXY) 
via knox (auth:TOKEN)
2017-01-23 10:29:17,437 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:logAllUserInfo(1776)) - +RealUGI: knox (auth:TOKEN)
2017-01-23 10:29:17,437 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:logAllUserInfo(1776)) - +RealUGI: knox (auth:TOKEN)
2017-01-23 10:29:17,437 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:logAllUserInfo(1778)) - +LoginUGI: 
dn/ha20t5001dn.tech.hdp.example@tech.hdp.example.com (auth:KERBEROS)
2017-01-23 10:29:17,437 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:logAllUserInfo(1778)) - +LoginUGI: 
dn/ha20t5001dn.tech.hdp.example@tech.hdp.example.com (auth:KERBEROS)
2017-01-23 10:29:17,438 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:logAllUserInfo(1780)) - +UGI token:Kind: 
HDFS_DELEGATION_TOKEN, Service: ha-hdfs:tech, Ident: (HDFS_DELEGATION_TOKEN 
token 14676 for gss2002)
2017-01-23 10:29:17,438 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:logAllUserInfo(1780)) - +UGI token:Kind: 
HDFS_DELEGATION_TOKEN, Service: ha-hdfs:tech, Ident: (HDFS_DELEGATION_TOKEN 
token 14676 for gss2002)
2017-01-23 10:29:17,438 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:logAllUserInfo(1780)) - +UGI token:Kind: 
HDFS_DELEGATION_TOKEN, Service: 10.70.33.7:8020, Ident: (HDFS_DELEGATION_TOKEN 
token 14676 for gss2002)
2017-01-23 10:29:17,438 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:logAllUserInfo(1780)) - +UGI token:Kind: 
HDFS_DELEGATION_TOKEN, Service: 10.70.33.7:8020, Ident: (HDFS_DELEGATION_TOKEN 
token 14676 for gss2002)
2017-01-23 10:29:17,438 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:logAllUserInfo(1780)) - +UGI token:Kind: 
HDFS_DELEGATION_TOKEN, Service: 10.70.33.6:8020, Ident: (HDFS_DELEGATION_TOKEN 
token 14676 for gss2002)
2017-01-23 10:29:17,438 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:logAllUserInfo(1780)) - +UGI token:Kind: 
HDFS_DELEGATION_TOKEN, Service: 10.70.33.6:8020, Ident: (HDFS_DELEGATION_TOKEN 
token 14676 for gss2002)
2017-01-23 10:29:17,438 DEBUG kms.KMSClientProvider 
(KMSClientProvider.java:getActualUgi(1061)) - using loginUser no KMS Delegation 
Token no Kerberos Credentials
2017-01-23 10:29:17,438 DEBUG kms.KMSClientProvider 
(KMSClientProvider.java:getActualUgi(1061)) - using loginUser no KMS Delegation 
Token no Kerberos Credentials
2017-01-23 10:29:17,438 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:logPrivilegedAction(1767)) - PrivilegedAction 
as:dn/ha20t5001dn.tech.hdp.example@tech.hdp.example.com (auth:KERBEROS) 
from:org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:524)
2017-01-23 10:29:17,438 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:logPrivilegedAction(1767)) - PrivilegedAction 
as:dn/ha20t5001dn.tech.hdp.example@tech.hdp.example.com (auth:KERBEROS) 
from:org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:524)
2017-01-23 10:29:17,439 DEBUG security.UserGroupInformation 
(UserGroupInformation.java:getTGT(898)) - Found tgt Ticket (hex) = 


Client Principal = dn/ha20t5001dn.tech.hdp.example@tech.hdp.example.com
Server Principal = krbtgt/tech.hdp.example@tech.hdp.example.com
Session Key =

[jira] [Commented] (HADOOP-12990) lz4 incompatibility between OS and Hadoop

2017-01-23 Thread Jason Lowe (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834745#comment-15834745
 ] 

Jason Lowe commented on HADOOP-12990:
-

bq. Is there a streaming version of the decoder for Lz4? 

There is a streaming version of the lz4 compressor and decompressor.  See 
https://github.com/lz4/lz4/blob/dev/lib/lz4.h#L228 and 
https://github.com/lz4/lz4/blob/dev/lib/lz4.h#L273.  I'm not an expert on LZ4, 
but I'm assuming that these require the frame format rather than the block 
format.

bq. If I understood correctly, were you saying that I should be updating the 
Lz4Decompressor which is used within the Stream API?

It would be updating both the compressor and decompressor to use the streaming 
API provided by lz4 as documented in lz4.h.

As I mentioned earlier, the big decision is whether we should try to fix the 
existing Lz4Codec (i.e.: the one that claims the standard '.lz4' file 
extension) to be compatible with the existing, Hadoop-proprietary format and 
also with files using the .lz4 extension or we should create a completely 
separate codec that of course cannot use the '.lz4' extension.  I'd prefer the 
former, but there will be some cross-cluster incompatibilities if a 'new' 
Lz4Codec generates a compressed file and someone tries to decode it with the 
'old' Lz4Codec.


> lz4 incompatibility between OS and Hadoop
> -
>
> Key: HADOOP-12990
> URL: https://issues.apache.org/jira/browse/HADOOP-12990
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: io, native
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Priority: Minor
>
> {{hdfs dfs -text}} hit exception when trying to view the compression file 
> created by Linux lz4 tool.
> The Hadoop version has HADOOP-11184 "update lz4 to r123", thus it is using 
> LZ4 library in release r123.
> Linux lz4 version:
> {code}
> $ /tmp/lz4 -h 2>&1 | head -1
> *** LZ4 Compression CLI 64-bits r123, by Yann Collet (Apr  1 2016) ***
> {code}
> Test steps:
> {code}
> $ cat 10rows.txt
> 001|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 002|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 003|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 004|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 005|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 006|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 007|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 008|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 009|c1|c2|c3|c4|c5|c6|c7|c8|c9
> 010|c1|c2|c3|c4|c5|c6|c7|c8|c9
> $ /tmp/lz4 10rows.txt 10rows.txt.r123.lz4
> Compressed 310 bytes into 105 bytes ==> 33.87%
> $ hdfs dfs -put 10rows.txt.r123.lz4 /tmp
> $ hdfs dfs -text /tmp/10rows.txt.r123.lz4
> 16/04/01 08:19:07 INFO compress.CodecPool: Got brand-new decompressor [.lz4]
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at 
> org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:123)
> at 
> org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:98)
> at 
> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
> at java.io.InputStream.read(InputStream.java:101)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119)
> at org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:106)
> at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:101)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
> at 
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
> at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
> at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
> at 
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:118)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:315)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:372)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores

2017-01-23 Thread Steve Loughran (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-13946:

   Fix Version/s: (was: 3.0.0-alpha2)
  (was: 2.8.0)
Target Version/s: 2.8.1
  Status: Patch Available  (was: Open)

> Document how HDFS updates timestamps in the FS spec; compare with object 
> stores
> ---
>
> Key: HADOOP-13946
> URL: https://issues.apache.org/jira/browse/HADOOP-13946
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: documentation, fs
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13946-001.patch, HADOOP-13946-002.patch, 
> HADOOP-13946-003.patch, HADOOP-13946-004.patch
>
>
> SPARK-17159 shows that the behavior of when HDFS updates timestamps isn't 
> well documented. Document these in the FS spec.
> I'm not going to add tests for this, as it is so very dependent on FS 
> implementations, as in "POSIX filesystems may behave differently from HDFS". 
> If someone knows what happens there, their contribution is welcome.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores

2017-01-23 Thread Steve Loughran (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-13946:

Attachment: HADOOP-13946-004.patch

patch 004
* be more specific about HDFS, including its atime granularity
* mention where other filesystems are likely to vary
* be even vaguer about object stores

I think I've addressed chris's comments, and I've reached  the limits of my 
understanding, even with a bit more rummaging round the source tree.

> Document how HDFS updates timestamps in the FS spec; compare with object 
> stores
> ---
>
> Key: HADOOP-13946
> URL: https://issues.apache.org/jira/browse/HADOOP-13946
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: documentation, fs
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13946-001.patch, HADOOP-13946-002.patch, 
> HADOOP-13946-003.patch, HADOOP-13946-004.patch
>
>
> SPARK-17159 shows that the behavior of when HDFS updates timestamps isn't 
> well documented. Document these in the FS spec.
> I'm not going to add tests for this, as it is so very dependent on FS 
> implementations, as in "POSIX filesystems may behave differently from HDFS". 
> If someone knows what happens there, their contribution is welcome.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores

2017-01-23 Thread Steve Loughran (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-13946:

Status: Open  (was: Patch Available)

> Document how HDFS updates timestamps in the FS spec; compare with object 
> stores
> ---
>
> Key: HADOOP-13946
> URL: https://issues.apache.org/jira/browse/HADOOP-13946
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: documentation, fs
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13946-001.patch, HADOOP-13946-002.patch, 
> HADOOP-13946-003.patch
>
>
> SPARK-17159 shows that the behavior of when HDFS updates timestamps isn't 
> well documented. Document these in the FS spec.
> I'm not going to add tests for this, as it is so very dependent on FS 
> implementations, as in "POSIX filesystems may behave differently from HDFS". 
> If someone knows what happens there, their contribution is welcome.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores

2017-01-23 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834649#comment-15834649
 ] 

Hadoop QA commented on HADOOP-13946:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 36s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13946 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12848909/HADOOP-13946-003.patch
 |
| Optional Tests |  asflicense  mvnsite  |
| uname | Linux 2d91012570ca 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 3fa0d54 |
| whitespace | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/11495/artifact/patchprocess/whitespace-eol.txt
 |
| modules | C: hadoop-common-project/hadoop-common U: 
hadoop-common-project/hadoop-common |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/11495/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Document how HDFS updates timestamps in the FS spec; compare with object 
> stores
> ---
>
> Key: HADOOP-13946
> URL: https://issues.apache.org/jira/browse/HADOOP-13946
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: documentation, fs
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13946-001.patch, HADOOP-13946-002.patch, 
> HADOOP-13946-003.patch
>
>
> SPARK-17159 shows that the behavior of when HDFS updates timestamps isn't 
> well documented. Document these in the FS spec.
> I'm not going to add tests for this, as it is so very dependent on FS 
> implementations, as in "POSIX filesystems may behave differently from HDFS". 
> If someone knows what happens there, their contribution is welcome.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser

2017-01-23 Thread Greg Senia (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834630#comment-15834630
 ] 

Greg Senia commented on HADOOP-13988:
-

[~xyao] the second test fix seems to be working. I will leave it in my 
environment for a few days to make sure as kerberos tickets expire that the fix 
still works.

> KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
> 
>
> Key: HADOOP-13988
> URL: https://issues.apache.org/jira/browse/HADOOP-13988
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common, kms
>Affects Versions: 2.8.0, 2.7.3
> Environment: HDP 2.5.3.0 
> WebHDFSUser --> Knox --> HA NameNodes(WebHDFS) --> DataNodes
>Reporter: Greg Senia
> Attachments: HADOOP-13988.01.patch, HADOOP-13988.02.patch, 
> HADOOP-13988.patch, HADOOP-13988.patch
>
>
> After upgrading to HDP 2.5.3.0 noticed that all of the KMSClientProvider 
> issues have not been resolved. We put a test build together and applied 
> HADOOP-13558 and HADOOP-13749 these two fixes did still not solve the issue 
> with requests coming from WebHDFS through to Knox to a TDE zone.
> So we added some debug to our build and determined effectively what is 
> happening here is a double proxy situation which does not seem to work. So we 
> propose the following fix in getActualUgi Method:
> {noformat}
>  }
>  // Use current user by default
>  UserGroupInformation actualUgi = currentUgi;
>  if (currentUgi.getRealUser() != null) {
>// Use real user for proxy user
>if (LOG.isDebugEnabled()) {
>  LOG.debug("using RealUser for proxyUser);
>   }
>actualUgi = currentUgi.getRealUser();
>if (getDoAsUser() != null) {
> if (LOG.isDebugEnabled()) {
>   LOG.debug("doAsUser exists");
>   LOG.debug("currentUGI realUser shortName: {}", 
> currentUgi.getRealUser().getShortUserName());
>   LOG.debug("processUGI loginUser shortName: {}", 
> UserGroupInformation.getLoginUser().getShortUserName());
>   }
> if (currentUgi.getRealUser().getShortUserName() != 
> UserGroupInformation.getLoginUser().getShortUserName()) {
> if (LOG.isDebugEnabled()) {
>   LOG.debug("currentUGI.realUser does not match 
> UGI.processUser);
> }
> actualUgi = UserGroupInformation.getLoginUser();
> if (LOG.isDebugEnabled()) {
>   LOG.debug("LoginUser for Proxy: {}", 
> actualUgi.getLoginUser());
> }
> }
>}
>   
>  } else if (!currentUgiContainsKmsDt() &&
>  !currentUgi.hasKerberosCredentials()) {
>// Use login user for user that does not have either
>// Kerberos credential or KMS delegation token for KMS operations
>if (LOG.isDebugEnabled()) {
>  LOG.debug("using loginUser no KMS Delegation Token no Kerberos 
> Credentials");
>   }
>actualUgi = currentUgi.getLoginUser();
>  }
>  return actualUgi;
>}
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14019) fix some typos in the s3a docs

2017-01-23 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-14019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834627#comment-15834627
 ] 

Hadoop QA commented on HADOOP-14019:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
17s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m  1s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-14019 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12848906/HADOOP-14019-001.patch
 |
| Optional Tests |  asflicense  mvnsite  |
| uname | Linux 9154aaff881e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 3fa0d54 |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/11494/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> fix some typos in the s3a docs
> --
>
> Key: HADOOP-14019
> URL: https://issues.apache.org/jira/browse/HADOOP-14019
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: documentation, fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HADOOP-14019-001.patch
>
>
> There's a few errors in the s3a docs, including one cut-and-paste error 
> related to the per-bucket config and JCEKS files which is potentially 
> misleading.
> fix



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores

2017-01-23 Thread Steve Loughran (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-13946:

Attachment: HADOOP-13946-003.patch

patch 003. Make it even more clear we aren't sure what is happening. Add that 
rename() often/usually/may update the timestamps (ish)

> Document how HDFS updates timestamps in the FS spec; compare with object 
> stores
> ---
>
> Key: HADOOP-13946
> URL: https://issues.apache.org/jira/browse/HADOOP-13946
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: documentation, fs
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13946-001.patch, HADOOP-13946-002.patch, 
> HADOOP-13946-003.patch
>
>
> SPARK-17159 shows that the behavior of when HDFS updates timestamps isn't 
> well documented. Document these in the FS spec.
> I'm not going to add tests for this, as it is so very dependent on FS 
> implementations, as in "POSIX filesystems may behave differently from HDFS". 
> If someone knows what happens there, their contribution is welcome.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14019) fix some typos in the s3a docs

2017-01-23 Thread Steve Loughran (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-14019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-14019:

Status: Patch Available  (was: Open)

> fix some typos in the s3a docs
> --
>
> Key: HADOOP-14019
> URL: https://issues.apache.org/jira/browse/HADOOP-14019
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: documentation, fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HADOOP-14019-001.patch
>
>
> There's a few errors in the s3a docs, including one cut-and-paste error 
> related to the per-bucket config and JCEKS files which is potentially 
> misleading.
> fix



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14019) fix some typos in the s3a docs

2017-01-23 Thread Steve Loughran (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-14019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-14019:

Attachment: HADOOP-14019-001.patch

patch 001

> fix some typos in the s3a docs
> --
>
> Key: HADOOP-14019
> URL: https://issues.apache.org/jira/browse/HADOOP-14019
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: documentation, fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HADOOP-14019-001.patch
>
>
> There's a few errors in the s3a docs, including one cut-and-paste error 
> related to the per-bucket config and JCEKS files which is potentially 
> misleading.
> fix



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14019) fix some typos in the s3a docs

2017-01-23 Thread Steve Loughran (JIRA) 
Steve Loughran created HADOOP-14019:
---

 Summary: fix some typos in the s3a docs
 Key: HADOOP-14019
 URL: https://issues.apache.org/jira/browse/HADOOP-14019
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: documentation, fs/s3
Affects Versions: 2.8.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor


There's a few errors in the s3a docs, including one cut-and-paste error related 
to the per-bucket config and JCEKS files which is potentially misleading.

fix



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-12097) Allow port range to be specified while starting webapp

2017-01-23 Thread Varun Saxena (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-12097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834521#comment-15834521
 ] 

Varun Saxena commented on HADOOP-12097:
---

[~djp], just saw your comment. Will update the patch by tomorrow.



> Allow port range to be specified while starting webapp 
> ---
>
> Key: HADOOP-12097
> URL: https://issues.apache.org/jira/browse/HADOOP-12097
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>  Labels: webapp
> Attachments: HADOOP-12097.01.patch, HADOOP-12097.02.patch, tests.patch
>
>
> This can be used by AM while starting its webapp.
> Currently a random port is chosen, which is never a good idea.
>  
> And {{Webapps}} has facility to allow only a specific port to be configured 
> which may be occupied. A port range in case of AM, hence will be useful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL

2017-01-23 Thread Antonios Kouzoupis (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834427#comment-15834427
 ] 

Antonios Kouzoupis commented on HADOOP-13836:
-

[~kartheek] you can use org.apache.hadoop.security.ssl.KeyStoreTestUtils to 
create all the necessary cryptographic material before running your JUnit tests 
instead of shipping binaries.

> Securing Hadoop RPC using SSL
> -
>
> Key: HADOOP-13836
> URL: https://issues.apache.org/jira/browse/HADOOP-13836
> Project: Hadoop Common
>  Issue Type: New Feature
>  Components: ipc
>Reporter: kartheek muthyala
>Assignee: kartheek muthyala
> Attachments: HADOOP-13836.patch, HADOOP-13836-v2.patch, 
> HADOOP-13836-v3.patch, Secure IPC OSS Proposal-1.pdf, SecureIPC Performance 
> Analysis-OSS.pdf
>
>
> Today, RPC connections in Hadoop are encrypted using Simple Authentication & 
> Security Layer (SASL), with the Kerberos ticket based authentication or 
> Digest-md5 checksum based authentication protocols. This proposal is about 
> enhancing this cipher suite with SSL/TLS based encryption and authentication. 
> SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that 
> provides data security and integrity across two different end points in a 
> network. This protocol has made its way to a number of applications such as 
> web browsing, email, internet faxing, messaging, VOIP etc. And supporting 
> this cipher suite at the core of Hadoop would give a good synergy with the 
> applications on top and also bolster industry adoption of Hadoop.
> The Server and Client code in Hadoop IPC should support the following modes 
> of communication
> 1.Plain 
> 2. SASL encryption with an underlying authentication
> 3. SSL based encryption and authentication (x509 certificate)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13453) S3Guard: Instrument new functionality with Hadoop metrics.

2017-01-23 Thread Steve Loughran (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-13453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834359#comment-15834359
 ] 

Steve Loughran commented on HADOOP-13453:
-

Hi, don't worry about asking questions, we'll do our best to get you 
contributing code —it benefits all of us if you are adding code to Hadoop.

The split between low level increment named counter and more elegant "event 
with internal counters?". The event ones are cleaner, as they stop the rest of 
the code having to know exactly which counters/gauges to use. Consider the 
elegant ones the best approach, and the direct invocation us being lazy.

The S3aInstrumentation class also has a set of explicit named counters 
"filesDeleted" as well as lots of ones that are only listed in the arrays 
{{GAUGES_TO_CREATE}} and {{COUNTERS_TO_CREATE}}. That's evolution over time; I 
got bored of having to name and register lots of fields, and realised I could 
do it from the arrays, at the cost of a hash lookup on every increment.

Outside the S3a class itself, i've tried to have external inner classes to do 
the counting, with the results merged in at the end (example: the input and 
output streams), with the inner classes using simple long values, rather than 
atomics. Why? Eliminates any delays during increments, and lets us override the 
toString() values for input/output streams with dumps of the values (go on, try 
it!). We can have many input/output streams per FS instance, so the risk of 
contention for atomic int/log values is potentially quite high.

I think for s3guard we could add a new inner class passed in to each s3guard 
instance; it would export the various methods for events that s3guard could 
raise, such as {{tableCreated()}}, {{tableDeleted()}} —these can directly 
increment the atomic counters in the instrumentation, as we'd only have a 1:1 
map of S3aFS instance and a s3guard store instance.

Regarding access the statistics, that's hooked up to 
{{FileSystem.getStorageStatistics()}}, which is intended to provide the storage 
stats for any FS; s3a and HDFS share common statistic names for the common 
statistics. The latest versions of Tez do collect the statistics of jobs, and 
so give you the aggregate statistics across your entire query. Until now, only 
{{Filesystem.getStatistics()}} has been used, which returns a fixed set of 
values (bytes read/written, etc). Spark still only collects those; it'd take 
some migration to hadoop 2.8+ to pick up the new data. Until then, it's 
something we can use in tests.



> S3Guard: Instrument new functionality with Hadoop metrics.
> --
>
> Key: HADOOP-13453
> URL: https://issues.apache.org/jira/browse/HADOOP-13453
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Reporter: Chris Nauroth
>Assignee: Ai Deng
>
> Provide Hadoop metrics showing operational details of the S3Guard 
> implementation.
> The metrics will be implemented in this ticket:
> ● S3GuardRechecksNthPercentileLatency (MutableQuantiles) ­​ Percentile time 
> spent
> in rechecks attempting to achieve consistency. Repeated for multiple 
> percentile values
> of N.  This metric is an indicator of the additional latency cost of running 
> S3A with
> S3Guard.
> ● S3GuardRechecksNumOps (MutableQuantiles) ­​ Number of times a consistency
> recheck was required while attempting to achieve consistency.
> ● S3GuardStoreNthPercentileLatency (MutableQuantiles) ­​ Percentile time 
> spent in
> operations against the consistent store, including both write operations 
> during file system
> mutations and read operations during file system consistency checks. Repeated 
> for
> multiple percentile values of N. This metric is an indicator of latency to 
> the consistent
> store implementation.
> ● S3GuardConsistencyStoreNumOps (MutableQuantiles) ­​ Number of operations
> against the consistent store, including both write operations during file 
> system mutations
> and read operations during file system consistency checks.
> ● S3GuardConsistencyStoreFailures (MutableCounterLong) ­​ Number of failures
> during operations against the consistent store implementation.
> ● S3GuardConsistencyStoreTimeouts (MutableCounterLong) ­​ Number of timeouts
> during operations against the consistent store implementation.
> ● S3GuardInconsistencies (MutableCounterLong) ­ C​ ount of times S3Guard 
> failed to
> achieve consistency, even after exhausting all rechecks. A high count may 
> indicate
> unexpected out­of­band modification of the S3 bucket contents, such as by an 
> external
> tool that does not make corresponding updates to the consistent store.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail:

[jira] [Commented] (HADOOP-14012) Handled dynamo exceptions in translateException

2017-01-23 Thread Steve Loughran (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-14012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834331#comment-15834331
 ] 

Steve Loughran commented on HADOOP-14012:
-

For {{AbortedException}}, we could uprate to `InterruptedIOException`, assuming 
the primary cause was that some other thread cancelled the operation

> Handled dynamo exceptions in translateException
> ---
>
> Key: HADOOP-14012
> URL: https://issues.apache.org/jira/browse/HADOOP-14012
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Steve Loughran
>
> {{translateException}} is good at translating S3 exceptions into meaningful 
> IOEs. We need to make sure that dynamodb exceptions are also mapped 
> appropriately. 
> Action plan
> # break things
> # collect the stack traces
> # translate them where possible
> # discuss in troubleshooting section of s3guard



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14013) S3Guard: fix multi-bucket integration tests

2017-01-23 Thread Steve Loughran (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-14013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834323#comment-15834323
 ] 

Steve Loughran commented on HADOOP-14013:
-

Of course, I don't see this because I explicity set landsat-pds to to null 
store.

How about we

# in test/resources/core-site.xml make the default store for landsat-pds to be 
local
# in the s3guard testing docs, cover the "testing CSV and read-only data", so 
that if someone is using a different CSV source, they know what to do
# make sure the general s3guard docs covers read only buckets

This could be a good time to split out the testing bit of site/index.md to its 
own doc, say site/testing.md, which would then point at s3guard.md for its 
testing section

> S3Guard: fix multi-bucket integration tests
> ---
>
> Key: HADOOP-14013
> URL: https://issues.apache.org/jira/browse/HADOOP-14013
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Aaron Fabbri
>Assignee: Aaron Fabbri
>
> Clone of HADOOP-13876 to address multi-bucket test failures separately.
> One of the issues is with accessing read-only buckets.  If a user accesses a 
> read-only bucket with credentials that do not have DynamoDB write 
> permissions, they will get errors when trying to access the read-only bucket. 
>  This manifests causes test failures for {{ITestS3AAWSCredentialsProvider}}.
> Goals for this JIRA:
> - Fix {{ITestS3AAWSCredentialsProvider}} in a way that makes sense for the 
> real use-case.
> - Document limitations etc. in the s3guard.md site doc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14013) S3Guard: fix multi-bucket integration tests

2017-01-23 Thread Steve Loughran (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-14013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834317#comment-15834317
 ] 

Steve Loughran commented on HADOOP-14013:
-

If the test fs is r/o, then I'd expect most of the s3a test suite to fail. Is 
it that there's a special case where the code tries to read from the landsat 
repo and create a bucket for it?

> S3Guard: fix multi-bucket integration tests
> ---
>
> Key: HADOOP-14013
> URL: https://issues.apache.org/jira/browse/HADOOP-14013
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Aaron Fabbri
>Assignee: Aaron Fabbri
>
> Clone of HADOOP-13876 to address multi-bucket test failures separately.
> One of the issues is with accessing read-only buckets.  If a user accesses a 
> read-only bucket with credentials that do not have DynamoDB write 
> permissions, they will get errors when trying to access the read-only bucket. 
>  This manifests causes test failures for {{ITestS3AAWSCredentialsProvider}}.
> Goals for this JIRA:
> - Fix {{ITestS3AAWSCredentialsProvider}} in a way that makes sense for the 
> real use-case.
> - Document limitations etc. in the s3guard.md site doc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL

2017-01-23 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834089#comment-15834089
 ] 

Hadoop QA commented on HADOOP-13836:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 4 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 17m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  2m 
29s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color} 
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  4m 
22s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m 
44s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 40s{color} | {color:orange} root: The patch generated 46 new + 429 unchanged 
- 16 fixed = 475 total (was 445) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  8m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 
 6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch 19 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color} 
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  4m 
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m  2s{color} 
| {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
37s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}144m 40s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ipc.TestSSLIPC |
|   | hadoop.ipc.TestSSLSocketFactory |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13836 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12848837/HADOOP-13836-v3.patch 
|
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  xml  findbugs  checkstyle  |
| uname | Linux b1b951ed5040 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / a847903 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |