date:20150312


[ 
https://issues.apache.org/jira/browse/HADOOP-11705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358222#comment-14358222
 ] 

Kai Zheng commented on HADOOP-11705:


Committed in HDFS-7285 branch.

 Make erasure coder configurable
 ---

 Key: HADOOP-11705
 URL: https://issues.apache.org/jira/browse/HADOOP-11705
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: io
Reporter: Kai Zheng
Assignee: Kai Zheng
 Attachments: HADOOP-11705-v1.patch, HADOOP-11705-v2.patch


 This is to enhance erasure coder and raw coder to make them configurable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HADOOP-11705) Make erasure coder configurable


 [ 
https://issues.apache.org/jira/browse/HADOOP-11705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kai Zheng resolved HADOOP-11705.

   Resolution: Fixed
Fix Version/s: HDFS-7285
 Hadoop Flags: Reviewed

 Make erasure coder configurable
 ---

 Key: HADOOP-11705
 URL: https://issues.apache.org/jira/browse/HADOOP-11705
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: io
Reporter: Kai Zheng
Assignee: Kai Zheng
 Fix For: HDFS-7285

 Attachments: HADOOP-11705-v1.patch, HADOOP-11705-v2.patch


 This is to enhance erasure coder and raw coder to make them configurable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11638) OpensslSecureRandom.c pthreads_thread_id should support FreeBSD and Solaris in addition to Linux


[ 
https://issues.apache.org/jira/browse/HADOOP-11638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355396#comment-14355396
 ] 

Hudson commented on HADOOP-11638:
-

FAILURE: Integrated in Hadoop-trunk-Commit #7298 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/7298/])
Revert HADOOP-11638. OpensslSecureRandom.c pthreads_thread_id should support 
FreeBSD and Solaris in addition to Linux (Kiran Kumar M R via Colin P.  
McCabe) (cnauroth: rev 8d5b01e005b2647262861acf522e9b5c6d6f8bba)
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c
* hadoop-common-project/hadoop-common/CHANGES.txt


 OpensslSecureRandom.c pthreads_thread_id should support FreeBSD and Solaris 
 in addition to Linux
 

 Key: HADOOP-11638
 URL: https://issues.apache.org/jira/browse/HADOOP-11638
 Project: Hadoop Common
  Issue Type: Bug
  Components: native
Affects Versions: 2.6.0
Reporter: Dmitry Sivachenko
Assignee: Kiran Kumar M R
  Labels: freebsd
 Attachments: HADOOP-11638-001.patch, HADOOP-11638-002.patch, 
 HADOOP-11638-003.patch


 In OpensslSecureRandom.c you use Linux-specific syscall gettid():
 static unsigned long pthreads_thread_id(void)
 {
 return (unsigned long)syscall(SYS_gettid);
 }
 Man page says:
 gettid()  is Linux-specific and should not be used in programs that are
 intended to be portable.
 This breaks hadoop-2.6.0 compilation on FreeBSD (may be on other OSes too).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-9329) document native build dependencies in BUILDING.txt


 [ 
https://issues.apache.org/jira/browse/HADOOP-9329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vijay Bhat updated HADOOP-9329:
---
Release Note: 
Added a section to BUILDING.txt on how to install required / optional packages 
on a clean install of Ubuntu 14.04 LTS Desktop.

Went through the CMakeLists.txt files in the repo and found 3 optional library 
dependencies - Snappy, Bzip2, Linux FUSE and Jansson.

Updated the required packages / version numbers from the trunk branch version 
of BUILDING.txt.


  was:
Added a section to BUILDING.txt on how to install required / optional packages 
on a clean install of Ubuntu 14.04 LTS Desktop.

Went through the CMakeLists.txt files in the repo and found 3 optional library 
dependencies - Snappy, Bzip2 and Jansson.

Updated the required packages / version numbers from the trunk branch version 
of BUILDING.txt.



 document native build dependencies in BUILDING.txt
 --

 Key: HADOOP-9329
 URL: https://issues.apache.org/jira/browse/HADOOP-9329
 Project: Hadoop Common
  Issue Type: Task
  Components: documentation
Affects Versions: 2.1.0-beta
Reporter: Colin Patrick McCabe
Assignee: Vijay Bhat
Priority: Trivial
 Attachments: HADOOP-9329.001.patch, HADOOP-9329.002.patch


 {{BUILDING.txt}} describes {{-Pnative}}, but it does not specify what native 
 libraries are needed for the build.  We should address this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11659) o.a.h.fs.FileSystem.Cache#remove should use a single hash map lookup

2015-03-12 Thread Gera Shegalov (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358279#comment-14358279
 ] 

Gera Shegalov commented on HADOOP-11659:


fwiw we can do instead of 2 lookups via get and remove, just remove, if it was 
wrong (unlikely) just put it back:
{code}
  FileSystem cachedFs = map.remove(key);
  if (fs == cachedFs) {
toAutoClose.remove(key);
  } else if (cachedFs != null) {
map.put(key, cachedFs);
  }
{code}

 o.a.h.fs.FileSystem.Cache#remove should use a single hash map lookup
 

 Key: HADOOP-11659
 URL: https://issues.apache.org/jira/browse/HADOOP-11659
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Affects Versions: 2.6.0
Reporter: Gera Shegalov
Assignee: Brahma Reddy Battula
Priority: Minor
 Attachments: HADOOP-11659-002.patch, HADOOP-11659-003.patch, 
 HADOOP-11659.patch


 The method looks up the same key in the same hash map potentially 3 times
 {code}
 if (map.containsKey(key)  fs == map.get(key)) {
   map.remove(key)
 {code}
 Instead it could do a single lookup
 {code}
 FileSystem cachedFs = map.remove(key);
 {code}
 and then test cachedFs == fs or something else.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11647) Reed-Solomon ErasureCoder


 [ 
https://issues.apache.org/jira/browse/HADOOP-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kai Zheng updated HADOOP-11647:
---
Attachment: HADOOP-11647-v4.patch

Updated the patch accordingly.

 Reed-Solomon ErasureCoder
 -

 Key: HADOOP-11647
 URL: https://issues.apache.org/jira/browse/HADOOP-11647
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
 Attachments: HADOOP-11647-v2.patch, HADOOP-11647-v4.patch, 
 HDFS-7664-v1.patch


 This is to implement Reed-Solomon ErasureCoder using the API defined in 
 HADOOP-11646. It supports to plugin via configuration for concrete 
 RawErasureCoder, using either JRSErasureCoder added in HDFS-7418 or 
 IsaRSErasureCoder added in HDFS-7338.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11638) OpensslSecureRandom.c pthreads_thread_id should support FreeBSD and Solaris in addition to Linux


[ 
https://issues.apache.org/jira/browse/HADOOP-11638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356944#comment-14356944
 ] 

Hudson commented on HADOOP-11638:
-

FAILURE: Integrated in Hadoop-Hdfs-trunk #2061 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2061/])
Revert HADOOP-11638. OpensslSecureRandom.c pthreads_thread_id should support 
FreeBSD and Solaris in addition to Linux (Kiran Kumar M R via Colin P.  
McCabe) (cnauroth: rev 8d5b01e005b2647262861acf522e9b5c6d6f8bba)
* hadoop-common-project/hadoop-common/CHANGES.txt
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c


 OpensslSecureRandom.c pthreads_thread_id should support FreeBSD and Solaris 
 in addition to Linux
 

 Key: HADOOP-11638
 URL: https://issues.apache.org/jira/browse/HADOOP-11638
 Project: Hadoop Common
  Issue Type: Bug
  Components: native
Affects Versions: 2.6.0
Reporter: Dmitry Sivachenko
Assignee: Kiran Kumar M R
  Labels: freebsd
 Attachments: HADOOP-11638-001.patch, HADOOP-11638-002.patch, 
 HADOOP-11638-003.patch


 In OpensslSecureRandom.c you use Linux-specific syscall gettid():
 static unsigned long pthreads_thread_id(void)
 {
 return (unsigned long)syscall(SYS_gettid);
 }
 Man page says:
 gettid()  is Linux-specific and should not be used in programs that are
 intended to be portable.
 This breaks hadoop-2.6.0 compilation on FreeBSD (may be on other OSes too).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11638) OpensslSecureRandom.c pthreads_thread_id should support FreeBSD and Solaris in addition to Linux


[ 
https://issues.apache.org/jira/browse/HADOOP-11638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357011#comment-14357011
 ] 

Hudson commented on HADOOP-11638:
-

FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #129 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/129/])
Revert HADOOP-11638. OpensslSecureRandom.c pthreads_thread_id should support 
FreeBSD and Solaris in addition to Linux (Kiran Kumar M R via Colin P.  
McCabe) (cnauroth: rev 8d5b01e005b2647262861acf522e9b5c6d6f8bba)
* hadoop-common-project/hadoop-common/CHANGES.txt
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c


 OpensslSecureRandom.c pthreads_thread_id should support FreeBSD and Solaris 
 in addition to Linux
 

 Key: HADOOP-11638
 URL: https://issues.apache.org/jira/browse/HADOOP-11638
 Project: Hadoop Common
  Issue Type: Bug
  Components: native
Affects Versions: 2.6.0
Reporter: Dmitry Sivachenko
Assignee: Kiran Kumar M R
  Labels: freebsd
 Attachments: HADOOP-11638-001.patch, HADOOP-11638-002.patch, 
 HADOOP-11638-003.patch


 In OpensslSecureRandom.c you use Linux-specific syscall gettid():
 static unsigned long pthreads_thread_id(void)
 {
 return (unsigned long)syscall(SYS_gettid);
 }
 Man page says:
 gettid()  is Linux-specific and should not be used in programs that are
 intended to be portable.
 This breaks hadoop-2.6.0 compilation on FreeBSD (may be on other OSes too).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11638) OpensslSecureRandom.c pthreads_thread_id should support FreeBSD and Solaris in addition to Linux

2015-03-12 Thread Kiran Kumar M R (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357133#comment-14357133
 ] 

Kiran Kumar M R commented on HADOOP-11638:
--

I have added new patch-004 to include Mac OS specific code.
Mac has {{int pthread_threadid_np(pthread_t thread, uint64_t *thread_id)}} 
method to get thread id 
- Its declared in {{pthread.h}} Refer 
https://opensource.apple.com/source/libpthread/libpthread-105.1.4/pthread/pthread.h
- Its implementation can be seen at 
https://opensource.apple.com/source/libpthread/libpthread-105.1.4/src/pthread.c
- Its usage can be seen at 
https://opensource.apple.com/source/libpthread/libpthread-105.1.4/src/pthread_rwlock.c
 as below
{code}
  uint64_t myid;
  (void)pthread_threadid_np(pthread_self(), myid);
{code}

Now my patch code is:
{code}
static unsigned long pthreads_thread_id(void)
{
  unsigned long thread_id = 0;
#if defined(__linux__)
  thread_id = (unsigned long)syscall(SYS_gettid);
#elif defined(__FreeBSD__)
  thread_id = (unsigned long)pthread_getthreadid_np();
#elif defined(__sun)
  thread_id = (unsigned long)pthread_self();
#elif defined(__APPLE__ )
  (void)pthread_threadid_np(pthread_self(), thread_id);
#else
#error Platform not supported
#endif
  return thread_id;
}
{code}

[~cnauroth], please check if patch-004 works on Mac


 OpensslSecureRandom.c pthreads_thread_id should support FreeBSD and Solaris 
 in addition to Linux
 

 Key: HADOOP-11638
 URL: https://issues.apache.org/jira/browse/HADOOP-11638
 Project: Hadoop Common
  Issue Type: Bug
  Components: native
Affects Versions: 2.6.0
Reporter: Dmitry Sivachenko
Assignee: Kiran Kumar M R
  Labels: freebsd
 Attachments: HADOOP-11638-001.patch, HADOOP-11638-002.patch, 
 HADOOP-11638-003.patch, HADOOP-11638-004.patch


 In OpensslSecureRandom.c you use Linux-specific syscall gettid():
 static unsigned long pthreads_thread_id(void)
 {
 return (unsigned long)syscall(SYS_gettid);
 }
 Man page says:
 gettid()  is Linux-specific and should not be used in programs that are
 intended to be portable.
 This breaks hadoop-2.6.0 compilation on FreeBSD (may be on other OSes too).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (HADOOP-11703) Gitignore should ignore .DS_Store files for mac

2015-03-12 Thread Abin Shahab (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-11703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HADOOP-11703 started by Abin Shahab.

 Gitignore should ignore .DS_Store files for mac
 ---

 Key: HADOOP-11703
 URL: https://issues.apache.org/jira/browse/HADOOP-11703
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Abin Shahab
Assignee: Abin Shahab





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HADOOP-11137) put up guard rails around pid and log file handling


 [ 
https://issues.apache.org/jira/browse/HADOOP-11137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer reassigned HADOOP-11137:
-

Assignee: Allen Wittenauer  (was: Brahma Reddy Battula)

 put up guard rails around pid and log file handling
 ---

 Key: HADOOP-11137
 URL: https://issues.apache.org/jira/browse/HADOOP-11137
 Project: Hadoop Common
  Issue Type: Improvement
  Components: scripts, security
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
  Labels: newbie, scripts, security

 We should do a better job of protecting against symlink attacks in the pid 
 and log file handling code:
 a) Change the default location to have a user or id.str component
 b) Check to make sure a pid file is actually a pid file (single line, nothing 
 but numbers)
 ... maybe other stuff?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-7494) Add -c option for FSshell -tail


 [ 
https://issues.apache.org/jira/browse/HADOOP-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-7494:
-
Status: Open  (was: Patch Available)

 Add -c option for FSshell -tail
 ---

 Key: HADOOP-7494
 URL: https://issues.apache.org/jira/browse/HADOOP-7494
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Affects Versions: 0.23.0
Reporter: XieXianshan
Assignee: XieXianshan
Priority: Trivial
 Attachments: HADOOP-7494-v0.2.patch, HADOOP-7494-v0.3.patch, 
 HADOOP-7494.patch


 Add the -c option for FSshell -tail to allow users to specify the output 
 bytes(currently,it's -1024 by default).
 For instance:
 $ hdfs dfs -tail -c -10 /user/hadoop/xiexs
 or
 $ hdfs dfs -tail -c+10 /user/hadoop/xiexs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11659) o.a.h.fs.FileSystem.Cache#remove should use a single hash map lookup

2015-03-12 Thread Brahma Reddy Battula (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358393#comment-14358393
 ] 

Brahma Reddy Battula commented on HADOOP-11659:
---

Thanks for review,Updated the patch ..Kindly review..

 o.a.h.fs.FileSystem.Cache#remove should use a single hash map lookup
 

 Key: HADOOP-11659
 URL: https://issues.apache.org/jira/browse/HADOOP-11659
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Affects Versions: 2.6.0
Reporter: Gera Shegalov
Assignee: Brahma Reddy Battula
Priority: Minor
 Attachments: HADDOOP-11659-004.patch, HADOOP-11659-002.patch, 
 HADOOP-11659-003.patch, HADOOP-11659.patch


 The method looks up the same key in the same hash map potentially 3 times
 {code}
 if (map.containsKey(key)  fs == map.get(key)) {
   map.remove(key)
 {code}
 Instead it could do a single lookup
 {code}
 FileSystem cachedFs = map.remove(key);
 {code}
 and then test cachedFs == fs or something else.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11638) OpensslSecureRandom.c pthreads_thread_id should support FreeBSD and Solaris in addition to Linux


[ 
https://issues.apache.org/jira/browse/HADOOP-11638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357040#comment-14357040
 ] 

Hudson commented on HADOOP-11638:
-

FAILURE: Integrated in Hadoop-Mapreduce-trunk #2079 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2079/])
Revert HADOOP-11638. OpensslSecureRandom.c pthreads_thread_id should support 
FreeBSD and Solaris in addition to Linux (Kiran Kumar M R via Colin P.  
McCabe) (cnauroth: rev 8d5b01e005b2647262861acf522e9b5c6d6f8bba)
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c
* hadoop-common-project/hadoop-common/CHANGES.txt


 OpensslSecureRandom.c pthreads_thread_id should support FreeBSD and Solaris 
 in addition to Linux
 

 Key: HADOOP-11638
 URL: https://issues.apache.org/jira/browse/HADOOP-11638
 Project: Hadoop Common
  Issue Type: Bug
  Components: native
Affects Versions: 2.6.0
Reporter: Dmitry Sivachenko
Assignee: Kiran Kumar M R
  Labels: freebsd
 Attachments: HADOOP-11638-001.patch, HADOOP-11638-002.patch, 
 HADOOP-11638-003.patch


 In OpensslSecureRandom.c you use Linux-specific syscall gettid():
 static unsigned long pthreads_thread_id(void)
 {
 return (unsigned long)syscall(SYS_gettid);
 }
 Man page says:
 gettid()  is Linux-specific and should not be used in programs that are
 intended to be portable.
 This breaks hadoop-2.6.0 compilation on FreeBSD (may be on other OSes too).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11647) Reed-Solomon ErasureCoder


[ 
https://issues.apache.org/jira/browse/HADOOP-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358330#comment-14358330
 ] 

Kai Zheng commented on HADOOP-11647:


In the update I left the configuration of raw coder for other issue to think 
about, because it involves non-trivial work that's better to be done elsewhere.

 Reed-Solomon ErasureCoder
 -

 Key: HADOOP-11647
 URL: https://issues.apache.org/jira/browse/HADOOP-11647
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
 Attachments: HADOOP-11647-v2.patch, HADOOP-11647-v4.patch, 
 HDFS-7664-v1.patch


 This is to implement Reed-Solomon ErasureCoder using the API defined in 
 HADOOP-11646. It supports to plugin via configuration for concrete 
 RawErasureCoder, using either JRSErasureCoder added in HDFS-7418 or 
 IsaRSErasureCoder added in HDFS-7338.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11526) Memory leak in Bzip2Compressor and Bzip2Decompressor

2015-03-12 Thread Johannes Zillmann (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358328#comment-14358328
 ] 

Johannes Zillmann commented on HADOOP-11526:


Question on the implications of this bug...
Given you open a couple of hundreds bzip compressed files, all mostly below 
1MB, could that crash an 2.5 GB jvm with an OutOfMemory ?

 Memory leak in Bzip2Compressor and Bzip2Decompressor
 

 Key: HADOOP-11526
 URL: https://issues.apache.org/jira/browse/HADOOP-11526
 Project: Hadoop Common
  Issue Type: Bug
  Components: io, native
Affects Versions: 2.6.0
Reporter: Ian Rogers
Assignee: Anu Engineer
 Fix For: 2.7.0

 Attachments: HADOOP-11526.001.patch, HADOOP-11526.002.patch


 The use of JNI's GetStringUTFChars should be paired with 
 ReleaseStringUTFChars or else the utf-8 char* created by Java's JNI 
 implementation is leaked. It isn't in Bzip2Decompressor.c:
 https://apache.googlesource.com/hadoop-common/+/refs/heads/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/bzip2/Bzip2Decompressor.c#45
 A less error-prone way of handling JNI resources like local references and 
 UTF strings is to use a smart pointer like the Apache licensed code in 
 Android's ScopedLocalRef and ScopedUtfChars:
 https://android.googlesource.com/platform/libnativehelper/+/jb-mr1.1-dev-plus-aosp/include/nativehelper/ScopedLocalRef.h
 https://android.googlesource.com/platform/libnativehelper/+/jb-mr1.1-dev-plus-aosp/include/nativehelper/ScopedUtfChars.h



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-10709) Reuse Filters across web apps

2015-03-12 Thread Ryan Sasson (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-10709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357244#comment-14357244
 ] 

Ryan Sasson commented on HADOOP-10709:
--

This is true, alt kerberos must be set for both core-site.xml and 
hdfs-site.xml. Currently this is not documented, I think [~aw] has filed a 
ticket or will file a ticket about it.

 Reuse Filters across web apps
 -

 Key: HADOOP-10709
 URL: https://issues.apache.org/jira/browse/HADOOP-10709
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Reporter: Benoy Antony
Assignee: Benoy Antony
 Attachments: HADOOP-10709-002.patch, HADOOP-10709.patch, 
 configForFilterReuse.xml


 Currently, we need to define separate authentication filters for webhdfs and 
 general webui. This also involves defining parameters for those filters.
 It will be better if one could reuse filters for web apps if desired. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11698) remove distcpv1 from hadoop-extras

2015-03-12 Thread Brahma Reddy Battula (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357304#comment-14357304
 ] 

Brahma Reddy Battula commented on HADOOP-11698:
---

{{Logalyzer}} is using distcpv1 while archiving the logs..so can we keep this 
one or give alternative or remove {{logalyzer}} also..

[~allend] can you please give your opinion..?

{code}
 public void
doArchive(String logListURI, String archiveDirectory)
throws IOException
  {
String destURL = FileSystem.getDefaultUri(fsConfig) + archiveDirectory;
DistCpV1.copy(new JobConf(fsConfig), logListURI, destURL, null, true, 
false);
  }
{code}

 remove distcpv1 from hadoop-extras
 --

 Key: HADOOP-11698
 URL: https://issues.apache.org/jira/browse/HADOOP-11698
 Project: Hadoop Common
  Issue Type: Improvement
  Components: tools/distcp
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Brahma Reddy Battula

 distcpv1 is pretty much unsupported. we should just remove it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11693) Azure Storage FileSystem rename operations are throttled too aggressively to complete HBase WAL archiving.

2015-03-12 Thread Chris Nauroth (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357332#comment-14357332
 ] 

Chris Nauroth commented on HADOOP-11693:


Thanks for the explanation about normal case vs. the new backoff policy.  That 
makes sense.

The eclipse:eclipse failure looks unrelated.  I couldn't reproduce it locally.

Sorry to nitpick, but there are still some lines in 
{{AzureNativeFileSystemStore}} that exceed the 80 character limit.  I know 
there are some existing lines in this file that already break the rule.  Don't 
worry about cleaning up all of the existing code, but please make sure all 
lines touched in the patch adhere to the 80 character limit.

The findbugs warning is legitimate.  I'm not sure why it's triggering now with 
this patch, as it appears the problem existed before the patch.  We can fix 
this by changing the {{catch (Exception e)}} so that there are 2 separate catch 
clauses for {{catch (StorageException e)}} and {{catch (URISyntaxException 
e)}}.  Each one can be rethrown wrapped as an {{AzureException}}.

We're almost there.  Thanks, Duo!

 Azure Storage FileSystem rename operations are throttled too aggressively to 
 complete HBase WAL archiving.
 --

 Key: HADOOP-11693
 URL: https://issues.apache.org/jira/browse/HADOOP-11693
 Project: Hadoop Common
  Issue Type: Bug
  Components: tools
Reporter: Duo Xu
Assignee: Duo Xu
 Attachments: HADOOP-11681.01.patch, HADOOP-11681.02.patch


 One of our customers' production HBase clusters was periodically throttled by 
 Azure storage, when HBase was archiving old WALs. HMaster aborted the region 
 server and tried to restart it.
 However, since the cluster was still being throttled by Azure storage, the 
 upcoming distributed log splitting also failed. Sometimes hbase:meta table 
 was on this region server and finally showed offline, which cause the whole 
 cluster in bad state.
 {code}
 2015-03-01 18:36:45,623 ERROR org.apache.hadoop.hbase.master.HMaster: Region 
 server 
 workernode4.hbaseproddb4001.f5.internal.cloudapp.net,60020,1424845421044 
 reported a fatal error:
 ABORTING region server 
 workernode4.hbaseproddb4001.f5.internal.cloudapp.net,60020,1424845421044: IOE 
 in log roller
 Cause:
 org.apache.hadoop.fs.azure.AzureException: 
 com.microsoft.windowsazure.storage.StorageException: The server is busy.
   at 
 org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2446)
   at 
 org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2367)
   at 
 org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1960)
   at 
 org.apache.hadoop.hbase.util.FSUtils.renameAndSetModifyTime(FSUtils.java:1719)
   at 
 org.apache.hadoop.hbase.regionserver.wal.FSHLog.archiveLogFile(FSHLog.java:798)
   at 
 org.apache.hadoop.hbase.regionserver.wal.FSHLog.cleanOldLogs(FSHLog.java:656)
   at 
 org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:593)
   at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:97)
   at java.lang.Thread.run(Thread.java:745)
 Caused by: com.microsoft.windowsazure.storage.StorageException: The server is 
 busy.
   at 
 com.microsoft.windowsazure.storage.StorageException.translateException(StorageException.java:163)
   at 
 com.microsoft.windowsazure.storage.core.StorageRequest.materializeException(StorageRequest.java:306)
   at 
 com.microsoft.windowsazure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:229)
   at 
 com.microsoft.windowsazure.storage.blob.CloudBlob.startCopyFromBlob(CloudBlob.java:762)
   at 
 org.apache.hadoop.fs.azurenative.StorageInterfaceImpl$CloudBlobWrapperImpl.startCopyFromBlob(StorageInterfaceImpl.java:350)
   at 
 org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2439)
   ... 8 more
 2015-03-01 18:43:29,072 ERROR org.apache.hadoop.hbase.executor.EventHandler: 
 Caught throwable while processing event M_META_SERVER_SHUTDOWN
 java.io.IOException: failed log splitting for 
 workernode13.hbaseproddb4001.f5.internal.cloudapp.net,60020,1424845307901, 
 will retry
   at 
 org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.process(MetaServerShutdownHandler.java:71)
   at 
 org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:745)
 Caused by:

[jira] [Updated] (HADOOP-11659) o.a.h.fs.FileSystem.Cache#remove should use a single hash map lookup

2015-03-12 Thread Brahma Reddy Battula (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-11659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula updated HADOOP-11659:
--
Attachment: HADDOOP-11659-004.patch

 o.a.h.fs.FileSystem.Cache#remove should use a single hash map lookup
 

 Key: HADOOP-11659
 URL: https://issues.apache.org/jira/browse/HADOOP-11659
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Affects Versions: 2.6.0
Reporter: Gera Shegalov
Assignee: Brahma Reddy Battula
Priority: Minor
 Attachments: HADDOOP-11659-004.patch, HADOOP-11659-002.patch, 
 HADOOP-11659-003.patch, HADOOP-11659.patch


 The method looks up the same key in the same hash map potentially 3 times
 {code}
 if (map.containsKey(key)  fs == map.get(key)) {
   map.remove(key)
 {code}
 Instead it could do a single lookup
 {code}
 FileSystem cachedFs = map.remove(key);
 {code}
 and then test cachedFs == fs or something else.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-9477) posixGroups support for LDAP groups mapping service

2015-03-12 Thread Dapeng Sun (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357094#comment-14357094
 ] 

Dapeng Sun commented on HADOOP-9477:


Hi [~yzhangal] 
Thank you very much for your comments, I'm totally agreed with you, I had 
updated the patch. If you have any comments, please feel free to let me know.

Regards
Dapeng

 posixGroups support for LDAP groups mapping service
 ---

 Key: HADOOP-9477
 URL: https://issues.apache.org/jira/browse/HADOOP-9477
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.0.4-alpha
Reporter: Kai Zheng
Assignee: Dapeng Sun
 Attachments: HADOOP-9477.003.patch, HADOOP-9477.004.patch, 
 HADOOP-9477.005.patch, HADOOP-9477.006.patch, HADOOP-9477.007.patch, 
 HADOOP-9477.008.patch, HADOOP-9477.patch, HADOOP-9477.patch

   Original Estimate: 168h
  Remaining Estimate: 168h

 It would be nice to support posixGroups for LdapGroupsMapping service. Below 
 is from current description for the provider:
 hadoop.security.group.mapping.ldap.search.filter.group:
 An additional filter to use when searching for LDAP groups. This should be
 changed when resolving groups against a non-Active Directory installation.
 posixGroups are currently not a supported group class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-7494) Add -c option for FSshell -tail


[ 
https://issues.apache.org/jira/browse/HADOOP-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357277#comment-14357277
 ] 

Hadoop QA commented on HADOOP-7494:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12490411/HADOOP-7494-v0.3.patch
  against trunk revision 30c428a.

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/5916//console

This message is automatically generated.

 Add -c option for FSshell -tail
 ---

 Key: HADOOP-7494
 URL: https://issues.apache.org/jira/browse/HADOOP-7494
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Affects Versions: 0.23.0
Reporter: XieXianshan
Assignee: XieXianshan
Priority: Trivial
 Attachments: HADOOP-7494-v0.2.patch, HADOOP-7494-v0.3.patch, 
 HADOOP-7494.patch


 Add the -c option for FSshell -tail to allow users to specify the output 
 bytes(currently,it's -1024 by default).
 For instance:
 $ hdfs dfs -tail -c -10 /user/hadoop/xiexs
 or
 $ hdfs dfs -tail -c+10 /user/hadoop/xiexs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11706) Refine a little bit erasure coder API

Kai Zheng created HADOOP-11706:
--

 Summary: Refine a little bit erasure coder API
 Key: HADOOP-11706
 URL: https://issues.apache.org/jira/browse/HADOOP-11706
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Kai Zheng
Assignee: Kai Zheng
Priority: Minor


In HADOOP-11705 it allows erasure coder class to be configurable, but not the 
interface, thus introcues not elegant codes of type casting. This is minor to 
refine the codes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11706) Refine a little bit erasure coder API


 [ 
https://issues.apache.org/jira/browse/HADOOP-11706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kai Zheng updated HADOOP-11706:
---
Attachment: HADOOP-11706-v1.patch

Uploaded the patch. Ready for review.

 Refine a little bit erasure coder API
 -

 Key: HADOOP-11706
 URL: https://issues.apache.org/jira/browse/HADOOP-11706
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Kai Zheng
Assignee: Kai Zheng
Priority: Minor
 Fix For: HDFS-7285

 Attachments: HADOOP-11706-v1.patch


 In HADOOP-11705 it allows erasure coder class to be configurable, but not the 
 interface, thus introcues not elegant codes of type casting. This is minor to 
 refine the codes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11706) Refine a little bit erasure coder API


 [ 
https://issues.apache.org/jira/browse/HADOOP-11706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kai Zheng updated HADOOP-11706:
---
Fix Version/s: HDFS-7285

 Refine a little bit erasure coder API
 -

 Key: HADOOP-11706
 URL: https://issues.apache.org/jira/browse/HADOOP-11706
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Kai Zheng
Assignee: Kai Zheng
Priority: Minor
 Fix For: HDFS-7285


 In HADOOP-11705 it allows erasure coder class to be configurable, but not the 
 interface, thus introcues not elegant codes of type casting. This is minor to 
 refine the codes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11693) Azure Storage FileSystem rename operations are throttled too aggressively to complete HBase WAL archiving.


[ 
https://issues.apache.org/jira/browse/HADOOP-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357590#comment-14357590
 ] 

Hadoop QA commented on HADOOP-11693:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12704005/HADOOP-11681.03.patch
  against trunk revision 344d7cb.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-tools/hadoop-azure.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/5919//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/5919//console

This message is automatically generated.

 Azure Storage FileSystem rename operations are throttled too aggressively to 
 complete HBase WAL archiving.
 --

 Key: HADOOP-11693
 URL: https://issues.apache.org/jira/browse/HADOOP-11693
 Project: Hadoop Common
  Issue Type: Bug
  Components: tools
Reporter: Duo Xu
Assignee: Duo Xu
 Attachments: HADOOP-11681.01.patch, HADOOP-11681.02.patch, 
 HADOOP-11681.03.patch


 One of our customers' production HBase clusters was periodically throttled by 
 Azure storage, when HBase was archiving old WALs. HMaster aborted the region 
 server and tried to restart it.
 However, since the cluster was still being throttled by Azure storage, the 
 upcoming distributed log splitting also failed. Sometimes hbase:meta table 
 was on this region server and finally showed offline, which cause the whole 
 cluster in bad state.
 {code}
 2015-03-01 18:36:45,623 ERROR org.apache.hadoop.hbase.master.HMaster: Region 
 server 
 workernode4.hbaseproddb4001.f5.internal.cloudapp.net,60020,1424845421044 
 reported a fatal error:
 ABORTING region server 
 workernode4.hbaseproddb4001.f5.internal.cloudapp.net,60020,1424845421044: IOE 
 in log roller
 Cause:
 org.apache.hadoop.fs.azure.AzureException: 
 com.microsoft.windowsazure.storage.StorageException: The server is busy.
   at 
 org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2446)
   at 
 org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2367)
   at 
 org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1960)
   at 
 org.apache.hadoop.hbase.util.FSUtils.renameAndSetModifyTime(FSUtils.java:1719)
   at 
 org.apache.hadoop.hbase.regionserver.wal.FSHLog.archiveLogFile(FSHLog.java:798)
   at 
 org.apache.hadoop.hbase.regionserver.wal.FSHLog.cleanOldLogs(FSHLog.java:656)
   at 
 org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:593)
   at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:97)
   at java.lang.Thread.run(Thread.java:745)
 Caused by: com.microsoft.windowsazure.storage.StorageException: The server is 
 busy.
   at 
 com.microsoft.windowsazure.storage.StorageException.translateException(StorageException.java:163)
   at 
 com.microsoft.windowsazure.storage.core.StorageRequest.materializeException(StorageRequest.java:306)
   at 
 com.microsoft.windowsazure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:229)
   at 
 com.microsoft.windowsazure.storage.blob.CloudBlob.startCopyFromBlob(CloudBlob.java:762)
   at 
 org.apache.hadoop.fs.azurenative.StorageInterfaceImpl$CloudBlobWrapperImpl.startCopyFromBlob(StorageInterfaceImpl.java:350)
   at 
 org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2439)
   ... 8 more
 2015-03-01 18:43:29,072 ERROR org.apache.hadoop.hbase.executor.EventHandler: 
 Caught throwable while processing event M_META_SERVER_SHUTDOWN
 java.io.IOException: failed log splitting for 
 workernode13.hbaseproddb4001.f5.internal.cloudapp.net,60020,1424845307901, 
 will retry
   at 
 org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.process(MetaServerShutdownHandler.java:71)
   at

[jira] [Created] (HADOOP-11707) Add factory to create raw erasure coder

Kai Zheng created HADOOP-11707:
--

 Summary: Add factory to create raw erasure coder
 Key: HADOOP-11707
 URL: https://issues.apache.org/jira/browse/HADOOP-11707
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng


We have {{RawErasureEncoder}} and {{RawErasureDecoder}} interface separately, 
which simplifies the implementation of raw coders. This would require to 
configure raw encoder and decoder respectively for a {{ErasureCoder}}, which 
isn't convenient. To simplify the configuration, we would have coder factory to 
group encoder and decoder together so only a factory class needs to be 
configured.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11707) Add factory to create raw erasure coder


 [ 
https://issues.apache.org/jira/browse/HADOOP-11707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kai Zheng updated HADOOP-11707:
---
Fix Version/s: HDFS-7285

 Add factory to create raw erasure coder
 ---

 Key: HADOOP-11707
 URL: https://issues.apache.org/jira/browse/HADOOP-11707
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: io
Reporter: Kai Zheng
Assignee: Kai Zheng
 Fix For: HDFS-7285


 We have {{RawErasureEncoder}} and {{RawErasureDecoder}} interface separately, 
 which simplifies the implementation of raw coders. This would require to 
 configure raw encoder and decoder respectively for a {{ErasureCoder}}, which 
 isn't convenient. To simplify the configuration, we would have coder factory 
 to group encoder and decoder together so only a factory class needs to be 
 configured.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11647) Reed-Solomon ErasureCoder


[ 
https://issues.apache.org/jira/browse/HADOOP-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358530#comment-14358530
 ] 

Kai Zheng commented on HADOOP-11647:


HADOOP-11707 was opened to help implement the configuration of raw coder for a 
{{ErasureCoder}}.

 Reed-Solomon ErasureCoder
 -

 Key: HADOOP-11647
 URL: https://issues.apache.org/jira/browse/HADOOP-11647
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
 Attachments: HADOOP-11647-v2.patch, HADOOP-11647-v4.patch, 
 HDFS-7664-v1.patch


 This is to implement Reed-Solomon ErasureCoder using the API defined in 
 HADOOP-11646. It supports to plugin via configuration for concrete 
 RawErasureCoder, using either JRSErasureCoder added in HDFS-7418 or 
 IsaRSErasureCoder added in HDFS-7338.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-10027) *Compressor_deflateBytesDirect passes instance instead of jclass to GetStaticObjectField


[ 
https://issues.apache.org/jira/browse/HADOOP-10027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358536#comment-14358536
 ] 

Hudson commented on HADOOP-10027:
-

FAILURE: Integrated in Hadoop-Yarn-trunk #864 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/864/])
HADOOP-10027. *Compressor_deflateBytesDirect passes instance instead of jclass 
to GetStaticObjectField. Contributed by Hui Zheng. (cnauroth: rev 
ff83ae72318fe6c0f266a7bf08138bd2fdb51cbd)
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/bzip2/Bzip2Compressor.c
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/zlib/TestZlibCompressorDecompressor.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/Bzip2Compressor.java
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
* hadoop-common-project/hadoop-common/CHANGES.txt
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/lz4/TestLz4CompressorDecompressor.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/snappy/TestSnappyCompressorDecompressor.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zlib/ZlibDecompressor.c
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/Bzip2Decompressor.java
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/bzip2/Bzip2Decompressor.c
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyCompressor.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/ZlibCompressor.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/ZlibDecompressor.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/bzip2/TestBzip2CompressorDecompressor.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java


 *Compressor_deflateBytesDirect passes instance instead of jclass to 
 GetStaticObjectField
 

 Key: HADOOP-10027
 URL: https://issues.apache.org/jira/browse/HADOOP-10027
 Project: Hadoop Common
  Issue Type: Bug
  Components: native
Reporter: Eric Abbott
Assignee: Hui Zheng
Priority: Minor
 Fix For: 2.8.0

 Attachments: HADOOP-10027.1.patch, HADOOP-10027.2.patch, 
 HADOOP-10027.3.patch, HADOOP-10027.4.patch, HADOOP-10027.5.patch


 http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c?view=markup
 This pattern appears in all the native compressors.
 // Get members of ZlibCompressor
 jobject clazz = (*env)-GetStaticObjectField(env, this,
  ZlibCompressor_clazz);
 The 2nd argument to GetStaticObjectField is supposed to be a jclass, not a 
 jobject. Adding the JVM param -Xcheck:jni will cause FATAL ERROR in native 
 method: JNI received a class argument that is not a class and a core dump 
 such as the following.
 (gdb) 
 #0 0x7f02e4aef8a5 in raise () from /lib64/libc.so.6
 #1 0x7f02e4af1085 in abort () from /lib64/libc.so.6
 #2 0x7f02e45bd727 in os::abort(bool) () from 
 /opt/jdk1.6.0_31/jre/lib/amd64/server/libjvm.so
 #3 0x7f02e43cec63 in jniCheck::validate_class(JavaThread*, _jclass*, 
 bool) () from /opt/jdk1.6.0_31/jre/lib/amd64/server/libjvm.so
 #4 0x7f02e43ea669 in checked_jni_GetStaticObjectField () from 
 /opt/jdk1.6.0_31/jre/lib/amd64/server/libjvm.so
 #5 0x7f02d38eaf79 in 
 Java_org_apache_hadoop_io_compress_zlib_ZlibCompressor_deflateBytesDirect () 
 from /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
 In addition, that clazz object is only used for synchronization. In the case 
 of the native method _deflateBytesDirect, the result is a class wide lock 
 used to access the instance field

[jira] [Commented] (HADOOP-11703) git should ignore .DS_Store files on Mac OS X


[ 
https://issues.apache.org/jira/browse/HADOOP-11703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358534#comment-14358534
 ] 

Hudson commented on HADOOP-11703:
-

FAILURE: Integrated in Hadoop-Yarn-trunk #864 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/864/])
HADOOP-11703. git should ignore .DS_Store files on Mac OS X (Abin Shahab via 
aw) (aw: rev 344d7cb6dcd3b1752c82b62271799be9e4e10416)
* .gitignore
* hadoop-common-project/hadoop-common/CHANGES.txt


 git should ignore .DS_Store files on Mac OS X
 -

 Key: HADOOP-11703
 URL: https://issues.apache.org/jira/browse/HADOOP-11703
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Abin Shahab
Assignee: Abin Shahab
 Fix For: 3.0.0

 Attachments: HADOOP-11703.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11665) Provide and unify cross platform byteorder support in native code

2015-03-12 Thread Ayappan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-11665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ayappan updated HADOOP-11665:
-
Priority: Blocker  (was: Major)

 Provide and unify cross platform byteorder support in native code
 -

 Key: HADOOP-11665
 URL: https://issues.apache.org/jira/browse/HADOOP-11665
 Project: Hadoop Common
  Issue Type: Bug
  Components: util
Affects Versions: 2.4.1, 2.6.0
 Environment: PowerPC Big Endian  other Big Endian platforms
Reporter: Binglin Chang
Assignee: Binglin Chang
Priority: Blocker
 Attachments: HADOOP-11665.001.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-10846) DataChecksum#calculateChunkedSums not working for PPC when buffers not backed by array

2015-03-12 Thread Ayappan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-10846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ayappan updated HADOOP-10846:
-
Priority: Blocker  (was: Major)

 DataChecksum#calculateChunkedSums not working for PPC when buffers not backed 
 by array
 --

 Key: HADOOP-10846
 URL: https://issues.apache.org/jira/browse/HADOOP-10846
 Project: Hadoop Common
  Issue Type: Bug
  Components: util
Affects Versions: 2.4.1, 2.5.2
 Environment: PowerPC platform
Reporter: Jinghui Wang
Assignee: Ayappan
Priority: Blocker
 Attachments: HADOOP-10846-v1.patch, HADOOP-10846-v2.patch, 
 HADOOP-10846-v3.patch, HADOOP-10846-v4.patch, HADOOP-10846.patch


 Got the following exception when running Hadoop on Power PC. The 
 implementation for computing checksum when the data buffer and checksum 
 buffer are not backed by arrays.
 13/09/16 04:06:57 ERROR security.UserGroupInformation: 
 PriviledgedActionException as:biadmin (auth:SIMPLE) 
 cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): 
 org.apache.hadoop.fs.ChecksumException: Checksum error



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11693) Azure Storage FileSystem rename operations are throttled too aggressively to complete HBase WAL archiving.


[ 
https://issues.apache.org/jira/browse/HADOOP-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358752#comment-14358752
 ] 

Hudson commented on HADOOP-11693:
-

FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #121 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/121/])
HADOOP-11693. Azure Storage FileSystem rename operations are throttled too 
aggressively to complete HBase WAL archiving. Contributed by Duo Xu. (cnauroth: 
rev 7a346bcb4fa5b56191ed00a39e72e51c9bdf1b56)
* 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterface.java
* 
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/MockStorageInterface.java
* 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
* 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterfaceImpl.java
* hadoop-common-project/hadoop-common/CHANGES.txt


 Azure Storage FileSystem rename operations are throttled too aggressively to 
 complete HBase WAL archiving.
 --

 Key: HADOOP-11693
 URL: https://issues.apache.org/jira/browse/HADOOP-11693
 Project: Hadoop Common
  Issue Type: Bug
  Components: tools
Reporter: Duo Xu
Assignee: Duo Xu
 Fix For: 2.7.0

 Attachments: HADOOP-11681.01.patch, HADOOP-11681.02.patch, 
 HADOOP-11681.03.patch


 One of our customers' production HBase clusters was periodically throttled by 
 Azure storage, when HBase was archiving old WALs. HMaster aborted the region 
 server and tried to restart it.
 However, since the cluster was still being throttled by Azure storage, the 
 upcoming distributed log splitting also failed. Sometimes hbase:meta table 
 was on this region server and finally showed offline, which cause the whole 
 cluster in bad state.
 {code}
 2015-03-01 18:36:45,623 ERROR org.apache.hadoop.hbase.master.HMaster: Region 
 server 
 workernode4.hbaseproddb4001.f5.internal.cloudapp.net,60020,1424845421044 
 reported a fatal error:
 ABORTING region server 
 workernode4.hbaseproddb4001.f5.internal.cloudapp.net,60020,1424845421044: IOE 
 in log roller
 Cause:
 org.apache.hadoop.fs.azure.AzureException: 
 com.microsoft.windowsazure.storage.StorageException: The server is busy.
   at 
 org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2446)
   at 
 org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2367)
   at 
 org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1960)
   at 
 org.apache.hadoop.hbase.util.FSUtils.renameAndSetModifyTime(FSUtils.java:1719)
   at 
 org.apache.hadoop.hbase.regionserver.wal.FSHLog.archiveLogFile(FSHLog.java:798)
   at 
 org.apache.hadoop.hbase.regionserver.wal.FSHLog.cleanOldLogs(FSHLog.java:656)
   at 
 org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:593)
   at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:97)
   at java.lang.Thread.run(Thread.java:745)
 Caused by: com.microsoft.windowsazure.storage.StorageException: The server is 
 busy.
   at 
 com.microsoft.windowsazure.storage.StorageException.translateException(StorageException.java:163)
   at 
 com.microsoft.windowsazure.storage.core.StorageRequest.materializeException(StorageRequest.java:306)
   at 
 com.microsoft.windowsazure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:229)
   at 
 com.microsoft.windowsazure.storage.blob.CloudBlob.startCopyFromBlob(CloudBlob.java:762)
   at 
 org.apache.hadoop.fs.azurenative.StorageInterfaceImpl$CloudBlobWrapperImpl.startCopyFromBlob(StorageInterfaceImpl.java:350)
   at 
 org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2439)
   ... 8 more
 2015-03-01 18:43:29,072 ERROR org.apache.hadoop.hbase.executor.EventHandler: 
 Caught throwable while processing event M_META_SERVER_SHUTDOWN
 java.io.IOException: failed log splitting for 
 workernode13.hbaseproddb4001.f5.internal.cloudapp.net,60020,1424845307901, 
 will retry
   at 
 org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.process(MetaServerShutdownHandler.java:71)
   at 
 org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:745)
 Caused by: org.apache.hadoop.fs.azure.AzureException: 
 com.microsoft.windowsazure.storage.StorageException: The server is busy.
   at

[jira] [Commented] (HADOOP-11693) Azure Storage FileSystem rename operations are throttled too aggressively to complete HBase WAL archiving.


[ 
https://issues.apache.org/jira/browse/HADOOP-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358734#comment-14358734
 ] 

Hudson commented on HADOOP-11693:
-

FAILURE: Integrated in Hadoop-Hdfs-trunk #2062 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2062/])
HADOOP-11693. Azure Storage FileSystem rename operations are throttled too 
aggressively to complete HBase WAL archiving. Contributed by Duo Xu. (cnauroth: 
rev 7a346bcb4fa5b56191ed00a39e72e51c9bdf1b56)
* 
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/MockStorageInterface.java
* hadoop-common-project/hadoop-common/CHANGES.txt
* 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
* 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterfaceImpl.java
* 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterface.java


 Azure Storage FileSystem rename operations are throttled too aggressively to 
 complete HBase WAL archiving.
 --

 Key: HADOOP-11693
 URL: https://issues.apache.org/jira/browse/HADOOP-11693
 Project: Hadoop Common
  Issue Type: Bug
  Components: tools
Reporter: Duo Xu
Assignee: Duo Xu
 Fix For: 2.7.0

 Attachments: HADOOP-11681.01.patch, HADOOP-11681.02.patch, 
 HADOOP-11681.03.patch


 One of our customers' production HBase clusters was periodically throttled by 
 Azure storage, when HBase was archiving old WALs. HMaster aborted the region 
 server and tried to restart it.
 However, since the cluster was still being throttled by Azure storage, the 
 upcoming distributed log splitting also failed. Sometimes hbase:meta table 
 was on this region server and finally showed offline, which cause the whole 
 cluster in bad state.
 {code}
 2015-03-01 18:36:45,623 ERROR org.apache.hadoop.hbase.master.HMaster: Region 
 server 
 workernode4.hbaseproddb4001.f5.internal.cloudapp.net,60020,1424845421044 
 reported a fatal error:
 ABORTING region server 
 workernode4.hbaseproddb4001.f5.internal.cloudapp.net,60020,1424845421044: IOE 
 in log roller
 Cause:
 org.apache.hadoop.fs.azure.AzureException: 
 com.microsoft.windowsazure.storage.StorageException: The server is busy.
   at 
 org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2446)
   at 
 org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2367)
   at 
 org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1960)
   at 
 org.apache.hadoop.hbase.util.FSUtils.renameAndSetModifyTime(FSUtils.java:1719)
   at 
 org.apache.hadoop.hbase.regionserver.wal.FSHLog.archiveLogFile(FSHLog.java:798)
   at 
 org.apache.hadoop.hbase.regionserver.wal.FSHLog.cleanOldLogs(FSHLog.java:656)
   at 
 org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:593)
   at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:97)
   at java.lang.Thread.run(Thread.java:745)
 Caused by: com.microsoft.windowsazure.storage.StorageException: The server is 
 busy.
   at 
 com.microsoft.windowsazure.storage.StorageException.translateException(StorageException.java:163)
   at 
 com.microsoft.windowsazure.storage.core.StorageRequest.materializeException(StorageRequest.java:306)
   at 
 com.microsoft.windowsazure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:229)
   at 
 com.microsoft.windowsazure.storage.blob.CloudBlob.startCopyFromBlob(CloudBlob.java:762)
   at 
 org.apache.hadoop.fs.azurenative.StorageInterfaceImpl$CloudBlobWrapperImpl.startCopyFromBlob(StorageInterfaceImpl.java:350)
   at 
 org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2439)
   ... 8 more
 2015-03-01 18:43:29,072 ERROR org.apache.hadoop.hbase.executor.EventHandler: 
 Caught throwable while processing event M_META_SERVER_SHUTDOWN
 java.io.IOException: failed log splitting for 
 workernode13.hbaseproddb4001.f5.internal.cloudapp.net,60020,1424845307901, 
 will retry
   at 
 org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.process(MetaServerShutdownHandler.java:71)
   at 
 org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:745)
 Caused by: org.apache.hadoop.fs.azure.AzureException: 
 com.microsoft.windowsazure.storage.StorageException: The server is busy.
   at

[jira] [Commented] (HADOOP-10027) *Compressor_deflateBytesDirect passes instance instead of jclass to GetStaticObjectField


[ 
https://issues.apache.org/jira/browse/HADOOP-10027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358735#comment-14358735
 ] 

Hudson commented on HADOOP-10027:
-

FAILURE: Integrated in Hadoop-Hdfs-trunk #2062 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2062/])
HADOOP-10027. *Compressor_deflateBytesDirect passes instance instead of jclass 
to GetStaticObjectField. Contributed by Hui Zheng. (cnauroth: rev 
ff83ae72318fe6c0f266a7bf08138bd2fdb51cbd)
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/zlib/TestZlibCompressorDecompressor.java
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/bzip2/TestBzip2CompressorDecompressor.java
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zlib/ZlibDecompressor.c
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/lz4/TestLz4CompressorDecompressor.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyCompressor.java
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/bzip2/Bzip2Compressor.c
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/snappy/TestSnappyCompressorDecompressor.java
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
* hadoop-common-project/hadoop-common/CHANGES.txt
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/Bzip2Decompressor.java
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/bzip2/Bzip2Decompressor.c
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/Bzip2Compressor.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/ZlibDecompressor.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/ZlibCompressor.java


 *Compressor_deflateBytesDirect passes instance instead of jclass to 
 GetStaticObjectField
 

 Key: HADOOP-10027
 URL: https://issues.apache.org/jira/browse/HADOOP-10027
 Project: Hadoop Common
  Issue Type: Bug
  Components: native
Reporter: Eric Abbott
Assignee: Hui Zheng
Priority: Minor
 Fix For: 2.8.0

 Attachments: HADOOP-10027.1.patch, HADOOP-10027.2.patch, 
 HADOOP-10027.3.patch, HADOOP-10027.4.patch, HADOOP-10027.5.patch


 http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c?view=markup
 This pattern appears in all the native compressors.
 // Get members of ZlibCompressor
 jobject clazz = (*env)-GetStaticObjectField(env, this,
  ZlibCompressor_clazz);
 The 2nd argument to GetStaticObjectField is supposed to be a jclass, not a 
 jobject. Adding the JVM param -Xcheck:jni will cause FATAL ERROR in native 
 method: JNI received a class argument that is not a class and a core dump 
 such as the following.
 (gdb) 
 #0 0x7f02e4aef8a5 in raise () from /lib64/libc.so.6
 #1 0x7f02e4af1085 in abort () from /lib64/libc.so.6
 #2 0x7f02e45bd727 in os::abort(bool) () from 
 /opt/jdk1.6.0_31/jre/lib/amd64/server/libjvm.so
 #3 0x7f02e43cec63 in jniCheck::validate_class(JavaThread*, _jclass*, 
 bool) () from /opt/jdk1.6.0_31/jre/lib/amd64/server/libjvm.so
 #4 0x7f02e43ea669 in checked_jni_GetStaticObjectField () from 
 /opt/jdk1.6.0_31/jre/lib/amd64/server/libjvm.so
 #5 0x7f02d38eaf79 in 
 Java_org_apache_hadoop_io_compress_zlib_ZlibCompressor_deflateBytesDirect () 
 from /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
 In addition, that clazz object is only used for synchronization. In the case 
 of the native method _deflateBytesDirect, the result is a class wide lock 
 used to access the instance field

[jira] [Commented] (HADOOP-11703) git should ignore .DS_Store files on Mac OS X


[ 
https://issues.apache.org/jira/browse/HADOOP-11703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358733#comment-14358733
 ] 

Hudson commented on HADOOP-11703:
-

FAILURE: Integrated in Hadoop-Hdfs-trunk #2062 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2062/])
HADOOP-11703. git should ignore .DS_Store files on Mac OS X (Abin Shahab via 
aw) (aw: rev 344d7cb6dcd3b1752c82b62271799be9e4e10416)
* hadoop-common-project/hadoop-common/CHANGES.txt
* .gitignore


 git should ignore .DS_Store files on Mac OS X
 -

 Key: HADOOP-11703
 URL: https://issues.apache.org/jira/browse/HADOOP-11703
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Abin Shahab
Assignee: Abin Shahab
 Fix For: 3.0.0

 Attachments: HADOOP-11703.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11703) git should ignore .DS_Store files on Mac OS X


[ 
https://issues.apache.org/jira/browse/HADOOP-11703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358751#comment-14358751
 ] 

Hudson commented on HADOOP-11703:
-

FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #121 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/121/])
HADOOP-11703. git should ignore .DS_Store files on Mac OS X (Abin Shahab via 
aw) (aw: rev 344d7cb6dcd3b1752c82b62271799be9e4e10416)
* hadoop-common-project/hadoop-common/CHANGES.txt
* .gitignore


 git should ignore .DS_Store files on Mac OS X
 -

 Key: HADOOP-11703
 URL: https://issues.apache.org/jira/browse/HADOOP-11703
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Abin Shahab
Assignee: Abin Shahab
 Fix For: 3.0.0

 Attachments: HADOOP-11703.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-10027) *Compressor_deflateBytesDirect passes instance instead of jclass to GetStaticObjectField


[ 
https://issues.apache.org/jira/browse/HADOOP-10027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358753#comment-14358753
 ] 

Hudson commented on HADOOP-10027:
-

FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #121 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/121/])
HADOOP-10027. *Compressor_deflateBytesDirect passes instance instead of jclass 
to GetStaticObjectField. Contributed by Hui Zheng. (cnauroth: rev 
ff83ae72318fe6c0f266a7bf08138bd2fdb51cbd)
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/snappy/TestSnappyCompressorDecompressor.java
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/bzip2/Bzip2Decompressor.c
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/Bzip2Decompressor.java
* hadoop-common-project/hadoop-common/CHANGES.txt
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/bzip2/TestBzip2CompressorDecompressor.java
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/zlib/TestZlibCompressorDecompressor.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/ZlibDecompressor.java
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/bzip2/Bzip2Compressor.c
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zlib/ZlibDecompressor.c
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/Bzip2Compressor.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyCompressor.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.java
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/ZlibCompressor.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/lz4/TestLz4CompressorDecompressor.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java


 *Compressor_deflateBytesDirect passes instance instead of jclass to 
 GetStaticObjectField
 

 Key: HADOOP-10027
 URL: https://issues.apache.org/jira/browse/HADOOP-10027
 Project: Hadoop Common
  Issue Type: Bug
  Components: native
Reporter: Eric Abbott
Assignee: Hui Zheng
Priority: Minor
 Fix For: 2.8.0

 Attachments: HADOOP-10027.1.patch, HADOOP-10027.2.patch, 
 HADOOP-10027.3.patch, HADOOP-10027.4.patch, HADOOP-10027.5.patch


 http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c?view=markup
 This pattern appears in all the native compressors.
 // Get members of ZlibCompressor
 jobject clazz = (*env)-GetStaticObjectField(env, this,
  ZlibCompressor_clazz);
 The 2nd argument to GetStaticObjectField is supposed to be a jclass, not a 
 jobject. Adding the JVM param -Xcheck:jni will cause FATAL ERROR in native 
 method: JNI received a class argument that is not a class and a core dump 
 such as the following.
 (gdb) 
 #0 0x7f02e4aef8a5 in raise () from /lib64/libc.so.6
 #1 0x7f02e4af1085 in abort () from /lib64/libc.so.6
 #2 0x7f02e45bd727 in os::abort(bool) () from 
 /opt/jdk1.6.0_31/jre/lib/amd64/server/libjvm.so
 #3 0x7f02e43cec63 in jniCheck::validate_class(JavaThread*, _jclass*, 
 bool) () from /opt/jdk1.6.0_31/jre/lib/amd64/server/libjvm.so
 #4 0x7f02e43ea669 in checked_jni_GetStaticObjectField () from 
 /opt/jdk1.6.0_31/jre/lib/amd64/server/libjvm.so
 #5 0x7f02d38eaf79 in 
 Java_org_apache_hadoop_io_compress_zlib_ZlibCompressor_deflateBytesDirect () 
 from /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
 In addition, that clazz object is only used for synchronization. In the case 
 of the native method _deflateBytesDirect, the result is a class wide lock 
 used to access the instance

[jira] [Updated] (HADOOP-8320) FileSystem#checkPath and AbstractFileSystem#checkPath should share code


 [ 
https://issues.apache.org/jira/browse/HADOOP-8320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vijay Bhat updated HADOOP-8320:
---
Release Note: Moved the checkPath logic from the AbstractFileSystem and 
FileSystem classes to a checkFileSystemPath method in the Path class. The 
AbstracFileSystem and FileSystem checkPath method will call the common code in 
checkFileSystemPath (adapter pattern). This is because the two classes aren't 
derived from any common ancestor.
  Status: Patch Available  (was: In Progress)

 FileSystem#checkPath and AbstractFileSystem#checkPath should share code
 ---

 Key: HADOOP-8320
 URL: https://issues.apache.org/jira/browse/HADOOP-8320
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Affects Versions: 2.0.0-alpha
Reporter: Aaron T. Myers
Assignee: Vijay Bhat
Priority: Minor
 Attachments: HADOOP-8320.001.patch


 Per the discussion on HADOOP-8310, these two methods can be refactored to 
 share some code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-8320) FileSystem#checkPath and AbstractFileSystem#checkPath should share code


[ 
https://issues.apache.org/jira/browse/HADOOP-8320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358759#comment-14358759
 ] 

Hadoop QA commented on HADOOP-8320:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12704050/HADOOP-8320.001.patch
  against trunk revision ff83ae7.

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/5924//console

This message is automatically generated.

 FileSystem#checkPath and AbstractFileSystem#checkPath should share code
 ---

 Key: HADOOP-8320
 URL: https://issues.apache.org/jira/browse/HADOOP-8320
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Affects Versions: 2.0.0-alpha
Reporter: Aaron T. Myers
Assignee: Vijay Bhat
Priority: Minor
 Attachments: HADOOP-8320.001.patch


 Per the discussion on HADOOP-8310, these two methods can be refactored to 
 share some code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11711) Provide a default value for AES/CTR/NoPadding CryptoCodec classes


[ 
https://issues.apache.org/jira/browse/HADOOP-11711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359879#comment-14359879
 ] 

Yi Liu commented on HADOOP-11711:
-

+1, Thanks Andrew!

 Provide a default value for AES/CTR/NoPadding CryptoCodec classes
 -

 Key: HADOOP-11711
 URL: https://issues.apache.org/jira/browse/HADOOP-11711
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.6.0
Reporter: Andrew Wang
Assignee: Andrew Wang
Priority: Minor
 Attachments: hadoop-11711.001.patch, hadoop-11711.002.patch


 Users can configure the desired class to use for a given codec via a property 
 like {{hadoop.security.crypto.codec.classes.aes.ctr.nopadding}}. However, 
 even though we provide a default value for this codec in 
 {{core-default.xml}}, this default is not also done in the code.
 As a result, client deployments that do not include {{core-default.xml}} 
 cannot resolve any codecs, and get an NPE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11710) Make CryptoOutputStream behave like DFSOutputStream wrt synchronization


 [ 
https://issues.apache.org/jira/browse/HADOOP-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yi Liu updated HADOOP-11710:

  Resolution: Fixed
   Fix Version/s: 2.7.0
Target Version/s: 2.7.0
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Committed to trunk and branch-2.

 Make CryptoOutputStream behave like DFSOutputStream wrt synchronization
 ---

 Key: HADOOP-11710
 URL: https://issues.apache.org/jira/browse/HADOOP-11710
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs
Affects Versions: 2.6.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Priority: Critical
 Fix For: 2.7.0

 Attachments: HADOOP-11710.1.patch.txt, HADOOP-11710.2.patch.txt, 
 HADOOP-11710.3.patch.txt


 per discussion on parent, as an intermediate solution make CryptoOutputStream 
 behave like DFSOutputStream



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HADOOP-11710) Make CryptoOutputStream behave like DFSOutputStream wrt synchronization


[ 
https://issues.apache.org/jira/browse/HADOOP-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359876#comment-14359876
 ] 

Yi Liu edited comment on HADOOP-11710 at 3/13/15 3:25 AM:
--

Committed to trunk and branch-2.

The test failure is not related.


was (Author: hitliuyi):
Committed to trunk and branch-2.

 Make CryptoOutputStream behave like DFSOutputStream wrt synchronization
 ---

 Key: HADOOP-11710
 URL: https://issues.apache.org/jira/browse/HADOOP-11710
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs
Affects Versions: 2.6.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Priority: Critical
 Fix For: 2.7.0

 Attachments: HADOOP-11710.1.patch.txt, HADOOP-11710.2.patch.txt, 
 HADOOP-11710.3.patch.txt


 per discussion on parent, as an intermediate solution make CryptoOutputStream 
 behave like DFSOutputStream



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11676) Add API to NetworkTopology for getting all racks


 [ 
https://issues.apache.org/jira/browse/HADOOP-11676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Walter Su updated HADOOP-11676:
---
Issue Type: Improvement  (was: Sub-task)
Parent: (was: HADOOP-11264)

 Add API to NetworkTopology for getting all racks
 

 Key: HADOOP-11676
 URL: https://issues.apache.org/jira/browse/HADOOP-11676
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Walter Su
Assignee: Walter Su
 Attachments: HADOOP-11676.002.patch, HADOOP-11676.patch


 The existing two NetworkTopology.chooseRandom(..) API support choosing node 
 from scope and choosing outside scope. BlockPlacementPolicyDefault class use 
 these two API to choose node from one rack or choose outside one rack.
 We want to implement a new placement policy called 
 BlockPlacementPolicyFaultTolerant which tries its best to place replicas to 
 most racks. To achieve this, We need to know how many replicas each rack has. 
 And first, we need to get all racks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11707) Add factory to create raw erasure coder


 [ 
https://issues.apache.org/jira/browse/HADOOP-11707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kai Zheng updated HADOOP-11707:
---
Attachment: HADOOP-11707-v1.patch

Uploaded the patch. Ready for review. 

 Add factory to create raw erasure coder
 ---

 Key: HADOOP-11707
 URL: https://issues.apache.org/jira/browse/HADOOP-11707
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: io
Reporter: Kai Zheng
Assignee: Kai Zheng
 Fix For: HDFS-7285

 Attachments: HADOOP-11707-v1.patch


 We have {{RawErasureEncoder}} and {{RawErasureDecoder}} interface separately, 
 which simplifies the implementation of raw coders. This would require to 
 configure raw encoder and decoder respectively for a {{ErasureCoder}}, which 
 isn't convenient. To simplify the configuration, we would have coder factory 
 to group encoder and decoder together so only a factory class needs to be 
 configured.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-10420) Add support to Swift-FS to support tempAuth

2015-03-12 Thread Jim VanOosten (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-10420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jim VanOosten updated HADOOP-10420:
---
Attachment: HADOOP-10420-006.patch

Fixed a problem with a recursive call on authentication failures (http status 
code = 401). 

New error output: 

15/03/13 01:26:37 WARN httpclient.HttpMethodDirector: Unable to respond to any 
of these challenges: {swift=Swift realm=unknown}
15/03/13 01:26:37 ERROR http.SwiftRestClient: Athentication error
ls: GET https://dal05.objectstorage.service.networklayer.com/auth/v1.0 failed 
on exception: Authentication Failure: Auth as tenant 'null' user 
'IBMO999-999:j...@us.ibm.com' with key of length 64  auth 
https://dal05.objectstorage.service.networklayer.com/auth/v1.0 = 401 : 
htmlh1Unauthorized/h1pThis server could not verify that you are 
authorized to access the document you requested./p/html

 Add support to Swift-FS to support tempAuth
 ---

 Key: HADOOP-10420
 URL: https://issues.apache.org/jira/browse/HADOOP-10420
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs, fs/swift, tools
Affects Versions: 2.3.0
Reporter: Jinghui Wang
 Attachments: HADOOP-10420-002.patch, HADOOP-10420-003.patch, 
 HADOOP-10420-004.patch, HADOOP-10420-005.patch, HADOOP-10420-006.patch, 
 HADOOP-10420.patch


 Currently, hadoop-openstack Swift FS supports keystone authentication. The 
 attached patch adds support for tempAuth. Users will be able to configure 
 which authentication to use.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-9477) Add posixGroups support for LDAP groups mapping service

2015-03-12 Thread Dapeng Sun (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dapeng Sun updated HADOOP-9477:
---
Release Note: Add posixGroups support for LDAP groups mapping service. The 
change in LDAPGroupMapping is compatible with previous scenario. In LDAP, the 
group mapping between {{posixAccount}} and {{posixGroup}} is different from the 
general LDAPGroupMapping, one of the differences is the {{memberUid}} will be 
used to mapping {{posixAccount}} and {{posixGroup}}. The feature will handle 
the mapping in internal when configuration 
{{hadoop.security.group.mapping.ldap.search.filter.user}} is set as 
posixAccount and {{hadoop.security.group.mapping.ldap.search.filter.group}} 
is posixGroup.

 Add posixGroups support for LDAP groups mapping service
 ---

 Key: HADOOP-9477
 URL: https://issues.apache.org/jira/browse/HADOOP-9477
 Project: Hadoop Common
  Issue Type: New Feature
Affects Versions: 2.0.4-alpha
Reporter: Kai Zheng
Assignee: Dapeng Sun
 Fix For: 2.8.0

 Attachments: HADOOP-9477.003.patch, HADOOP-9477.004.patch, 
 HADOOP-9477.005.patch, HADOOP-9477.006.patch, HADOOP-9477.007.patch, 
 HADOOP-9477.008.patch, HADOOP-9477.009.patch, HADOOP-9477.patch, 
 HADOOP-9477.patch

   Original Estimate: 168h
  Remaining Estimate: 168h

 It would be nice to support posixGroups for LdapGroupsMapping service. Below 
 is from current description for the provider:
 hadoop.security.group.mapping.ldap.search.filter.group:
 An additional filter to use when searching for LDAP groups. This should be
 changed when resolving groups against a non-Active Directory installation.
 posixGroups are currently not a supported group class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11711) Provide a default value for AES/CTR/NoPadding CryptoCodec classes


[ 
https://issues.apache.org/jira/browse/HADOOP-11711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359839#comment-14359839
 ] 

Yi Liu commented on HADOOP-11711:
-

Thanks [~andrew.wang] for the patch, it looks good to me, +1 pending Jenkins.
I find a really small nit in the test, it would be better if you could address:
{code}
public static final String
  HADOOP_SECURITY_CRYPTO_CODEC_CLASSES_AES_CTR_NOPADDING_KEY =
  HADOOP_SECURITY_CRYPTO_CODEC_CLASSES_KEY_PREFIX
  + CipherSuite.AES_CTR_NOPADDING.getConfigSuffix();
{code}
In {{CommonConfigurationKeysPublic.java}}, 
{{HADOOP_SECURITY_CRYPTO_CODEC_CLASSES_AES_CTR_NOPADDING_KEY}}  is defined, we 
could use it in TestCryptoStreamsWithJceAesCtrCryptoCodec.java and 
TestCryptoStreamsWithOpensslAesCtrCryptoCodec.java instead of constructing the 
string again.

 Provide a default value for AES/CTR/NoPadding CryptoCodec classes
 -

 Key: HADOOP-11711
 URL: https://issues.apache.org/jira/browse/HADOOP-11711
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.6.0
Reporter: Andrew Wang
Assignee: Andrew Wang
Priority: Minor
 Attachments: hadoop-11711.001.patch


 Users can configure the desired class to use for a given codec via a property 
 like {{hadoop.security.crypto.codec.classes.aes.ctr.nopadding}}. However, 
 even though we provide a default value for this codec in 
 {{core-default.xml}}, this default is not also done in the code.
 As a result, client deployments that do not include {{core-default.xml}} 
 cannot resolve any codecs, and get an NPE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11710) Make CryptoOutputStream behave like DFSOutputStream wrt synchronization


[ 
https://issues.apache.org/jira/browse/HADOOP-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359887#comment-14359887
 ] 

Hudson commented on HADOOP-11710:
-

FAILURE: Integrated in Hadoop-trunk-Commit #7314 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/7314/])
HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt 
synchronization. (Sean Busbey via yliu) (yliu: rev 
a85291003cf3e3fd79b6addcf59d4f43dc72d356)
* hadoop-common-project/hadoop-common/CHANGES.txt
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoOutputStream.java


 Make CryptoOutputStream behave like DFSOutputStream wrt synchronization
 ---

 Key: HADOOP-11710
 URL: https://issues.apache.org/jira/browse/HADOOP-11710
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs
Affects Versions: 2.6.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Priority: Critical
 Fix For: 2.7.0

 Attachments: HADOOP-11710.1.patch.txt, HADOOP-11710.2.patch.txt, 
 HADOOP-11710.3.patch.txt


 per discussion on parent, as an intermediate solution make CryptoOutputStream 
 behave like DFSOutputStream



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11711) Provide a default value for AES/CTR/NoPadding CryptoCodec classes


[ 
https://issues.apache.org/jira/browse/HADOOP-11711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359938#comment-14359938
 ] 

Hudson commented on HADOOP-11711:
-

FAILURE: Integrated in Hadoop-trunk-Commit #7315 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/7315/])
HADOOP-11711. Provide a default value for AES/CTR/NoPadding CryptoCodec 
classes. (wang: rev 387f271c81f7b3bf53bddc5368d5f4486530c2e1)
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsWithJceAesCtrCryptoCodec.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsForLocalFS.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsWithOpensslAesCtrCryptoCodec.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoCodec.java


 Provide a default value for AES/CTR/NoPadding CryptoCodec classes
 -

 Key: HADOOP-11711
 URL: https://issues.apache.org/jira/browse/HADOOP-11711
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.6.0
Reporter: Andrew Wang
Assignee: Andrew Wang
Priority: Minor
 Fix For: 2.8.0

 Attachments: hadoop-11711.001.patch, hadoop-11711.002.patch


 Users can configure the desired class to use for a given codec via a property 
 like {{hadoop.security.crypto.codec.classes.aes.ctr.nopadding}}. However, 
 even though we provide a default value for this codec in 
 {{core-default.xml}}, this default is not also done in the code.
 As a result, client deployments that do not include {{core-default.xml}} 
 cannot resolve any codecs, and get an NPE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-10420) Add support to Swift-FS to support tempAuth

2015-03-12 Thread Jim VanOosten (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-10420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359981#comment-14359981
 ] 

Jim VanOosten commented on HADOOP-10420:


Steve, 

For the public endpoint for testing with SoftLayer's object storage.  Do you 
have a rough idea on how much space you'll need and how long the endpoint needs 
to be active?

 Add support to Swift-FS to support tempAuth
 ---

 Key: HADOOP-10420
 URL: https://issues.apache.org/jira/browse/HADOOP-10420
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs, fs/swift, tools
Affects Versions: 2.3.0
Reporter: Jinghui Wang
 Attachments: HADOOP-10420-002.patch, HADOOP-10420-003.patch, 
 HADOOP-10420-004.patch, HADOOP-10420-005.patch, HADOOP-10420-006.patch, 
 HADOOP-10420.patch


 Currently, hadoop-openstack Swift FS supports keystone authentication. The 
 attached patch adds support for tempAuth. Users will be able to configure 
 which authentication to use.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11710) Make CryptoOutputStream behave like DFSOutputStream wrt synchronization


 [ 
https://issues.apache.org/jira/browse/HADOOP-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HADOOP-11710:
-
Attachment: HADOOP-11710.3.patch.txt

patch v3, uses {{try ... finally}} to always set closed on first call to 
close().

locally passed all the cypto tests:

{code}

---
 T E S T S
---
Running org.apache.hadoop.crypto.key.TestKeyProviderCryptoExtension
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.753 sec - in 
org.apache.hadoop.crypto.key.TestKeyProviderCryptoExtension
Running org.apache.hadoop.crypto.TestCryptoCodec
Tests run: 3, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 11.974 sec - in 
org.apache.hadoop.crypto.TestCryptoCodec
Running org.apache.hadoop.crypto.TestCryptoStreams
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.953 sec - 
in org.apache.hadoop.crypto.TestCryptoStreams
Running org.apache.hadoop.crypto.TestCryptoStreamsForLocalFS
Tests run: 14, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 15.692 sec - 
in org.apache.hadoop.crypto.TestCryptoStreamsForLocalFS
Running org.apache.hadoop.crypto.TestCryptoStreamsNormal
Tests run: 14, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 6.46 sec - in 
org.apache.hadoop.crypto.TestCryptoStreamsNormal
Running org.apache.hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.867 sec - 
in org.apache.hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec

Results :

Tests run: 61, Failures: 0, Errors: 0, Skipped: 15




---
 T E S T S
---
Running org.apache.hadoop.cli.TestCryptoAdminCLI
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.867 sec - in 
org.apache.hadoop.cli.TestCryptoAdminCLI
Running org.apache.hadoop.hdfs.crypto.TestHdfsCryptoStreams
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 37.574 sec - 
in org.apache.hadoop.hdfs.crypto.TestHdfsCryptoStreams

Results :

Tests run: 15, Failures: 0, Errors: 0, Skipped: 0


{code}

 Make CryptoOutputStream behave like DFSOutputStream wrt synchronization
 ---

 Key: HADOOP-11710
 URL: https://issues.apache.org/jira/browse/HADOOP-11710
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs
Affects Versions: 2.6.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Priority: Critical
 Attachments: HADOOP-11710.1.patch.txt, HADOOP-11710.2.patch.txt, 
 HADOOP-11710.3.patch.txt


 per discussion on parent, as an intermediate solution make CryptoOutputStream 
 behave like DFSOutputStream



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11649) Allow to configure multiple erasure codecs


 [ 
https://issues.apache.org/jira/browse/HADOOP-11649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kai Zheng updated HADOOP-11649:
---
Summary: Allow to configure multiple erasure codecs  (was: Allow to 
configure erasure codec in core-site)

 Allow to configure multiple erasure codecs
 --

 Key: HADOOP-11649
 URL: https://issues.apache.org/jira/browse/HADOOP-11649
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: io
Reporter: Kai Zheng
Assignee: Kai Zheng
 Attachments: HADOOP-11649-v1.patch


 This is to allow to configure erasure codec and coder in core-site 
 configuration file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11676) Add API to NetworkTopology for getting all racks


 [ 
https://issues.apache.org/jira/browse/HADOOP-11676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Walter Su updated HADOOP-11676:
---
Attachment: HADOOP-11676.002.patch

 Add API to NetworkTopology for getting all racks
 

 Key: HADOOP-11676
 URL: https://issues.apache.org/jira/browse/HADOOP-11676
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Walter Su
Assignee: Walter Su
 Attachments: HADOOP-11676.002.patch, HADOOP-11676.patch


 The existing two NetworkTopology.chooseRandom(..) API support choosing node 
 from scope and choosing outside scope. BlockPlacementPolicyDefault class use 
 these two API to choose node from one rack or choose outside one rack.
 We want to implement a new placement policy called 
 BlockPlacementPolicyFaultTolerant which tries its best to place replicas to 
 most racks. To achieve this, We need to know how many replicas each rack has. 
 And first, we need to get all racks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11676) Add API to NetworkTopology for getting all racks


 [ 
https://issues.apache.org/jira/browse/HADOOP-11676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Walter Su updated HADOOP-11676:
---
Status: Patch Available  (was: Open)

 Add API to NetworkTopology for getting all racks
 

 Key: HADOOP-11676
 URL: https://issues.apache.org/jira/browse/HADOOP-11676
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Walter Su
Assignee: Walter Su
 Attachments: HADOOP-11676.002.patch, HADOOP-11676.patch


 The existing two NetworkTopology.chooseRandom(..) API support choosing node 
 from scope and choosing outside scope. BlockPlacementPolicyDefault class use 
 these two API to choose node from one rack or choose outside one rack.
 We want to implement a new placement policy called 
 BlockPlacementPolicyFaultTolerant which tries its best to place replicas to 
 most racks. To achieve this, We need to know how many replicas each rack has. 
 And first, we need to get all racks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11711) Provide a default value for AES/CTR/NoPadding CryptoCodec classes

2015-03-12 Thread Andrew Wang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-11711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang updated HADOOP-11711:
-
   Resolution: Fixed
Fix Version/s: 2.8.0
   Status: Resolved  (was: Patch Available)

Committed to trunk and branch-2, thanks again Yi for reviewing!

 Provide a default value for AES/CTR/NoPadding CryptoCodec classes
 -

 Key: HADOOP-11711
 URL: https://issues.apache.org/jira/browse/HADOOP-11711
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.6.0
Reporter: Andrew Wang
Assignee: Andrew Wang
Priority: Minor
 Fix For: 2.8.0

 Attachments: hadoop-11711.001.patch, hadoop-11711.002.patch


 Users can configure the desired class to use for a given codec via a property 
 like {{hadoop.security.crypto.codec.classes.aes.ctr.nopadding}}. However, 
 even though we provide a default value for this codec in 
 {{core-default.xml}}, this default is not also done in the code.
 As a result, client deployments that do not include {{core-default.xml}} 
 cannot resolve any codecs, and get an NPE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11710) Make CryptoOutputStream behave like DFSOutputStream wrt synchronization


 [ 
https://issues.apache.org/jira/browse/HADOOP-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HADOOP-11710:
-
Status: Open  (was: Patch Available)

 Make CryptoOutputStream behave like DFSOutputStream wrt synchronization
 ---

 Key: HADOOP-11710
 URL: https://issues.apache.org/jira/browse/HADOOP-11710
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs
Affects Versions: 2.6.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Priority: Critical
 Attachments: HADOOP-11710.1.patch.txt, HADOOP-11710.2.patch.txt


 per discussion on parent, as an intermediate solution make CryptoOutputStream 
 behave like DFSOutputStream



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11676) Add API to NetworkTopology for getting all racks


[ 
https://issues.apache.org/jira/browse/HADOOP-11676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359874#comment-14359874
 ] 

Walter Su commented on HADOOP-11676:


Thanks [~zhz] for thorough review. Good point using ArrayList instead of a 
HashSet. Also because racks.add is more often, and racks.remove is less often, 
there is no performance issue using ArrayList. And sorry for lack of code 
comments.
Thanks [~aw], we should deal with HDFS expansion very carefully. Thanks [~zhz] 
again for answering the question.

 Add API to NetworkTopology for getting all racks
 

 Key: HADOOP-11676
 URL: https://issues.apache.org/jira/browse/HADOOP-11676
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Walter Su
Assignee: Walter Su
 Attachments: HADOOP-11676.patch


 The existing two NetworkTopology.chooseRandom(..) API support choosing node 
 from scope and choosing outside scope. BlockPlacementPolicyDefault class use 
 these two API to choose node from one rack or choose outside one rack.
 We want to implement a new placement policy called 
 BlockPlacementPolicyFaultTolerant which tries its best to place replicas to 
 most racks. To achieve this, We need to know how many replicas each rack has. 
 And first, we need to get all racks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-9477) Add posixGroups support for LDAP groups mapping service

2015-03-12 Thread Dapeng Sun (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359816#comment-14359816
 ] 

Dapeng Sun commented on HADOOP-9477:


Thanks [~yzhangal] for helping commit it. Also thanks Kai Zheng, Daryn Sharp 
and Charles Lamb for your help.

I will update the {{release note}}, thanks [~yzhangal] again for your reminder.

 Add posixGroups support for LDAP groups mapping service
 ---

 Key: HADOOP-9477
 URL: https://issues.apache.org/jira/browse/HADOOP-9477
 Project: Hadoop Common
  Issue Type: New Feature
Affects Versions: 2.0.4-alpha
Reporter: Kai Zheng
Assignee: Dapeng Sun
 Fix For: 2.8.0

 Attachments: HADOOP-9477.003.patch, HADOOP-9477.004.patch, 
 HADOOP-9477.005.patch, HADOOP-9477.006.patch, HADOOP-9477.007.patch, 
 HADOOP-9477.008.patch, HADOOP-9477.009.patch, HADOOP-9477.patch, 
 HADOOP-9477.patch

   Original Estimate: 168h
  Remaining Estimate: 168h

 It would be nice to support posixGroups for LdapGroupsMapping service. Below 
 is from current description for the provider:
 hadoop.security.group.mapping.ldap.search.filter.group:
 An additional filter to use when searching for LDAP groups. This should be
 changed when resolving groups against a non-Active Directory installation.
 posixGroups are currently not a supported group class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11710) Make CryptoOutputStream behave like DFSOutputStream wrt synchronization


[ 
https://issues.apache.org/jira/browse/HADOOP-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359815#comment-14359815
 ] 

Yi Liu commented on HADOOP-11710:
-

+1 pending Jenkins.

 Make CryptoOutputStream behave like DFSOutputStream wrt synchronization
 ---

 Key: HADOOP-11710
 URL: https://issues.apache.org/jira/browse/HADOOP-11710
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs
Affects Versions: 2.6.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Priority: Critical
 Attachments: HADOOP-11710.1.patch.txt, HADOOP-11710.2.patch.txt, 
 HADOOP-11710.3.patch.txt


 per discussion on parent, as an intermediate solution make CryptoOutputStream 
 behave like DFSOutputStream



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11704) ProxyUsers expects ipaddress due to MachineList but callers send in remote host

2015-03-12 Thread Anubhav Dhoot (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359963#comment-14359963
 ] 

Anubhav Dhoot commented on HADOOP-11704:


I think the logic is sound, as it normalize the configured 
ipaddress/hostname/cidr to ipaddress and then expects ipaddress to be passed 
in. Otherwise supporting both hostname and ipaddress to be passed in can be 
challenging. I can add a warning if the input is not an ipaddress to catch 
existing bugs like this. 

 ProxyUsers expects ipaddress due to MachineList but callers send in remote 
 host 
 

 Key: HADOOP-11704
 URL: https://issues.apache.org/jira/browse/HADOOP-11704
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Anubhav Dhoot
Assignee: Anubhav Dhoot
 Attachments: HADOOP-11704.001.patch


 DelegationTokenAuthenticationHandler and DelegationTokenAuthenticationFilter 
 are using ServletRequest#getRemoteHost which can send an address if possible. 
 It should use getRemoteAddr instead



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-10420) Add support to Swift-FS to support tempAuth

2015-03-12 Thread Jim VanOosten (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-10420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jim VanOosten updated HADOOP-10420:
---
Attachment: HADOOP-10420-007.patch

Fixed a typo in the error message (Athentication - Authentication).

 Add support to Swift-FS to support tempAuth
 ---

 Key: HADOOP-10420
 URL: https://issues.apache.org/jira/browse/HADOOP-10420
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs, fs/swift, tools
Affects Versions: 2.3.0
Reporter: Jinghui Wang
 Attachments: HADOOP-10420-002.patch, HADOOP-10420-003.patch, 
 HADOOP-10420-004.patch, HADOOP-10420-005.patch, HADOOP-10420-006.patch, 
 HADOOP-10420-007.patch, HADOOP-10420.patch


 Currently, hadoop-openstack Swift FS supports keystone authentication. The 
 attached patch adds support for tempAuth. Users will be able to configure 
 which authentication to use.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11683) Need a plugin API to translate long principal names to local OS user names arbitrarily


[ 
https://issues.apache.org/jira/browse/HADOOP-11683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358809#comment-14358809
 ] 

Allen Wittenauer commented on HADOOP-11683:
---

Just a pre-emptive comment: :)

Be aware that HadoopKerberosName is now exposed to users in trunk.  We should 
make sure that the solution here also works there. 

 Need a plugin API to translate long principal names to local OS user names 
 arbitrarily
 --

 Key: HADOOP-11683
 URL: https://issues.apache.org/jira/browse/HADOOP-11683
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Reporter: Sunny Cheung
Assignee: Sunny Cheung

 We need a plugin API to translate long principal names (e.g. 
 john@example.com) to local OS user names (e.g. user123456) arbitrarily.
 For some organizations the name translation is straightforward (e.g. 
 john@example.com to john_doe), and the hadoop.security.auth_to_local 
 configurable mapping is sufficient to resolve this (see HADOOP-6526). 
 However, in some other cases the name translation is arbitrary and cannot be 
 generalized by a set of translation rules easily.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-7947) Validate XMLs if a relevant tool is available, when using scripts

2015-03-12 Thread Kengo Seki (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-7947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358804#comment-14358804
 ] 

Kengo Seki commented on HADOOP-7947:


I have another idea. We add a new constructor like 'public 
GenericOptionsParser(String[] args, boolean flag)' so as to utilize 
GenericOptionsParser, and make it avoid calling  processGeneralOptions() in 
parseGeneralOptions() if the second argument is false. But I'm not sure this 
idea is good or bad...

 Validate XMLs if a relevant tool is available, when using scripts
 -

 Key: HADOOP-7947
 URL: https://issues.apache.org/jira/browse/HADOOP-7947
 Project: Hadoop Common
  Issue Type: Wish
  Components: scripts
Affects Versions: 2.7.0
Reporter: Harsh J
Assignee: Kengo Seki
  Labels: newbie
 Attachments: HADOOP-7947.001.patch, HADOOP-7947.002.patch, 
 HADOOP-7947.003.patch


 Given that we are locked down to using only XML for configuration and most of 
 the administrators need to manage it by themselves (unless a tool that 
 manages for you is used), it would be good to also validate the provided 
 config XML (*-site.xml) files with a tool like {{xmllint}} or maybe Xerces 
 somehow, when running a command or (at least) when starting up daemons.
 We should use this only if a relevant tool is available, and optionally be 
 silent if the env. requests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11709) Time.NANOSECONDS_PER_MILLISECOND - use class level final constant instead of method variable

2015-03-12 Thread Tsuyoshi Ozawa (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-11709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsuyoshi Ozawa updated HADOOP-11709:

Labels: beginner newbie  (was: beginner)

 Time.NANOSECONDS_PER_MILLISECOND - use class level final constant instead of 
 method variable 
 -

 Key: HADOOP-11709
 URL: https://issues.apache.org/jira/browse/HADOOP-11709
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Ajith S
Assignee: Ajith S
Priority: Trivial
  Labels: beginner, newbie

 NANOSECONDS_PER_MILLISECOND constant can be moved to class level instead of 
 creating it in each method call.
 {code}
 org.apache.hadoop.util.Time.java
  public static long monotonicNow() {
 final long NANOSECONDS_PER_MILLISECOND = 100;
 return System.nanoTime() / NANOSECONDS_PER_MILLISECOND;
   }
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-10027) *Compressor_deflateBytesDirect passes instance instead of jclass to GetStaticObjectField


[ 
https://issues.apache.org/jira/browse/HADOOP-10027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358884#comment-14358884
 ] 

Hudson commented on HADOOP-10027:
-

FAILURE: Integrated in Hadoop-Mapreduce-trunk #2080 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2080/])
HADOOP-10027. *Compressor_deflateBytesDirect passes instance instead of jclass 
to GetStaticObjectField. Contributed by Hui Zheng. (cnauroth: rev 
ff83ae72318fe6c0f266a7bf08138bd2fdb51cbd)
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/snappy/TestSnappyCompressorDecompressor.java
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/ZlibDecompressor.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/zlib/TestZlibCompressorDecompressor.java
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zlib/ZlibDecompressor.c
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/bzip2/TestBzip2CompressorDecompressor.java
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/bzip2/Bzip2Compressor.c
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/Bzip2Decompressor.java
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
* hadoop-common-project/hadoop-common/CHANGES.txt
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.java
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/bzip2/Bzip2Decompressor.c
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyCompressor.java
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/lz4/TestLz4CompressorDecompressor.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/Bzip2Compressor.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/ZlibCompressor.java


 *Compressor_deflateBytesDirect passes instance instead of jclass to 
 GetStaticObjectField
 

 Key: HADOOP-10027
 URL: https://issues.apache.org/jira/browse/HADOOP-10027
 Project: Hadoop Common
  Issue Type: Bug
  Components: native
Reporter: Eric Abbott
Assignee: Hui Zheng
Priority: Minor
 Fix For: 2.8.0

 Attachments: HADOOP-10027.1.patch, HADOOP-10027.2.patch, 
 HADOOP-10027.3.patch, HADOOP-10027.4.patch, HADOOP-10027.5.patch


 http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c?view=markup
 This pattern appears in all the native compressors.
 // Get members of ZlibCompressor
 jobject clazz = (*env)-GetStaticObjectField(env, this,
  ZlibCompressor_clazz);
 The 2nd argument to GetStaticObjectField is supposed to be a jclass, not a 
 jobject. Adding the JVM param -Xcheck:jni will cause FATAL ERROR in native 
 method: JNI received a class argument that is not a class and a core dump 
 such as the following.
 (gdb) 
 #0 0x7f02e4aef8a5 in raise () from /lib64/libc.so.6
 #1 0x7f02e4af1085 in abort () from /lib64/libc.so.6
 #2 0x7f02e45bd727 in os::abort(bool) () from 
 /opt/jdk1.6.0_31/jre/lib/amd64/server/libjvm.so
 #3 0x7f02e43cec63 in jniCheck::validate_class(JavaThread*, _jclass*, 
 bool) () from /opt/jdk1.6.0_31/jre/lib/amd64/server/libjvm.so
 #4 0x7f02e43ea669 in checked_jni_GetStaticObjectField () from 
 /opt/jdk1.6.0_31/jre/lib/amd64/server/libjvm.so
 #5 0x7f02d38eaf79 in 
 Java_org_apache_hadoop_io_compress_zlib_ZlibCompressor_deflateBytesDirect () 
 from /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
 In addition, that clazz object is only used for synchronization. In the case 
 of the native method _deflateBytesDirect, the result is a class wide lock 
 used to access the instance

[jira] [Commented] (HADOOP-11703) git should ignore .DS_Store files on Mac OS X


[ 
https://issues.apache.org/jira/browse/HADOOP-11703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358882#comment-14358882
 ] 

Hudson commented on HADOOP-11703:
-

FAILURE: Integrated in Hadoop-Mapreduce-trunk #2080 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2080/])
HADOOP-11703. git should ignore .DS_Store files on Mac OS X (Abin Shahab via 
aw) (aw: rev 344d7cb6dcd3b1752c82b62271799be9e4e10416)
* hadoop-common-project/hadoop-common/CHANGES.txt
* .gitignore


 git should ignore .DS_Store files on Mac OS X
 -

 Key: HADOOP-11703
 URL: https://issues.apache.org/jira/browse/HADOOP-11703
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Abin Shahab
Assignee: Abin Shahab
 Fix For: 3.0.0

 Attachments: HADOOP-11703.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11693) Azure Storage FileSystem rename operations are throttled too aggressively to complete HBase WAL archiving.


[ 
https://issues.apache.org/jira/browse/HADOOP-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358883#comment-14358883
 ] 

Hudson commented on HADOOP-11693:
-

FAILURE: Integrated in Hadoop-Mapreduce-trunk #2080 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2080/])
HADOOP-11693. Azure Storage FileSystem rename operations are throttled too 
aggressively to complete HBase WAL archiving. Contributed by Duo Xu. (cnauroth: 
rev 7a346bcb4fa5b56191ed00a39e72e51c9bdf1b56)
* 
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/MockStorageInterface.java
* 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterfaceImpl.java
* 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
* 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterface.java
* hadoop-common-project/hadoop-common/CHANGES.txt


 Azure Storage FileSystem rename operations are throttled too aggressively to 
 complete HBase WAL archiving.
 --

 Key: HADOOP-11693
 URL: https://issues.apache.org/jira/browse/HADOOP-11693
 Project: Hadoop Common
  Issue Type: Bug
  Components: tools
Reporter: Duo Xu
Assignee: Duo Xu
 Fix For: 2.7.0

 Attachments: HADOOP-11681.01.patch, HADOOP-11681.02.patch, 
 HADOOP-11681.03.patch


 One of our customers' production HBase clusters was periodically throttled by 
 Azure storage, when HBase was archiving old WALs. HMaster aborted the region 
 server and tried to restart it.
 However, since the cluster was still being throttled by Azure storage, the 
 upcoming distributed log splitting also failed. Sometimes hbase:meta table 
 was on this region server and finally showed offline, which cause the whole 
 cluster in bad state.
 {code}
 2015-03-01 18:36:45,623 ERROR org.apache.hadoop.hbase.master.HMaster: Region 
 server 
 workernode4.hbaseproddb4001.f5.internal.cloudapp.net,60020,1424845421044 
 reported a fatal error:
 ABORTING region server 
 workernode4.hbaseproddb4001.f5.internal.cloudapp.net,60020,1424845421044: IOE 
 in log roller
 Cause:
 org.apache.hadoop.fs.azure.AzureException: 
 com.microsoft.windowsazure.storage.StorageException: The server is busy.
   at 
 org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2446)
   at 
 org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2367)
   at 
 org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1960)
   at 
 org.apache.hadoop.hbase.util.FSUtils.renameAndSetModifyTime(FSUtils.java:1719)
   at 
 org.apache.hadoop.hbase.regionserver.wal.FSHLog.archiveLogFile(FSHLog.java:798)
   at 
 org.apache.hadoop.hbase.regionserver.wal.FSHLog.cleanOldLogs(FSHLog.java:656)
   at 
 org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:593)
   at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:97)
   at java.lang.Thread.run(Thread.java:745)
 Caused by: com.microsoft.windowsazure.storage.StorageException: The server is 
 busy.
   at 
 com.microsoft.windowsazure.storage.StorageException.translateException(StorageException.java:163)
   at 
 com.microsoft.windowsazure.storage.core.StorageRequest.materializeException(StorageRequest.java:306)
   at 
 com.microsoft.windowsazure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:229)
   at 
 com.microsoft.windowsazure.storage.blob.CloudBlob.startCopyFromBlob(CloudBlob.java:762)
   at 
 org.apache.hadoop.fs.azurenative.StorageInterfaceImpl$CloudBlobWrapperImpl.startCopyFromBlob(StorageInterfaceImpl.java:350)
   at 
 org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2439)
   ... 8 more
 2015-03-01 18:43:29,072 ERROR org.apache.hadoop.hbase.executor.EventHandler: 
 Caught throwable while processing event M_META_SERVER_SHUTDOWN
 java.io.IOException: failed log splitting for 
 workernode13.hbaseproddb4001.f5.internal.cloudapp.net,60020,1424845307901, 
 will retry
   at 
 org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.process(MetaServerShutdownHandler.java:71)
   at 
 org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:745)
 Caused by: org.apache.hadoop.fs.azure.AzureException: 
 com.microsoft.windowsazure.storage.StorageException: The server is busy.
   at

[jira] [Commented] (HADOOP-11702) Allow StringSignerSecretProvider to read secret from a file


[ 
https://issues.apache.org/jira/browse/HADOOP-11702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358919#comment-14358919
 ] 

Allen Wittenauer commented on HADOOP-11702:
---

I just realized that it is a different config option to use this file support, 
so existent file filters need to use the old method too unless we are willing 
to break bw compat. :(

 Allow StringSignerSecretProvider to read secret from a file
 ---

 Key: HADOOP-11702
 URL: https://issues.apache.org/jira/browse/HADOOP-11702
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Arun Suresh
Assignee: Arun Suresh
Priority: Minor
 Attachments: HADOOP-11702.1.patch, HADOOP-11702.2.patch


 Currently, The {{StringSignerSecretProvider}} has to be provide the secret in 
 a configuration file. This patch allows the secret string to be read from a 
 separate file as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11708) CryptoOutputStream synchronization differences from DFSOutputStream break HBase

Sean Busbey created HADOOP-11708:


 Summary: CryptoOutputStream synchronization differences from 
DFSOutputStream break HBase
 Key: HADOOP-11708
 URL: https://issues.apache.org/jira/browse/HADOOP-11708
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 2.6.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Priority: Critical


For the write-ahead-log, HBase writes to DFS from a single thread and sends 
sync/flush/hflush from a configurable number of other threads (default 5).

FSDataOutputStream does not document anything about being thread safe, and it 
is not thread safe for concurrent writes.

However, DFSOutputStream is thread safe for concurrent writes + syncs. When it 
is the stream FSDataOutputStream wraps, the combination is threadsafe for 1 
writer and multiple syncs (the exact behavior HBase relies on).

When HDFS Transparent Encryption is turned on, CryptoOutputStream is inserted 
between FSDataOutputStream and DFSOutputStream. It is proactively labeled as 
not thread safe, and this composition is not thread safe for any operations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11656) Classpath isolation for downstream clients

2015-03-12 Thread Sangjin Lee (JIRA)

[
https://issues.apache.org/jira/browse/HADOOP-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358835#comment-14358835
]

Sangjin Lee commented on HADOOP-11656:
--

[~ozawa], I don't think jigsaw would be a viable option at this point. If
nothing else, it's slated to be released as part of JDK 9 which is somewhat out
there. Moreover, there were a lot of challenges and issues with the project
getting off the ground (originally it was going to be part of JDK 8:
http://cr.openjdk.java.net/~mr/jigsaw/notes/jigsaw-big-picture-01). So if OSGi
was going to be a challenge, IMO jigsaw would be even more of a challenge, and
its quality when and if it ships is an unknown at this point.

Classpath isolation for downstream clients
--

Key: HADOOP-11656
URL: https://issues.apache.org/jira/browse/HADOOP-11656
Project: Hadoop Common
Issue Type: New Feature
Reporter: Sean Busbey
Assignee: Sean Busbey
Labels: classloading, classpath, dependencies, scripts, shell

Currently, Hadoop exposes downstream clients to a variety of third party
libraries. As our code base grows and matures we increase the set of
libraries we rely on. At the same time, as our user base grows we increase
the likelihood that some downstream project will run into a conflict while
attempting to use a different version of some library we depend on. This has
already happened with i.e. Guava several times for HBase, Accumulo, and Spark
(and I'm sure others).
While YARN-286 and MAPREDUCE-1700 provided an initial effort, they default to
off and they don't do anything to help dependency conflicts on the driver
side or for folks talking to HDFS directly. This should serve as an umbrella
for changes needed to do things thoroughly on the next major version.
We should ensure that downstream clients
1) can depend on a client artifact for each of HDFS, YARN, and MapReduce that
doesn't pull in any third party dependencies
2) only see our public API classes (or as close to this as feasible) when
executing user provided code, whether client side in a launcher/driver or on
the cluster in a container or within MR.
This provides us with a double benefit: users get less grief when they want
to run substantially ahead or behind the versions we need and the project is
freer to change our own dependency versions because they'll no longer be in
our compatibility promises.
Project specific task jiras to follow after I get some justifying use cases
written in the comments.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-10027) *Compressor_deflateBytesDirect passes instance instead of jclass to GetStaticObjectField


[ 
https://issues.apache.org/jira/browse/HADOOP-10027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358831#comment-14358831
 ] 

Hudson commented on HADOOP-10027:
-

FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #130 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/130/])
HADOOP-10027. *Compressor_deflateBytesDirect passes instance instead of jclass 
to GetStaticObjectField. Contributed by Hui Zheng. (cnauroth: rev 
ff83ae72318fe6c0f266a7bf08138bd2fdb51cbd)
* hadoop-common-project/hadoop-common/CHANGES.txt
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyCompressor.java
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zlib/ZlibDecompressor.c
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/bzip2/Bzip2Compressor.c
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/ZlibDecompressor.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/lz4/TestLz4CompressorDecompressor.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/bzip2/TestBzip2CompressorDecompressor.java
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/zlib/TestZlibCompressorDecompressor.java
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.java
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/bzip2/Bzip2Decompressor.c
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/Bzip2Decompressor.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/snappy/TestSnappyCompressorDecompressor.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/Bzip2Compressor.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/ZlibCompressor.java
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java


 *Compressor_deflateBytesDirect passes instance instead of jclass to 
 GetStaticObjectField
 

 Key: HADOOP-10027
 URL: https://issues.apache.org/jira/browse/HADOOP-10027
 Project: Hadoop Common
  Issue Type: Bug
  Components: native
Reporter: Eric Abbott
Assignee: Hui Zheng
Priority: Minor
 Fix For: 2.8.0

 Attachments: HADOOP-10027.1.patch, HADOOP-10027.2.patch, 
 HADOOP-10027.3.patch, HADOOP-10027.4.patch, HADOOP-10027.5.patch


 http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c?view=markup
 This pattern appears in all the native compressors.
 // Get members of ZlibCompressor
 jobject clazz = (*env)-GetStaticObjectField(env, this,
  ZlibCompressor_clazz);
 The 2nd argument to GetStaticObjectField is supposed to be a jclass, not a 
 jobject. Adding the JVM param -Xcheck:jni will cause FATAL ERROR in native 
 method: JNI received a class argument that is not a class and a core dump 
 such as the following.
 (gdb) 
 #0 0x7f02e4aef8a5 in raise () from /lib64/libc.so.6
 #1 0x7f02e4af1085 in abort () from /lib64/libc.so.6
 #2 0x7f02e45bd727 in os::abort(bool) () from 
 /opt/jdk1.6.0_31/jre/lib/amd64/server/libjvm.so
 #3 0x7f02e43cec63 in jniCheck::validate_class(JavaThread*, _jclass*, 
 bool) () from /opt/jdk1.6.0_31/jre/lib/amd64/server/libjvm.so
 #4 0x7f02e43ea669 in checked_jni_GetStaticObjectField () from 
 /opt/jdk1.6.0_31/jre/lib/amd64/server/libjvm.so
 #5 0x7f02d38eaf79 in 
 Java_org_apache_hadoop_io_compress_zlib_ZlibCompressor_deflateBytesDirect () 
 from /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
 In addition, that clazz object is only used for synchronization. In the case 
 of the native method _deflateBytesDirect, the result is a class wide lock 
 used to access the

[jira] [Commented] (HADOOP-11703) git should ignore .DS_Store files on Mac OS X


[ 
https://issues.apache.org/jira/browse/HADOOP-11703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358829#comment-14358829
 ] 

Hudson commented on HADOOP-11703:
-

FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #130 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/130/])
HADOOP-11703. git should ignore .DS_Store files on Mac OS X (Abin Shahab via 
aw) (aw: rev 344d7cb6dcd3b1752c82b62271799be9e4e10416)
* hadoop-common-project/hadoop-common/CHANGES.txt
* .gitignore


 git should ignore .DS_Store files on Mac OS X
 -

 Key: HADOOP-11703
 URL: https://issues.apache.org/jira/browse/HADOOP-11703
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Abin Shahab
Assignee: Abin Shahab
 Fix For: 3.0.0

 Attachments: HADOOP-11703.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Moved] (HADOOP-11709) Time.NANOSECONDS_PER_MILLISECOND - use class level final constant instead of method variable


 [ 
https://issues.apache.org/jira/browse/HADOOP-11709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey moved HDFS-7919 to HADOOP-11709:


Key: HADOOP-11709  (was: HDFS-7919)
Project: Hadoop Common  (was: Hadoop HDFS)

 Time.NANOSECONDS_PER_MILLISECOND - use class level final constant instead of 
 method variable 
 -

 Key: HADOOP-11709
 URL: https://issues.apache.org/jira/browse/HADOOP-11709
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Ajith S
Assignee: Ajith S
Priority: Trivial
  Labels: beginner

 NANOSECONDS_PER_MILLISECOND constant can be moved to class level instead of 
 creating it in each method call.
 {code}
 org.apache.hadoop.util.Time.java
  public static long monotonicNow() {
 final long NANOSECONDS_PER_MILLISECOND = 100;
 return System.nanoTime() / NANOSECONDS_PER_MILLISECOND;
   }
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11693) Azure Storage FileSystem rename operations are throttled too aggressively to complete HBase WAL archiving.


[ 
https://issues.apache.org/jira/browse/HADOOP-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358830#comment-14358830
 ] 

Hudson commented on HADOOP-11693:
-

FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #130 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/130/])
HADOOP-11693. Azure Storage FileSystem rename operations are throttled too 
aggressively to complete HBase WAL archiving. Contributed by Duo Xu. (cnauroth: 
rev 7a346bcb4fa5b56191ed00a39e72e51c9bdf1b56)
* 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
* 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterface.java
* 
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/MockStorageInterface.java
* 
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterfaceImpl.java
* hadoop-common-project/hadoop-common/CHANGES.txt


 Azure Storage FileSystem rename operations are throttled too aggressively to 
 complete HBase WAL archiving.
 --

 Key: HADOOP-11693
 URL: https://issues.apache.org/jira/browse/HADOOP-11693
 Project: Hadoop Common
  Issue Type: Bug
  Components: tools
Reporter: Duo Xu
Assignee: Duo Xu
 Fix For: 2.7.0

 Attachments: HADOOP-11681.01.patch, HADOOP-11681.02.patch, 
 HADOOP-11681.03.patch


 One of our customers' production HBase clusters was periodically throttled by 
 Azure storage, when HBase was archiving old WALs. HMaster aborted the region 
 server and tried to restart it.
 However, since the cluster was still being throttled by Azure storage, the 
 upcoming distributed log splitting also failed. Sometimes hbase:meta table 
 was on this region server and finally showed offline, which cause the whole 
 cluster in bad state.
 {code}
 2015-03-01 18:36:45,623 ERROR org.apache.hadoop.hbase.master.HMaster: Region 
 server 
 workernode4.hbaseproddb4001.f5.internal.cloudapp.net,60020,1424845421044 
 reported a fatal error:
 ABORTING region server 
 workernode4.hbaseproddb4001.f5.internal.cloudapp.net,60020,1424845421044: IOE 
 in log roller
 Cause:
 org.apache.hadoop.fs.azure.AzureException: 
 com.microsoft.windowsazure.storage.StorageException: The server is busy.
   at 
 org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2446)
   at 
 org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2367)
   at 
 org.apache.hadoop.fs.azurenative.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1960)
   at 
 org.apache.hadoop.hbase.util.FSUtils.renameAndSetModifyTime(FSUtils.java:1719)
   at 
 org.apache.hadoop.hbase.regionserver.wal.FSHLog.archiveLogFile(FSHLog.java:798)
   at 
 org.apache.hadoop.hbase.regionserver.wal.FSHLog.cleanOldLogs(FSHLog.java:656)
   at 
 org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:593)
   at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:97)
   at java.lang.Thread.run(Thread.java:745)
 Caused by: com.microsoft.windowsazure.storage.StorageException: The server is 
 busy.
   at 
 com.microsoft.windowsazure.storage.StorageException.translateException(StorageException.java:163)
   at 
 com.microsoft.windowsazure.storage.core.StorageRequest.materializeException(StorageRequest.java:306)
   at 
 com.microsoft.windowsazure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:229)
   at 
 com.microsoft.windowsazure.storage.blob.CloudBlob.startCopyFromBlob(CloudBlob.java:762)
   at 
 org.apache.hadoop.fs.azurenative.StorageInterfaceImpl$CloudBlobWrapperImpl.startCopyFromBlob(StorageInterfaceImpl.java:350)
   at 
 org.apache.hadoop.fs.azurenative.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2439)
   ... 8 more
 2015-03-01 18:43:29,072 ERROR org.apache.hadoop.hbase.executor.EventHandler: 
 Caught throwable while processing event M_META_SERVER_SHUTDOWN
 java.io.IOException: failed log splitting for 
 workernode13.hbaseproddb4001.f5.internal.cloudapp.net,60020,1424845307901, 
 will retry
   at 
 org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.process(MetaServerShutdownHandler.java:71)
   at 
 org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:745)
 Caused by: org.apache.hadoop.fs.azure.AzureException: 
 com.microsoft.windowsazure.storage.StorageException: The server is busy.
   at

[jira] [Updated] (HADOOP-8320) FileSystem#checkPath and AbstractFileSystem#checkPath should share code


 [ 
https://issues.apache.org/jira/browse/HADOOP-8320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vijay Bhat updated HADOOP-8320:
---
Attachment: HADOOP-8320.001.patch

 FileSystem#checkPath and AbstractFileSystem#checkPath should share code
 ---

 Key: HADOOP-8320
 URL: https://issues.apache.org/jira/browse/HADOOP-8320
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Affects Versions: 2.0.0-alpha
Reporter: Aaron T. Myers
Assignee: Vijay Bhat
Priority: Minor
 Attachments: HADOOP-8320.001.patch


 Per the discussion on HADOOP-8310, these two methods can be refactored to 
 share some code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-10703) Duplicate filter instances are created during HttpServer2 initialization

2015-03-12 Thread Arpit Agarwal (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-10703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359049#comment-14359049
 ] 

Arpit Agarwal commented on HADOOP-10703:


+1 for the patch. Thanks [~benoyantony].

 Duplicate filter instances are created during HttpServer2 initialization
 

 Key: HADOOP-10703
 URL: https://issues.apache.org/jira/browse/HADOOP-10703
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Affects Versions: 2.4.0
Reporter: Benoy Antony
Assignee: Benoy Antony
 Attachments: HADOOP-10703-002.patch, HADOOP-10703.patch, 
 multiple-authenticationfilter-inits.log


 The HttpServer2.defineFilter creates a Filter instance for each context. By 
 default, there are 3 contexts.
 So there will be 3 separate AuthenticationFilter instances and corresponding 
 AuthenticationHandler instances. This also results in 3 separate 
 initializations of AuthenticationHandler.
 The log file illustrating this repeated initialization is attached.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11708) CryptoOutputStream synchronization differences from DFSOutputStream break HBase

2015-03-12 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359026#comment-14359026
 ] 

Steve Loughran commented on HADOOP-11708:
-

More succinctly, a pre-emptive -1 to any changes to DFSOutputStream sync logic 
in 2.7, as it needs to be accompanied by thought, investigation  pretty 
rigorous specifications — currently in Z-pretending-to-be-Python, but I'll 
happily take TLA+ if you prefer.

Changing HBase WAL to work with unsynced streams or fixing CryptoOutputStream 
to implement same expectations of HDFS are much lower risk.

 CryptoOutputStream synchronization differences from DFSOutputStream break 
 HBase
 ---

 Key: HADOOP-11708
 URL: https://issues.apache.org/jira/browse/HADOOP-11708
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 2.6.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Priority: Critical

 For the write-ahead-log, HBase writes to DFS from a single thread and sends 
 sync/flush/hflush from a configurable number of other threads (default 5).
 FSDataOutputStream does not document anything about being thread safe, and it 
 is not thread safe for concurrent writes.
 However, DFSOutputStream is thread safe for concurrent writes + syncs. When 
 it is the stream FSDataOutputStream wraps, the combination is threadsafe for 
 1 writer and multiple syncs (the exact behavior HBase relies on).
 When HDFS Transparent Encryption is turned on, CryptoOutputStream is inserted 
 between FSDataOutputStream and DFSOutputStream. It is proactively labeled as 
 not thread safe, and this composition is not thread safe for any operations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11708) CryptoOutputStream synchronization differences from DFSOutputStream break HBase


[ 
https://issues.apache.org/jira/browse/HADOOP-11708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358999#comment-14358999
 ] 

Sean Busbey commented on HADOOP-11708:
--

(We could also just remove the synchronization from DFSOutputStream, release 
note the change,  and require HBase to correct its behavior)

 CryptoOutputStream synchronization differences from DFSOutputStream break 
 HBase
 ---

 Key: HADOOP-11708
 URL: https://issues.apache.org/jira/browse/HADOOP-11708
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 2.6.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Priority: Critical

 For the write-ahead-log, HBase writes to DFS from a single thread and sends 
 sync/flush/hflush from a configurable number of other threads (default 5).
 FSDataOutputStream does not document anything about being thread safe, and it 
 is not thread safe for concurrent writes.
 However, DFSOutputStream is thread safe for concurrent writes + syncs. When 
 it is the stream FSDataOutputStream wraps, the combination is threadsafe for 
 1 writer and multiple syncs (the exact behavior HBase relies on).
 When HDFS Transparent Encryption is turned on, CryptoOutputStream is inserted 
 between FSDataOutputStream and DFSOutputStream. It is proactively labeled as 
 not thread safe, and this composition is not thread safe for any operations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-7947) Validate XMLs if a relevant tool is available, when using scripts


[ 
https://issues.apache.org/jira/browse/HADOOP-7947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359015#comment-14359015
 ] 

Allen Wittenauer commented on HADOOP-7947:
--

I'd definitely want a second, third, ... opinion on that as well. Hopefully the 
other folks watching will chime in...

I'm inclined to say yes, that's probably the better direction but that's 
clearly a different JIRA.

 Validate XMLs if a relevant tool is available, when using scripts
 -

 Key: HADOOP-7947
 URL: https://issues.apache.org/jira/browse/HADOOP-7947
 Project: Hadoop Common
  Issue Type: Wish
  Components: scripts
Affects Versions: 2.7.0
Reporter: Harsh J
Assignee: Kengo Seki
  Labels: newbie
 Attachments: HADOOP-7947.001.patch, HADOOP-7947.002.patch, 
 HADOOP-7947.003.patch


 Given that we are locked down to using only XML for configuration and most of 
 the administrators need to manage it by themselves (unless a tool that 
 manages for you is used), it would be good to also validate the provided 
 config XML (*-site.xml) files with a tool like {{xmllint}} or maybe Xerces 
 somehow, when running a command or (at least) when starting up daemons.
 We should use this only if a relevant tool is available, and optionally be 
 silent if the env. requests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-10703) Duplicate filter instances are created during HttpServer2 initialization

2015-03-12 Thread Arpit Agarwal (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-10703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359066#comment-14359066
 ] 

Arpit Agarwal commented on HADOOP-10703:


Also I am holding off committing since [~tucu00] has not signed off yet. For my 
part, I didn't see anything in the documentation requiring the instances to be 
unique.

 Duplicate filter instances are created during HttpServer2 initialization
 

 Key: HADOOP-10703
 URL: https://issues.apache.org/jira/browse/HADOOP-10703
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Affects Versions: 2.4.0
Reporter: Benoy Antony
Assignee: Benoy Antony
 Attachments: HADOOP-10703-002.patch, HADOOP-10703.patch, 
 multiple-authenticationfilter-inits.log


 The HttpServer2.defineFilter creates a Filter instance for each context. By 
 default, there are 3 contexts.
 So there will be 3 separate AuthenticationFilter instances and corresponding 
 AuthenticationHandler instances. This also results in 3 separate 
 initializations of AuthenticationHandler.
 The log file illustrating this repeated initialization is attached.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11708) CryptoOutputStream synchronization differences from DFSOutputStream break HBase

2015-03-12 Thread Xiaoyu Yao (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358943#comment-14358943
 ] 

Xiaoyu Yao commented on HADOOP-11708:
-

[~busbey], we found the same issue a couple of days ago and [~hitliuyi] is 
working on a fix under 
[HDFS-7911|https://issues.apache.org/jira/browse/HDFS-7911] .

 CryptoOutputStream synchronization differences from DFSOutputStream break 
 HBase
 ---

 Key: HADOOP-11708
 URL: https://issues.apache.org/jira/browse/HADOOP-11708
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 2.6.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Priority: Critical

 For the write-ahead-log, HBase writes to DFS from a single thread and sends 
 sync/flush/hflush from a configurable number of other threads (default 5).
 FSDataOutputStream does not document anything about being thread safe, and it 
 is not thread safe for concurrent writes.
 However, DFSOutputStream is thread safe for concurrent writes + syncs. When 
 it is the stream FSDataOutputStream wraps, the combination is threadsafe for 
 1 writer and multiple syncs (the exact behavior HBase relies on).
 When HDFS Transparent Encryption is turned on, CryptoOutputStream is inserted 
 between FSDataOutputStream and DFSOutputStream. It is proactively labeled as 
 not thread safe, and this composition is not thread safe for any operations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11708) CryptoOutputStream synchronization differences from DFSOutputStream break HBase

2015-03-12 Thread Steve Loughran (JIRA)

[
https://issues.apache.org/jira/browse/HADOOP-11708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359009#comment-14359009
]

Steve Loughran commented on HADOOP-11708:
-

hmmm.

# HADOOP-9361 skipped on the output stream stuff in the rush to get things into
hadoop2.5, and for all its stuff is very vague about concurrency guarantees. As
it extends {{java.io.OutputStream}}, the real concurrency semantics should be
defined there. And, looking at that openjdk code, it appears to be no
guarantees. HBase have some expectations that may already be invalid.
# We've also seen from HDFS-6735 that HBase likes non-blocking reads for
performance enhancements...they clearly have different/needs expectations.
HBASE-8722 is where someone proposed writing them down. If you look at
HDFS-6735 we had to spend a lot of time thinking about looking at what is
going on, trying to understand the implicit concurrency guarantees of HDFS
the expectations of code as seen by IDE-based scans for usage.
# If you are proposing we define those concurrency semantics more strictly, in
the filesystem specification and the code that would be great: someone did need
to sit down and define the APIs as based on HDFS behaviour. This is not
something we can rush into in the last minute though. I also worry about the
proposed culling of +10 sync blocks, especially on the basis that they appear
to be unneeded. That's the kind of question that needs to be answered by
looking at the git/svn history of those lines correlating them with past
JIRAs, before doing what could potentially cause problems. And, at write()
time, that means data-loss/corruption problems beyond just encrypted data.

Short term? I'd say note the concurrency expectations of HBase and an output
stream come up with a change for the CryptoOutputStream which implements
consistent concurrency semantics. The runup to a release is not the time to
change something so foundational.

Longer term? Decide what the concurrency guarantees should be, scan through the
core code stack to identify risky uses of flush() (actually I'd add log/count
to DFSOutputStream see if we could detect log re-entrant ops across
threads: flush/write overlap., flush/flush concurrency, hflush+write, ...). As
with HADOOP-9361, HDFS defines the defacto semantics, but its the uses of that
code which really set the expectations of applications. Here we've just found
HBases

CryptoOutputStream synchronization differences from DFSOutputStream break
HBase
---

Key: HADOOP-11708
URL: https://issues.apache.org/jira/browse/HADOOP-11708
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.6.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Priority: Critical

For the write-ahead-log, HBase writes to DFS from a single thread and sends
sync/flush/hflush from a configurable number of other threads (default 5).
FSDataOutputStream does not document anything about being thread safe, and it
is not thread safe for concurrent writes.
However, DFSOutputStream is thread safe for concurrent writes + syncs. When
it is the stream FSDataOutputStream wraps, the combination is threadsafe for
1 writer and multiple syncs (the exact behavior HBase relies on).
When HDFS Transparent Encryption is turned on, CryptoOutputStream is inserted
between FSDataOutputStream and DFSOutputStream. It is proactively labeled as
not thread safe, and this composition is not thread safe for any operations.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11708) CryptoOutputStream synchronization differences from DFSOutputStream break HBase

[
https://issues.apache.org/jira/browse/HADOOP-11708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359054#comment-14359054
]

Sean Busbey commented on HADOOP-11708:
--

{quote}
The runup to a release is not the time to change something so foundational.
{quote}

FWIW, I just picked the first unreleased versions on the jira. I'm not working
to meet any particular release schedule.

{quote}
1. HADOOP-9361 skipped on the output stream stuff in the rush to get things
into hadoop2.5, and for all its stuff is very vague about concurrency
guarantees. As it extends java.io.OutputStream, the real concurrency semantics
should be defined there. And, looking at that openjdk code, it appears to be
no guarantees. HBase have some expectations that may already be invalid.
{quote}

I agree. DFSOutputStream is the first byte stream implementation I've seen that
tries to be thread safe.

{quote}
3. If you are proposing we define those concurrency semantics more strictly, in
the filesystem specification and the code that would be great: someone did need
to sit down and define the APIs as based on HDFS behaviour. This is not
something we can rush into in the last minute though. I also worry about the
proposed culling of +10 sync blocks, especially on the basis that they appear
to be unneeded. That's the kind of question that needs to be answered by
looking at the git/svn history of those lines correlating them with past
JIRAs, before doing what could potentially cause problems. And, at write()
time, that means data-loss/corruption problems beyond just encrypted data.
{quote}

That tracing is what I spent last night doing. AFAICT, most of those block come
without comment in commit or jira about why. They appear to be just matching
what was already present. The earliest ones I ran into problems tracing because
of svn merge commits in ~2009. The lack of thread safety when attempting to
write from FSDataOutputStream is also a big flag.

By they appear to be unneeded I mean I have some worked through
rationalizations (in the absence of written comments from their authors) about
what they're trying to protect and why that either isn't necessary or isn't
done correctly. I can get these polished up.

{quote}
Changing HBase WAL to work with unsynced streams or fixing CryptoOutputStream
to implement same expectations of HDFS are much lower risk.
{quote}

+1, working on this in HBASE-13221

CryptoOutputStream synchronization differences from DFSOutputStream break
HBase
---

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11708) CryptoOutputStream synchronization differences from DFSOutputStream break HBase


[ 
https://issues.apache.org/jira/browse/HADOOP-11708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359078#comment-14359078
 ] 

Sean Busbey commented on HADOOP-11708:
--

As a note, we can do the FSDataOutputStream fix by synchronizing on the wrapped 
stream without removing the synch that's in DFSOutputStream. That'll mean they 
both will rely on the same monitor, which is the lowest overhead we can get for 
a general solution (and the jit has the potential to eventually combine the 
monitor increment/decrement)

 CryptoOutputStream synchronization differences from DFSOutputStream break 
 HBase
 ---

 Key: HADOOP-11708
 URL: https://issues.apache.org/jira/browse/HADOOP-11708
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 2.6.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Priority: Critical

 For the write-ahead-log, HBase writes to DFS from a single thread and sends 
 sync/flush/hflush from a configurable number of other threads (default 5).
 FSDataOutputStream does not document anything about being thread safe, and it 
 is not thread safe for concurrent writes.
 However, DFSOutputStream is thread safe for concurrent writes + syncs. When 
 it is the stream FSDataOutputStream wraps, the combination is threadsafe for 
 1 writer and multiple syncs (the exact behavior HBase relies on).
 When HDFS Transparent Encryption is turned on, CryptoOutputStream is inserted 
 between FSDataOutputStream and DFSOutputStream. It is proactively labeled as 
 not thread safe, and this composition is not thread safe for any operations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-8320) FileSystem#checkPath and AbstractFileSystem#checkPath should share code


 [ 
https://issues.apache.org/jira/browse/HADOOP-8320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vijay Bhat updated HADOOP-8320:
---
Attachment: (was: HADOOP-8320.001.patch)

 FileSystem#checkPath and AbstractFileSystem#checkPath should share code
 ---

 Key: HADOOP-8320
 URL: https://issues.apache.org/jira/browse/HADOOP-8320
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Affects Versions: 2.0.0-alpha
Reporter: Aaron T. Myers
Assignee: Vijay Bhat
Priority: Minor
 Attachments: HADOOP-8320.001.patch


 Per the discussion on HADOOP-8310, these two methods can be refactored to 
 share some code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-10846) DataChecksum#calculateChunkedSums not working for PPC when buffers not backed by array

2015-03-12 Thread Steve Loughran (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-10846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-10846:

Priority: Major  (was: Blocker)
Target Version/s: 2.8.0  (was: 2.7.0)

Not down as a blocker for 2.7

 DataChecksum#calculateChunkedSums not working for PPC when buffers not backed 
 by array
 --

 Key: HADOOP-10846
 URL: https://issues.apache.org/jira/browse/HADOOP-10846
 Project: Hadoop Common
  Issue Type: Bug
  Components: util
Affects Versions: 2.4.1, 2.5.2
 Environment: PowerPC platform
Reporter: Jinghui Wang
Assignee: Ayappan
 Attachments: HADOOP-10846-v1.patch, HADOOP-10846-v2.patch, 
 HADOOP-10846-v3.patch, HADOOP-10846-v4.patch, HADOOP-10846.patch


 Got the following exception when running Hadoop on Power PC. The 
 implementation for computing checksum when the data buffer and checksum 
 buffer are not backed by arrays.
 13/09/16 04:06:57 ERROR security.UserGroupInformation: 
 PriviledgedActionException as:biadmin (auth:SIMPLE) 
 cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): 
 org.apache.hadoop.fs.ChecksumException: Checksum error



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-8320) FileSystem#checkPath and AbstractFileSystem#checkPath should share code


[ 
https://issues.apache.org/jira/browse/HADOOP-8320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358969#comment-14358969
 ] 

Hadoop QA commented on HADOOP-8320:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12704185/HADOOP-8320.001.patch
  against trunk revision ff83ae7.

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/5926//console

This message is automatically generated.

 FileSystem#checkPath and AbstractFileSystem#checkPath should share code
 ---

 Key: HADOOP-8320
 URL: https://issues.apache.org/jira/browse/HADOOP-8320
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Affects Versions: 2.0.0-alpha
Reporter: Aaron T. Myers
Assignee: Vijay Bhat
Priority: Minor
 Attachments: HADOOP-8320.001.patch


 Per the discussion on HADOOP-8310, these two methods can be refactored to 
 share some code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11708) CryptoOutputStream synchronization differences from DFSOutputStream break HBase

2015-03-12 Thread Xiaoyu Yao (JIRA)

[
https://issues.apache.org/jira/browse/HADOOP-11708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359044#comment-14359044
]

Xiaoyu Yao commented on HADOOP-11708:
-

Agree with [~stev...@iseran.com] on the risk assessment of changing
DFSOutputStream.
+1 for fixing CryptoOutputStream to implement the same expectations of HDFS.

As an alternative before the CryptoOutputStream is fixed, users can use HBase
native encryption at-rest introduced by
[HBASE-7544|https://issues.apache.org/jira/browse/HBASE-7544] to encypt HBase
HFile/WAL files and persist them with normal HDFS DFSOutputStream.

CryptoOutputStream synchronization differences from DFSOutputStream break
HBase
---

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11702) Allow StringSignerSecretProvider to read secret from a file


[ 
https://issues.apache.org/jira/browse/HADOOP-11702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359051#comment-14359051
 ] 

Hadoop QA commented on HADOOP-11702:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12704181/HADOOP-11702.2.patch
  against trunk revision ff83ae7.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common:

  org.apache.hadoop.security.TestAuthenticationFilter
  org.apache.hadoop.ipc.TestRPCWaitForProxy

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/5925//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/5925//console

This message is automatically generated.

 Allow StringSignerSecretProvider to read secret from a file
 ---

 Key: HADOOP-11702
 URL: https://issues.apache.org/jira/browse/HADOOP-11702
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Arun Suresh
Assignee: Arun Suresh
Priority: Minor
 Attachments: HADOOP-11702.1.patch, HADOOP-11702.2.patch


 Currently, The {{StringSignerSecretProvider}} has to be provide the secret in 
 a configuration file. This patch allows the secret string to be read from a 
 separate file as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11708) CryptoOutputStream synchronization differences from DFSOutputStream break HBase

2015-03-12 Thread Colin Patrick McCabe (JIRA)

[
https://issues.apache.org/jira/browse/HADOOP-11708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359106#comment-14359106
]

Colin Patrick McCabe commented on HADOOP-11708:
---

I agree with [~stev...@iseran.com] here... +1 for changing
{{CryptoOutputStream}} to behave the same as HDFS. It should be a pretty small
patch.

bq. Sean wrote: ...we could remove ~10 synchronization blocks in DFSOS (some of
them are unneeded and just about all of them are questionable, and I can't find
a rationalization for them). As a follow-on, we add a FSDataOutputStream that
isn't threadsafe and says as much. We can do this compatibly by either making
it an option (in FSDataOutputStream construction or in configs), by making it a
new API, or making it a documented breaking change.

I agree there is a lot to clean up here. Let's talk about this in a separate
JIRA. We have a bunch of options here and I think the discussion will take a
while.

CryptoOutputStream synchronization differences from DFSOutputStream break
HBase
---

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11665) Provide and unify cross platform byteorder support in native code

2015-03-12 Thread Vinod Kumar Vavilapalli (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359157#comment-14359157
 ] 

Vinod Kumar Vavilapalli commented on HADOOP-11665:
--

[~decster], this sounds like a new feature to me without looking at the 
details. If so, can we put this in 2.8? I'm trying to reduce more churn on 2.7 
in order to get to a release soon. Tx.

 Provide and unify cross platform byteorder support in native code
 -

 Key: HADOOP-11665
 URL: https://issues.apache.org/jira/browse/HADOOP-11665
 Project: Hadoop Common
  Issue Type: Bug
  Components: util
Affects Versions: 2.4.1, 2.6.0
 Environment: PowerPC Big Endian  other Big Endian platforms
Reporter: Binglin Chang
Assignee: Binglin Chang
Priority: Blocker
 Attachments: HADOOP-11665.001.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11526) Memory leak in Bzip2Compressor and Bzip2Decompressor

2015-03-12 Thread Chris Nauroth (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359408#comment-14359408
 ] 

Chris Nauroth commented on HADOOP-11526:


Hello [~oae].  This bug would have been a small one-time leak per process 
during codec initialization.  There would not have been an additional leak per 
file.  I doubt the effect of this leak would have been noticeable in any 
realistic usage scenario.

 Memory leak in Bzip2Compressor and Bzip2Decompressor
 

 Key: HADOOP-11526
 URL: https://issues.apache.org/jira/browse/HADOOP-11526
 Project: Hadoop Common
  Issue Type: Bug
  Components: io, native
Affects Versions: 2.6.0
Reporter: Ian Rogers
Assignee: Anu Engineer
 Fix For: 2.7.0

 Attachments: HADOOP-11526.001.patch, HADOOP-11526.002.patch


 The use of JNI's GetStringUTFChars should be paired with 
 ReleaseStringUTFChars or else the utf-8 char* created by Java's JNI 
 implementation is leaked. It isn't in Bzip2Decompressor.c:
 https://apache.googlesource.com/hadoop-common/+/refs/heads/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/bzip2/Bzip2Decompressor.c#45
 A less error-prone way of handling JNI resources like local references and 
 UTF strings is to use a smart pointer like the Apache licensed code in 
 Android's ScopedLocalRef and ScopedUtfChars:
 https://android.googlesource.com/platform/libnativehelper/+/jb-mr1.1-dev-plus-aosp/include/nativehelper/ScopedLocalRef.h
 https://android.googlesource.com/platform/libnativehelper/+/jb-mr1.1-dev-plus-aosp/include/nativehelper/ScopedUtfChars.h



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11708) CryptoOutputStream synchronization differences from DFSOutputStream break HBase

2015-03-12 Thread Colin Patrick McCabe (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359468#comment-14359468
 ] 

Colin Patrick McCabe commented on HADOOP-11708:
---

Just a heads up, I am +1 on the patch in HADOOP-11710 to make 
CryptoOutputstream behave like DFSOuptutStream, and would like to commit it for 
2.7.

We can continue the discussion about specs and possible other fixes in other 
subtasks

 CryptoOutputStream synchronization differences from DFSOutputStream break 
 HBase
 ---

 Key: HADOOP-11708
 URL: https://issues.apache.org/jira/browse/HADOOP-11708
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 2.6.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Priority: Critical

 For the write-ahead-log, HBase writes to DFS from a single thread and sends 
 sync/flush/hflush from a configurable number of other threads (default 5).
 FSDataOutputStream does not document anything about being thread safe, and it 
 is not thread safe for concurrent writes.
 However, DFSOutputStream is thread safe for concurrent writes + syncs. When 
 it is the stream FSDataOutputStream wraps, the combination is threadsafe for 
 1 writer and multiple syncs (the exact behavior HBase relies on).
 When HDFS Transparent Encryption is turned on, CryptoOutputStream is inserted 
 between FSDataOutputStream and DFSOutputStream. It is proactively labeled as 
 not thread safe, and this composition is not thread safe for any operations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-10679) Authorize webui access using ServiceAuthorizationManager

2015-03-12 Thread Haohui Mai (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-10679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359342#comment-14359342
 ] 

Haohui Mai commented on HADOOP-10679:
-

The idea makes sense. It's good to have an auth filter that works for both RPC 
and WebUI.

There is one loose end though: how will WebHDFS authorization be handled? I 
think that it is closely related to what eventually happens in HDFS-5796. 
Therefore it might make sense to settle HDFS-5796 first and revisit this one.

 Authorize webui access using ServiceAuthorizationManager
 

 Key: HADOOP-10679
 URL: https://issues.apache.org/jira/browse/HADOOP-10679
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: security
Reporter: Benoy Antony
Assignee: Benoy Antony
 Attachments: HADOOP-10679.patch, HADOOP-10679.patch, hadoop-10679.pdf


 Currently accessing Hadoop via RPC can be authorized using 
 _ServiceAuthorizationManager_. But there is no uniform authorization of the 
 HTTP access. Some of the servlets check for admin privilege. 
 This creates an inconsistency of authorization between access via RPC vs 
 HTTP. 
 The fix is to enable authorization of the webui access also using 
 _ServiceAuthorizationManager_. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11710) Make CryptoOutputStream behave like DFSOutputStream wrt synchronization