[jira] [Commented] (HADOOP-10389) Native RPCv9 client
[ https://issues.apache.org/jira/browse/HADOOP-10389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033486#comment-14033486 ] Binglin Chang commented on HADOOP-10389: I'd like to add more inputs on this: I wrote a c++ rpc/hdfs/yarn client(https://github.com/decster/libhadoopclient), it uses c++11, so it does not need boost(although many people use boost, they just for header only libraries, and public headers does not include boost, so there are no version issues). c++ 's main concern is abi compatibility, this can be resolved by using c or simple c++ class public headers, hiding real implementation. I think some issue using c++/pro using c are: 1. centos does not have enough support for c++11, c++11 is not generally available yet 2. remain libhdfs compatibility, since libhdfs is written in c, we might just continue using c as well Also there are some concerns about using c: 1. the protobuf-c library is just not so reliable as official protobuf library which is maintained and verified by google and many other companies/projects, I read some of the protobuf-c code, it uses a reflection style implementation to do serializing/deserializing, so performance, security, compatibility may all at risk. I see https://github.com/protobuf-c/protobuf-c only have 92 stars. 2. malloc/free/memset can easily generate buggy code, need additional care and checks, I see many those kinds of code recently(HDFS-6534,HADOOP-10640,HADOOP-10706). it is OK to use c, but we may need more care and effort. About JNIFS, why do we need jnifs if we already have nativefs? using dlopen/dlsym to replace jni apis is not trivial if both compile/runtime dependency is needed to be removed. > Native RPCv9 client > --- > > Key: HADOOP-10389 > URL: https://issues.apache.org/jira/browse/HADOOP-10389 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-10388 >Reporter: Binglin Chang >Assignee: Colin Patrick McCabe > Attachments: HADOOP-10388.001.patch, > HADOOP-10389-alternative.000.patch, HADOOP-10389.002.patch, > HADOOP-10389.004.patch, HADOOP-10389.005.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10711) Cleanup some extra dependencies from hadoop-auth
[
https://issues.apache.org/jira/browse/HADOOP-10711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033476#comment-14033476
]
Hadoop QA commented on HADOOP-10711:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650733/HADOOP-10711_trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-auth.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4080//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4080//console
This message is automatically generated.
> Cleanup some extra dependencies from hadoop-auth
>
>
> Key: HADOOP-10711
> URL: https://issues.apache.org/jira/browse/HADOOP-10711
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.5.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: HADOOP-10711_branch-2.patch,
> HADOOP-10711_branch-2.patch, HADOOP-10711_trunk.patch,
> HADOOP-10711_trunk.patch
>
>
> HADOOP-10322 added {{apacheds-kerberos-codec}} as a dependency, which brought
> in some additional dependencies.
> {noformat}
> [INFO] \-
> org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
> [INFO]+- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
> [INFO]+- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-asn1-ber:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-i18n:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-ldap-model:jar:1.0.0-M20:compile
> [INFO]| +- org.apache.mina:mina-core:jar:2.0.0-M5:compile
> [INFO]| +- antlr:antlr:jar:2.7.7:compile
> [INFO]| +- commons-lang:commons-lang:jar:2.6:compile
> [INFO]| \- commons-collections:commons-collections:jar:3.2.1:compile
> [INFO]+- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
> [INFO]\- net.sf.ehcache:ehcache-core:jar:2.4.4:compile
> {noformat}
> It looks like we don't need most of them.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (HADOOP-10710) hadoop.auth cookie is not properly constructed according to RFC2109
[
https://issues.apache.org/jira/browse/HADOOP-10710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Juan Yu reassigned HADOOP-10710:
Assignee: Juan Yu
> hadoop.auth cookie is not properly constructed according to RFC2109
> ---
>
> Key: HADOOP-10710
> URL: https://issues.apache.org/jira/browse/HADOOP-10710
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.0
>Reporter: Alejandro Abdelnur
>Assignee: Juan Yu
>
> It seems that HADOOP-10379 introduced a bug on how hadoop.auth cookies are
> being constructed.
> Before HADOOP-10379, cookies were constructed using Servlet's {{Cookie}}
> class and corresponding {{HttpServletResponse}} methods. This was taking care
> of setting attributes like 'Version=1' and double-quoting the cookie value if
> necessary.
> HADOOP-10379 changed the Cookie creation to use a {{StringBuillder}} and
> setting values and attributes by hand. This is not taking care of setting
> required attributes like Version and escaping the cookie value.
> While this is not breaking HadoopAuth {{AuthenticatedURL}} access, it is
> breaking access done using {{HtttpClient}}. I.e. Solr uses HttpClient and its
> access is broken since this change.
> It seems that HADOOP-10379 main objective was to set the 'secure' attribute.
> Note this can be done using the {{Cookie}} API.
> We should revert the cookie creation logic to use the {{Cookie}} API and take
> care of the security flag via {{setSecure(boolean)}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10711) Cleanup some extra dependencies from hadoop-auth
[
https://issues.apache.org/jira/browse/HADOOP-10711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033458#comment-14033458
]
Alejandro Abdelnur commented on HADOOP-10711:
-
+1 pending test-patch.
> Cleanup some extra dependencies from hadoop-auth
>
>
> Key: HADOOP-10711
> URL: https://issues.apache.org/jira/browse/HADOOP-10711
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.5.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: HADOOP-10711_branch-2.patch,
> HADOOP-10711_branch-2.patch, HADOOP-10711_trunk.patch,
> HADOOP-10711_trunk.patch
>
>
> HADOOP-10322 added {{apacheds-kerberos-codec}} as a dependency, which brought
> in some additional dependencies.
> {noformat}
> [INFO] \-
> org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
> [INFO]+- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
> [INFO]+- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-asn1-ber:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-i18n:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-ldap-model:jar:1.0.0-M20:compile
> [INFO]| +- org.apache.mina:mina-core:jar:2.0.0-M5:compile
> [INFO]| +- antlr:antlr:jar:2.7.7:compile
> [INFO]| +- commons-lang:commons-lang:jar:2.6:compile
> [INFO]| \- commons-collections:commons-collections:jar:3.2.1:compile
> [INFO]+- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
> [INFO]\- net.sf.ehcache:ehcache-core:jar:2.4.4:compile
> {noformat}
> It looks like we don't need most of them.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10711) Cleanup some extra dependencies from hadoop-auth
[
https://issues.apache.org/jira/browse/HADOOP-10711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated HADOOP-10711:
---
Attachment: HADOOP-10711_trunk.patch
HADOOP-10711_branch-2.patch
That's a good point. I've uploaded a new pair of patches that puts the version
in hadoop-project's pom and the exclusions in hadoop-auth's pom.
> Cleanup some extra dependencies from hadoop-auth
>
>
> Key: HADOOP-10711
> URL: https://issues.apache.org/jira/browse/HADOOP-10711
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.5.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: HADOOP-10711_branch-2.patch,
> HADOOP-10711_branch-2.patch, HADOOP-10711_trunk.patch,
> HADOOP-10711_trunk.patch
>
>
> HADOOP-10322 added {{apacheds-kerberos-codec}} as a dependency, which brought
> in some additional dependencies.
> {noformat}
> [INFO] \-
> org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
> [INFO]+- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
> [INFO]+- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-asn1-ber:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-i18n:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-ldap-model:jar:1.0.0-M20:compile
> [INFO]| +- org.apache.mina:mina-core:jar:2.0.0-M5:compile
> [INFO]| +- antlr:antlr:jar:2.7.7:compile
> [INFO]| +- commons-lang:commons-lang:jar:2.6:compile
> [INFO]| \- commons-collections:commons-collections:jar:3.2.1:compile
> [INFO]+- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
> [INFO]\- net.sf.ehcache:ehcache-core:jar:2.4.4:compile
> {noformat}
> It looks like we don't need most of them.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10683) Users authenticated with KERBEROS are recorded as being authenticated with SIMPLE
[ https://issues.apache.org/jira/browse/HADOOP-10683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033452#comment-14033452 ] Vinayakumar B commented on HADOOP-10683: Hi [~benoyantony], Sorry for coming late here as I was on travel, Nice catch and fix. Thanks for the patch and thanks Chris for review and commit . > Users authenticated with KERBEROS are recorded as being authenticated with > SIMPLE > - > > Key: HADOOP-10683 > URL: https://issues.apache.org/jira/browse/HADOOP-10683 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.0.0, 2.4.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Fix For: 3.0.0, 2.5.0 > > Attachments: HADOOP-10683.patch > > > We have enabled kerberos authentication in our clusters, but we see the > following in the log files > 2014-06-11 11:07:05,903 INFO SecurityLogger.org.apache.hadoop.ipc.Server: > Auth successful for x...@y.com (*auth:SIMPLE*) > 2014-06-11 11:07:05,914 INFO > SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: > Authorization successful for x...@y.com (auth:KERBEROS) for > protocol=interface > This is quite confusing for administrators. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10713) Document thread-safety of CryptoCodec#generateSecureRandom
[ https://issues.apache.org/jira/browse/HADOOP-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033447#comment-14033447 ] Yi Liu commented on HADOOP-10713: - Great, thanks [~andrew.wang], this can save some allocations. +1 to use byte[] as parameter. > Document thread-safety of CryptoCodec#generateSecureRandom > -- > > Key: HADOOP-10713 > URL: https://issues.apache.org/jira/browse/HADOOP-10713 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Trivial > Fix For: 3.0.0 > > Attachments: HADOOP-10713.001.patch > > > Random implementations have to deal with thread-safety; this should be > specified in the javadoc so implementors know to do this for CryptoCodec > subclasses. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10667) implement TCP connection reuse for native client
[ https://issues.apache.org/jira/browse/HADOOP-10667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-10667: -- Attachment: HADOOP-10667-pnative.003.patch rebase again after changes to the branch > implement TCP connection reuse for native client > > > Key: HADOOP-10667 > URL: https://issues.apache.org/jira/browse/HADOOP-10667 > Project: Hadoop Common > Issue Type: Sub-task > Components: native >Affects Versions: HADOOP-10388 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HADOOP-10667-pnative.001.patch, > HADOOP-10667-pnative.002.patch, HADOOP-10667-pnative.003.patch > > > The HDFS / YARN native clients should re-use TCP connections to avoid the > overhead of the three-way handshake, similar to how the Java code does. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10713) Document thread-safety of CryptoCodec#generateSecureRandom
[
https://issues.apache.org/jira/browse/HADOOP-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HADOOP-10713:
-
Attachment: HADOOP-10713.001.patch
Trivial patch attached. While I was looking at this, I also wondered whether
this API should be like {{SecureRandom#nextBytes(byte[])}}, taking a
{{byte[]}}, rather than taking an int numBytes and returning a new array.
[~hitliuyi] any thoughts? It might save us some allocations.
> Document thread-safety of CryptoCodec#generateSecureRandom
> --
>
> Key: HADOOP-10713
> URL: https://issues.apache.org/jira/browse/HADOOP-10713
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134)
>Reporter: Andrew Wang
>Assignee: Andrew Wang
>Priority: Trivial
> Fix For: 3.0.0
>
> Attachments: HADOOP-10713.001.patch
>
>
> Random implementations have to deal with thread-safety; this should be
> specified in the javadoc so implementors know to do this for CryptoCodec
> subclasses.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10711) Cleanup some extra dependencies from hadoop-auth
[
https://issues.apache.org/jira/browse/HADOOP-10711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033439#comment-14033439
]
Alejandro Abdelnur commented on HADOOP-10711:
-
I think the exclusions should be done directly in the hadoop-auth module. Else
it may trigger excluding those dependencies for the hadoop-mini-kdc. the
version, yes it belongs to the hadoop-project POM.
> Cleanup some extra dependencies from hadoop-auth
>
>
> Key: HADOOP-10711
> URL: https://issues.apache.org/jira/browse/HADOOP-10711
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.5.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: HADOOP-10711_branch-2.patch, HADOOP-10711_trunk.patch
>
>
> HADOOP-10322 added {{apacheds-kerberos-codec}} as a dependency, which brought
> in some additional dependencies.
> {noformat}
> [INFO] \-
> org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
> [INFO]+- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
> [INFO]+- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-asn1-ber:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-i18n:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-ldap-model:jar:1.0.0-M20:compile
> [INFO]| +- org.apache.mina:mina-core:jar:2.0.0-M5:compile
> [INFO]| +- antlr:antlr:jar:2.7.7:compile
> [INFO]| +- commons-lang:commons-lang:jar:2.6:compile
> [INFO]| \- commons-collections:commons-collections:jar:3.2.1:compile
> [INFO]+- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
> [INFO]\- net.sf.ehcache:ehcache-core:jar:2.4.4:compile
> {noformat}
> It looks like we don't need most of them.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10713) Document thread-safety of CryptoCodec#generateSecureRandom
Andrew Wang created HADOOP-10713: Summary: Document thread-safety of CryptoCodec#generateSecureRandom Key: HADOOP-10713 URL: https://issues.apache.org/jira/browse/HADOOP-10713 Project: Hadoop Common Issue Type: Sub-task Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) Reporter: Andrew Wang Assignee: Andrew Wang Priority: Trivial Random implementations have to deal with thread-safety; this should be specified in the javadoc so implementors know to do this for CryptoCodec subclasses. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-10706) Fix initialization of hrpc_sync_ctx
[
https://issues.apache.org/jira/browse/HADOOP-10706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe resolved HADOOP-10706.
---
Resolution: Fixed
Fix Version/s: HADOOP-10388
Target Version/s: HADOOP-10388
> Fix initialization of hrpc_sync_ctx
> ---
>
> Key: HADOOP-10706
> URL: https://issues.apache.org/jira/browse/HADOOP-10706
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Binglin Chang
>Assignee: Binglin Chang
> Fix For: HADOOP-10388
>
> Attachments: HADOOP-10706.v1.patch
>
>
> 1.
> {code}
> memset(&ctx, 0, sizeof(ctx));
> return ctx;
> {code}
> Doing this will alway make return value to 0
> 2.
> hrpc_release_sync_ctx should changed to hrpc_proxy_release_sync_ctx, all the
> functions in this .h/.c file follow this rule
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10706) fix some bug related to hrpc_sync_ctx
[
https://issues.apache.org/jira/browse/HADOOP-10706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033431#comment-14033431
]
Colin Patrick McCabe commented on HADOOP-10706:
---
Thanks, Binglin. +1.
> fix some bug related to hrpc_sync_ctx
> -
>
> Key: HADOOP-10706
> URL: https://issues.apache.org/jira/browse/HADOOP-10706
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Binglin Chang
>Assignee: Binglin Chang
> Attachments: HADOOP-10706.v1.patch
>
>
> 1.
> {code}
> memset(&ctx, 0, sizeof(ctx));
> return ctx;
> {code}
> Doing this will alway make return value to 0
> 2.
> hrpc_release_sync_ctx should changed to hrpc_proxy_release_sync_ctx, all the
> functions in this .h/.c file follow this rule
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10706) Fix initialization of hrpc_sync_ctx
[
https://issues.apache.org/jira/browse/HADOOP-10706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HADOOP-10706:
--
Summary: Fix initialization of hrpc_sync_ctx (was: fix some bug related to
hrpc_sync_ctx)
> Fix initialization of hrpc_sync_ctx
> ---
>
> Key: HADOOP-10706
> URL: https://issues.apache.org/jira/browse/HADOOP-10706
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Binglin Chang
>Assignee: Binglin Chang
> Attachments: HADOOP-10706.v1.patch
>
>
> 1.
> {code}
> memset(&ctx, 0, sizeof(ctx));
> return ctx;
> {code}
> Doing this will alway make return value to 0
> 2.
> hrpc_release_sync_ctx should changed to hrpc_proxy_release_sync_ctx, all the
> functions in this .h/.c file follow this rule
--
This message was sent by Atlassian JIRA
(v6.2#6252)
common-issues@hadoop.apache.org
[
https://issues.apache.org/jira/browse/HADOOP-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033430#comment-14033430
]
Colin Patrick McCabe commented on HADOOP-10636:
---
{code}
uv_sem_t sem, sem1;
{code}
It seems a bit confusing to have both a sem and sem1. Maybe name them 1 and 2,
or just use the same sem throughout?
{code}
void hrpc_call_deliver_resp_test_cb(struct hrpc_response *resp,
struct hadoop_err *err, void *cb_data)
{
uv_sem_t *sem = cb_data;
EXPECT_NONNULL(resp);
EXPECT_NULL(err);
free(resp->base);
uv_sem_post(sem);
}
{code}
We can't use {{EXPECT_NONNULL}} here, since it tries to return if the condition
is wrong, and this function is declared {{void}}. One easy way around this is
doing something like this:
{code}
void hrpc_call_deliver_resp_test_cb(struct hrpc_response *resp,
struct hadoop_err *err, void *cb_data)
{
if resp_test_cb(resp, err, data) {
abort();
}
}
static int resp_test_cb(struct hrpc_response *resp,
struct hadoop_err *err, void *cb_data)
{
do EXPECT, etc.
}
{code}
You could also set an error code from the callback (like you'd do in "real
code") that you check in {{main}}. But this way is a little simpler, maybe.
Looks good aside from that.
> Native Hadoop Client:add unit test case for call&client_id
> --
>
> Key: HADOOP-10636
> URL: https://issues.apache.org/jira/browse/HADOOP-10636
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: HADOOP-10388
>Reporter: Wenwu Peng
>Assignee: Wenwu Peng
> Attachments: HADOOP-10636-pnative.001.patch,
> HADOOP-10636-pnative.002.patch, HADOOP-10636-pnative.003.patch,
> HADOOP-10636-pnative.004.patch, HADOOP-10636-pnative.005.patch,
> HADOOP-10636-pnative.006.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10705) Fix namenode-rpc-unit warning reported by memory leak check tool(valgrind)
[
https://issues.apache.org/jira/browse/HADOOP-10705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033420#comment-14033420
]
Colin Patrick McCabe commented on HADOOP-10705:
---
You're hitting a bug in Ubuntu. See
https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/119029
Another way of seeing this is by looking at this call stack:
{code}
==12062== 16 bytes in 1 blocks are indirectly lost in loss record 3 of 12
==12062==at 0x4C2B6CD: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==12062==by 0x557EB99: __nss_lookup_function (nsswitch.c:456)
==12062==by 0x6048672: ???
==12062==by 0x553744C: getpwuid_r@@GLIBC_2.2.5 (getXXbyYY_r.c:256)
==12062==by 0x42681E: geteuid_string (user.c:67)
==12062==by 0x425ABD: main (namenode-rpc-unit.c:71)
{code}
If you check the man page, {{getpwuid_r}} does not allocate memory that the
user needs to clean up. So there is no way we could be to blame for this.
Sometimes, system libraries have little leaks in them like this. Normally, the
Linux distro will add "suppression files" to avoid having valgrind spew errors
when nothing is actually wrong. It looks like in this case, your Linux distro
doesn't have such a suppression. So you can either add one manually, or
upgrade to a newer distro without this bug.
> Fix namenode-rpc-unit warning reported by memory leak check tool(valgrind)
> ---
>
> Key: HADOOP-10705
> URL: https://issues.apache.org/jira/browse/HADOOP-10705
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: HADOOP-10388
>Reporter: Wenwu Peng
> Attachments: vargrind.log
>
>
> Rum valgrind to check memory leak for namenode-rpc.unit.
> There are many warning, need to fix it.
> valgrind --tool=memcheck --leak-check=full --show-reachable=yes
> ./namenode-rpc-unit
> ==10462== Memcheck, a memory error detector
> ==10462== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
> ==10462== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
> ==10462== Command: ./namenode-rpc-unit
> ==10462==
> ==10462== HEAP SUMMARY:
> ==10462== in use at exit: 428 bytes in 12 blocks
> ==10462== total heap usage: 91 allocs, 79 frees, 16,056 bytes allocated
> ==10462==
> ==10462== 16 bytes in 1 blocks are indirectly lost in loss record 1 of 12
> ==10462==at 0x4C2B6CD: malloc (in
> /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==10462==by 0x557EB99: __nss_lookup_function (nsswitch.c:456)
> ==10462==by 0x604863E: ???
> ==10462==by 0x553744C: getpwuid_r@@GLIBC_2.2.5 (getXXbyYY_r.c:256)
> ==10462==by 0x42681E: geteuid_string (user.c:67)
> ==10462==by 0x425ABD: main (namenode-rpc-unit.c:71)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10557) FsShell -cp -p does not preserve extended ACLs
[
https://issues.apache.org/jira/browse/HADOOP-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033414#comment-14033414
]
Hadoop QA commented on HADOOP-10557:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650698/HADOOP-10557.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4079//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4079//console
This message is automatically generated.
> FsShell -cp -p does not preserve extended ACLs
> --
>
> Key: HADOOP-10557
> URL: https://issues.apache.org/jira/browse/HADOOP-10557
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Attachments: HADOOP-10557.2.patch, HADOOP-10557.3.patch,
> HADOOP-10557.4.patch, HADOOP-10557.5.patch, HADOOP-10557.patch
>
>
> This issue tracks enhancing FsShell cp to add a new command-line option (-pa)
> for preserving extended ACLs.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10389) Native RPCv9 client
[
https://issues.apache.org/jira/browse/HADOOP-10389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033404#comment-14033404
]
Colin Patrick McCabe commented on HADOOP-10389:
---
To me this is not that interesting. There are a lot of clients already out
there in various languages... C, C++, even golang. Your code is much smaller
because you didn't implement generation of type-safe RPC wrappers (you just
wrote out everything manually in {{HdfsImpl::Exists}}), didn't integrate the
existing JNIFS code, are not parsing the config XMLs, don't handle errors, etc.
etc. {{boost::asio}} doesn't provide equivalent functionality to {{libuv}}.
It does wrap a lot of network stuff, but doesn't handle things like condition
variables, signals, etc. etc. It uses exceptions, which are problematic. Yes,
you can use boost for some of that stuff, but the boost versions are often not
that good. For example, the boost condition variable actually can let your
thread wake up without recovering the lock (it then has to sleep again). Boost
in general has a versioning problem which is really bad for libraries. If your
library links against version X of boost, any application linking to you needs
to use the same version... otherwise bad stuff happens. You could link it
statically, but it might be too big for that to work (I've never tried).
If you're interested in something even smaller and more minimal, you can check
out this: https://github.com/toddlipcon/libhrpc/ Though I think that that is
for an out-of-date version of hadoop RPC at this point.
> Native RPCv9 client
> ---
>
> Key: HADOOP-10389
> URL: https://issues.apache.org/jira/browse/HADOOP-10389
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: HADOOP-10388
>Reporter: Binglin Chang
>Assignee: Colin Patrick McCabe
> Attachments: HADOOP-10388.001.patch,
> HADOOP-10389-alternative.000.patch, HADOOP-10389.002.patch,
> HADOOP-10389.004.patch, HADOOP-10389.005.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10699) Fix build native library on mac osx
[ https://issues.apache.org/jira/browse/HADOOP-10699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033360#comment-14033360 ] Hudson commented on HADOOP-10699: - SUCCESS: Integrated in Hadoop-trunk-Commit #5717 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5717/]) HADOOP-10699. Fix build native library on mac osx. Contributed by Binglin Chang (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1603042) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c > Fix build native library on mac osx > --- > > Key: HADOOP-10699 > URL: https://issues.apache.org/jira/browse/HADOOP-10699 > Project: Hadoop Common > Issue Type: Bug >Reporter: Kirill A. Korinskiy >Assignee: Binglin Chang > Fix For: 3.0.0, 2.5.0 > > Attachments: HADOOP-10699-common.v3.patch, > HADOOP-9648-native-osx.1.0.4.patch, HADOOP-9648-native-osx.1.1.2.patch, > HADOOP-9648-native-osx.1.2.0.patch, > HADOOP-9648-native-osx.2.0.5-alpha-rc1.patch, HADOOP-9648.v2.patch > > > Some patches for fixing build a hadoop native library on os x 10.7/10.8. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10699) Fix build native library on mac osx
[ https://issues.apache.org/jira/browse/HADOOP-10699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated HADOOP-10699: Resolution: Fixed Fix Version/s: 2.5.0 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks, Binglin! I committed this to trunk and branch-2. > Fix build native library on mac osx > --- > > Key: HADOOP-10699 > URL: https://issues.apache.org/jira/browse/HADOOP-10699 > Project: Hadoop Common > Issue Type: Bug >Reporter: Kirill A. Korinskiy >Assignee: Binglin Chang > Fix For: 3.0.0, 2.5.0 > > Attachments: HADOOP-10699-common.v3.patch, > HADOOP-9648-native-osx.1.0.4.patch, HADOOP-9648-native-osx.1.1.2.patch, > HADOOP-9648-native-osx.1.2.0.patch, > HADOOP-9648-native-osx.2.0.5-alpha-rc1.patch, HADOOP-9648.v2.patch > > > Some patches for fixing build a hadoop native library on os x 10.7/10.8. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10699) Fix build native library on mac osx
[ https://issues.apache.org/jira/browse/HADOOP-10699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033341#comment-14033341 ] Jason Lowe commented on HADOOP-10699: - +1 lgtm, committing this. > Fix build native library on mac osx > --- > > Key: HADOOP-10699 > URL: https://issues.apache.org/jira/browse/HADOOP-10699 > Project: Hadoop Common > Issue Type: Bug >Reporter: Kirill A. Korinskiy >Assignee: Binglin Chang > Attachments: HADOOP-10699-common.v3.patch, > HADOOP-9648-native-osx.1.0.4.patch, HADOOP-9648-native-osx.1.1.2.patch, > HADOOP-9648-native-osx.1.2.0.patch, > HADOOP-9648-native-osx.2.0.5-alpha-rc1.patch, HADOOP-9648.v2.patch > > > Some patches for fixing build a hadoop native library on os x 10.7/10.8. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10711) Cleanup some extra dependencies from hadoop-auth
[
https://issues.apache.org/jira/browse/HADOOP-10711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033290#comment-14033290
]
Hadoop QA commented on HADOOP-10711:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650695/HADOOP-10711_trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-auth.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4078//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4078//console
This message is automatically generated.
> Cleanup some extra dependencies from hadoop-auth
>
>
> Key: HADOOP-10711
> URL: https://issues.apache.org/jira/browse/HADOOP-10711
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.5.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: HADOOP-10711_branch-2.patch, HADOOP-10711_trunk.patch
>
>
> HADOOP-10322 added {{apacheds-kerberos-codec}} as a dependency, which brought
> in some additional dependencies.
> {noformat}
> [INFO] \-
> org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
> [INFO]+- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
> [INFO]+- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-asn1-ber:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-i18n:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-ldap-model:jar:1.0.0-M20:compile
> [INFO]| +- org.apache.mina:mina-core:jar:2.0.0-M5:compile
> [INFO]| +- antlr:antlr:jar:2.7.7:compile
> [INFO]| +- commons-lang:commons-lang:jar:2.6:compile
> [INFO]| \- commons-collections:commons-collections:jar:3.2.1:compile
> [INFO]+- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
> [INFO]\- net.sf.ehcache:ehcache-core:jar:2.4.4:compile
> {noformat}
> It looks like we don't need most of them.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-9361) Strictly define the expected behavior of filesystem APIs and write tests to verify compliance
[
https://issues.apache.org/jira/browse/HADOOP-9361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033279#comment-14033279
]
Andrew Wang commented on HADOOP-9361:
-
Okay, I can see the value in that. Maybe there's some creative way to de-dupe
some of the repeated @BeforeClass and @AfterClass type stuff, but it seems hard
given the lack of multiple inheritance. I like your naming proposals (tests and
the xml).
I'll try to fire up some of these different cloud offerings in the meantime. I
guess it'd be good for at least one other person to go through the exercise of
running them.
> Strictly define the expected behavior of filesystem APIs and write tests to
> verify compliance
> -
>
> Key: HADOOP-9361
> URL: https://issues.apache.org/jira/browse/HADOOP-9361
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, test
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-9361-001.patch, HADOOP-9361-002.patch,
> HADOOP-9361-003.patch, HADOOP-9361-004.patch, HADOOP-9361-005.patch,
> HADOOP-9361-006.patch, HADOOP-9361-007.patch, HADOOP-9361-008.patch,
> HADOOP-9361-009.patch, HADOOP-9361-011.patch, HADOOP-9361-012.patch,
> HADOOP-9361-013.patch, HADOOP-9361-014.patch, HADOOP-9361-015.patch,
> HADOOP-9361.awang-addendum.patch
>
>
> {{FileSystem}} and {{FileContract}} aren't tested rigorously enough -while
> HDFS gets tested downstream, other filesystems, such as blobstore bindings,
> don't.
> The only tests that are common are those of {{FileSystemContractTestBase}},
> which HADOOP-9258 shows is incomplete.
> I propose
> # writing more tests which clarify expected behavior
> # testing operations in the interface being in their own JUnit4 test classes,
> instead of one big test suite.
> # Having each FS declare via a properties file what behaviors they offer,
> such as atomic-rename, atomic-delete, umask, immediate-consistency -test
> methods can downgrade to skipped test cases if a feature is missing.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10712) Add support for accessing the NFS gateway from the AIX NFS client
Aaron T. Myers created HADOOP-10712: --- Summary: Add support for accessing the NFS gateway from the AIX NFS client Key: HADOOP-10712 URL: https://issues.apache.org/jira/browse/HADOOP-10712 Project: Hadoop Common Issue Type: Bug Components: nfs Affects Versions: 2.4.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers We've identified two issues when trying to access the HDFS NFS Gateway from an AIX NFS client: # In the case of COMMITs, the AIX NFS client will always send 4096, or a multiple of the page size, for the offset to be committed, even if fewer bytes than this have ever, or will ever, be written to the file. This will cause a write to a file from the AIX NFS client to hang on close unless the size of that file is a multiple of 4096. # In the case of READDIR and READDIRPLUS, the AIX NFS client will send the same cookie verifier for a given directory seemingly forever after that directory is first accessed over NFS, instead of getting a new cookie verifier for every set of incremental readdir calls. This means that if a directory's mtime ever changes, the FS must be unmounted/remounted before readdir calls on that dir from AIX will ever succeed again. >From my interpretation of RFC-1813, the NFS Gateway is in fact doing the >correct thing in both cases, but we can introduce simple changes on the NFS >Gateway side to be able to optionally work around these incompatibilities. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10557) FsShell -cp -p does not preserve extended ACLs
[ https://issues.apache.org/jira/browse/HADOOP-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-10557: --- Attachment: HADOOP-10557.5.patch Thanks [~cnauroth] for the comment! Updated the patch. > FsShell -cp -p does not preserve extended ACLs > -- > > Key: HADOOP-10557 > URL: https://issues.apache.org/jira/browse/HADOOP-10557 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA > Attachments: HADOOP-10557.2.patch, HADOOP-10557.3.patch, > HADOOP-10557.4.patch, HADOOP-10557.5.patch, HADOOP-10557.patch > > > This issue tracks enhancing FsShell cp to add a new command-line option (-pa) > for preserving extended ACLs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10711) Cleanup some extra dependencies from hadoop-auth
[
https://issues.apache.org/jira/browse/HADOOP-10711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated HADOOP-10711:
---
Attachment: HADOOP-10711_trunk.patch
HADOOP-10711_branch-2.patch
The patch differs slightly between branch-2 and trunk because of other added
dependencies in trunk. Uploading a version of the patch for each.
> Cleanup some extra dependencies from hadoop-auth
>
>
> Key: HADOOP-10711
> URL: https://issues.apache.org/jira/browse/HADOOP-10711
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.5.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: HADOOP-10711_branch-2.patch, HADOOP-10711_trunk.patch
>
>
> HADOOP-10322 added {{apacheds-kerberos-codec}} as a dependency, which brought
> in some additional dependencies.
> {noformat}
> [INFO] \-
> org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
> [INFO]+- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
> [INFO]+- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-asn1-ber:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-i18n:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-ldap-model:jar:1.0.0-M20:compile
> [INFO]| +- org.apache.mina:mina-core:jar:2.0.0-M5:compile
> [INFO]| +- antlr:antlr:jar:2.7.7:compile
> [INFO]| +- commons-lang:commons-lang:jar:2.6:compile
> [INFO]| \- commons-collections:commons-collections:jar:3.2.1:compile
> [INFO]+- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
> [INFO]\- net.sf.ehcache:ehcache-core:jar:2.4.4:compile
> {noformat}
> It looks like we don't need most of them.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10711) Cleanup some extra dependencies from hadoop-auth
[
https://issues.apache.org/jira/browse/HADOOP-10711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated HADOOP-10711:
---
Attachment: (was: HADOOP-10711.patch)
> Cleanup some extra dependencies from hadoop-auth
>
>
> Key: HADOOP-10711
> URL: https://issues.apache.org/jira/browse/HADOOP-10711
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.5.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: HADOOP-10711_branch-2.patch, HADOOP-10711_trunk.patch
>
>
> HADOOP-10322 added {{apacheds-kerberos-codec}} as a dependency, which brought
> in some additional dependencies.
> {noformat}
> [INFO] \-
> org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
> [INFO]+- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
> [INFO]+- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-asn1-ber:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-i18n:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-ldap-model:jar:1.0.0-M20:compile
> [INFO]| +- org.apache.mina:mina-core:jar:2.0.0-M5:compile
> [INFO]| +- antlr:antlr:jar:2.7.7:compile
> [INFO]| +- commons-lang:commons-lang:jar:2.6:compile
> [INFO]| \- commons-collections:commons-collections:jar:3.2.1:compile
> [INFO]+- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
> [INFO]\- net.sf.ehcache:ehcache-core:jar:2.4.4:compile
> {noformat}
> It looks like we don't need most of them.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10673) Update rpc metrics when the call throws an exception
[
https://issues.apache.org/jira/browse/HADOOP-10673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033260#comment-14033260
]
Hadoop QA commented on HADOOP-10673:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650684/HADOOP-10673-6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4075//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4075//console
This message is automatically generated.
> Update rpc metrics when the call throws an exception
>
>
> Key: HADOOP-10673
> URL: https://issues.apache.org/jira/browse/HADOOP-10673
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HADOOP-10673-2.patch, HADOOP-10673-4.patch,
> HADOOP-10673-5.patch, HADOOP-10673-6.patch, HADOOP-10673.patch
>
>
> Currently RPC metrics isn't updated when the call throws an exception. We can
> either update the existing metrics or have a new set of metrics in the case
> of exception.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10695) KMSClientProvider should respect a configurable timeout.
[ https://issues.apache.org/jira/browse/HADOOP-10695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-10695: - Assignee: Mike Yoder > KMSClientProvider should respect a configurable timeout. > > > Key: HADOOP-10695 > URL: https://issues.apache.org/jira/browse/HADOOP-10695 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Mike Yoder > > It'd be good if KMSClientProvider used a timeout, so it doesn't hang forever > if the KMServer is down. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9361) Strictly define the expected behavior of filesystem APIs and write tests to verify compliance
[
https://issues.apache.org/jira/browse/HADOOP-9361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033255#comment-14033255
]
Steve Loughran commented on HADOOP-9361:
that was a bit oddly formatted, text editor wraps vs JIRA ones.
I was trying to say in having separate tests/feature is that you can work on a
feature at a time, rather than run the subclass of FileSystemContractBaseTest
you have today, which take long enough to make a decent coffee -or, in the case
of object stores, walk to a nearby cafe and back-
> Strictly define the expected behavior of filesystem APIs and write tests to
> verify compliance
> -
>
> Key: HADOOP-9361
> URL: https://issues.apache.org/jira/browse/HADOOP-9361
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, test
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-9361-001.patch, HADOOP-9361-002.patch,
> HADOOP-9361-003.patch, HADOOP-9361-004.patch, HADOOP-9361-005.patch,
> HADOOP-9361-006.patch, HADOOP-9361-007.patch, HADOOP-9361-008.patch,
> HADOOP-9361-009.patch, HADOOP-9361-011.patch, HADOOP-9361-012.patch,
> HADOOP-9361-013.patch, HADOOP-9361-014.patch, HADOOP-9361-015.patch,
> HADOOP-9361.awang-addendum.patch
>
>
> {{FileSystem}} and {{FileContract}} aren't tested rigorously enough -while
> HDFS gets tested downstream, other filesystems, such as blobstore bindings,
> don't.
> The only tests that are common are those of {{FileSystemContractTestBase}},
> which HADOOP-9258 shows is incomplete.
> I propose
> # writing more tests which clarify expected behavior
> # testing operations in the interface being in their own JUnit4 test classes,
> instead of one big test suite.
> # Having each FS declare via a properties file what behaviors they offer,
> such as atomic-rename, atomic-delete, umask, immediate-consistency -test
> methods can downgrade to skipped test cases if a feature is missing.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10696) Add optional tags to KeyProvider Options and Metadata
[
https://issues.apache.org/jira/browse/HADOOP-10696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033246#comment-14033246
]
Charles Lamb commented on HADOOP-10696:
---
Hi Tucu,
Just a few nits.
As has been previously mentioned, there are whitespace issues. There are a
bunch of places where there are trailing whitespace in the lines that should be
removed.
Is there any reason that KeyProvider.TAGS_FIELD can't be shared with
KMSRESTConstants.TAGS_FIELD?
TestKeyProvider.java: s/String[]{"a"/String[] {"a/
> Add optional tags to KeyProvider Options and Metadata
> -
>
> Key: HADOOP-10696
> URL: https://issues.apache.org/jira/browse/HADOOP-10696
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 3.0.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: HADOOP-10696.patch, HADOOP-10696.patch,
> HADOOP-10696.patch
>
>
> In addition to having an optional description, KeyProvider Options and
> Metadata should support optional tags to help categorize keys.
> This would be useful for visualization purposes.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-9361) Strictly define the expected behavior of filesystem APIs and write tests to verify compliance
[
https://issues.apache.org/jira/browse/HADOOP-9361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033241#comment-14033241
]
Steve Loughran commented on HADOOP-9361:
h3. Validating S3/Swift tests.
{quote}
It's still hard for me to run the S3 / Swift tests since I'd have to get
credentials,
{quote}
These tests cost ~$0 to run as they don't persist data; you can use your own
private
logins. I'd recommend >1 swift provider (I use RAX US, RAX UK and HP Cloud for
their different
auth, consistency and throttling). S3 has different consistency models for
US-east (worst) and
everywhere else. You need to create buckets in different places there.
h3. test groupings
{quote}
Only big feedback I have, is there some way we could have a single parent test
for each FS that runs all its contracts? All the concrete TestFooBarContract
classes are basically copy paste, I'd like to get rid of them. It's also harder
for poor me to run all the HDFS contract tests
{quote}
-1 to any unified supertest, as it doesn't isolate things the way having tests
per operation does. You get to
implement the features you want, and don't do a test for a RootContract if you
don't want your root FS deleted,
Append if you don't do append. etc. You also get to debug something
{{TestHDFSRenameContract}}
# We can do well known names like {{TestHDFSContract}}, I can see that I've
already done this for
the {{TestNativeS3*}} tests.
# There's Junit Categories[
http://junit.org/javadoc/4.11/org/junit/experimental/categories/Categories.html].
Maven [does now support
this|http://stackoverflow.com/questions/3100924/how-to-run-junit-tests-by-category-in-maven]
FWIW, categorizing the Hadoop tests would be a good idea at some point in the
future anyway "fast, slow, scale"
# There's JUnit suites -but use them and you will end up with your tests
running if the suite test runs -and
again if run individually.
I'd go for the naming, if everyone is happy with that and come up with a good
name for the HDFS tests that don't class with other
things. How about {{TestHDFSContract*}}?
h3. FSExceptionMessages
I picked them up from one file (HDFS?) and used there.
h3. Everything else
I'll do those...
How about {{"rejects-seek-past-eof"}}?
> Strictly define the expected behavior of filesystem APIs and write tests to
> verify compliance
> -
>
> Key: HADOOP-9361
> URL: https://issues.apache.org/jira/browse/HADOOP-9361
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, test
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-9361-001.patch, HADOOP-9361-002.patch,
> HADOOP-9361-003.patch, HADOOP-9361-004.patch, HADOOP-9361-005.patch,
> HADOOP-9361-006.patch, HADOOP-9361-007.patch, HADOOP-9361-008.patch,
> HADOOP-9361-009.patch, HADOOP-9361-011.patch, HADOOP-9361-012.patch,
> HADOOP-9361-013.patch, HADOOP-9361-014.patch, HADOOP-9361-015.patch,
> HADOOP-9361.awang-addendum.patch
>
>
> {{FileSystem}} and {{FileContract}} aren't tested rigorously enough -while
> HDFS gets tested downstream, other filesystems, such as blobstore bindings,
> don't.
> The only tests that are common are those of {{FileSystemContractTestBase}},
> which HADOOP-9258 shows is incomplete.
> I propose
> # writing more tests which clarify expected behavior
> # testing operations in the interface being in their own JUnit4 test classes,
> instead of one big test suite.
> # Having each FS declare via a properties file what behaviors they offer,
> such as atomic-rename, atomic-delete, umask, immediate-consistency -test
> methods can downgrade to skipped test cases if a feature is missing.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10711) Cleanup some extra dependencies from hadoop-auth
[
https://issues.apache.org/jira/browse/HADOOP-10711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033237#comment-14033237
]
Hadoop QA commented on HADOOP-10711:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650687/HADOOP-10711.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4077//console
This message is automatically generated.
> Cleanup some extra dependencies from hadoop-auth
>
>
> Key: HADOOP-10711
> URL: https://issues.apache.org/jira/browse/HADOOP-10711
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.5.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: HADOOP-10711.patch
>
>
> HADOOP-10322 added {{apacheds-kerberos-codec}} as a dependency, which brought
> in some additional dependencies.
> {noformat}
> [INFO] \-
> org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
> [INFO]+- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
> [INFO]+- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-asn1-ber:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-i18n:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-ldap-model:jar:1.0.0-M20:compile
> [INFO]| +- org.apache.mina:mina-core:jar:2.0.0-M5:compile
> [INFO]| +- antlr:antlr:jar:2.7.7:compile
> [INFO]| +- commons-lang:commons-lang:jar:2.6:compile
> [INFO]| \- commons-collections:commons-collections:jar:3.2.1:compile
> [INFO]+- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
> [INFO]\- net.sf.ehcache:ehcache-core:jar:2.4.4:compile
> {noformat}
> It looks like we don't need most of them.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-8943) Support multiple group mapping providers
[
https://issues.apache.org/jira/browse/HADOOP-8943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033234#comment-14033234
]
Hadoop QA commented on HADOOP-8943:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12561104/HADOOP-8943.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4076//console
This message is automatically generated.
> Support multiple group mapping providers
>
>
> Key: HADOOP-8943
> URL: https://issues.apache.org/jira/browse/HADOOP-8943
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Fix For: 2.5.0
>
> Attachments: HADOOP-8943.patch, HADOOP-8943.patch, HADOOP-8943.patch
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> Discussed with Natty about LdapGroupMapping, we need to improve it so that:
> 1. It's possible to do different group mapping for different
> users/principals. For example, AD user should go to LdapGroupMapping service
> for group, but service principals such as hdfs, mapred can still use the
> default one ShellBasedUnixGroupsMapping;
> 2. Multiple ADs can be supported to do LdapGroupMapping;
> 3. It's possible to configure what kind of users/principals (regarding
> domain/realm is an option) should use which group mapping service/mechanism.
> 4. It's possible to configure and combine multiple existing mapping providers
> without writing codes implementing new one.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10690) Lack of synchronization on access to InputStream in NativeAzureFileSystem#NativeAzureFsInputStream#close()
[
https://issues.apache.org/jira/browse/HADOOP-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033230#comment-14033230
]
Hadoop QA commented on HADOOP-10690:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650666/HADOOP-10690.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-azure.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4074//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4074//console
This message is automatically generated.
> Lack of synchronization on access to InputStream in
> NativeAzureFileSystem#NativeAzureFsInputStream#close()
> --
>
> Key: HADOOP-10690
> URL: https://issues.apache.org/jira/browse/HADOOP-10690
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Chen He
>Priority: Minor
> Attachments: HADOOP-10690.patch
>
>
> {code}
> public void close() throws IOException {
> in.close();
> }
> {code}
> The close() method should be protected by synchronized keyword.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10666) Remove Copyright /d/d/d/d Apache Software Foundation from the source files license header
[ https://issues.apache.org/jira/browse/HADOOP-10666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033226#comment-14033226 ] Henry Saputra commented on HADOOP-10666: Thanks Andrew! Really appreciate it. > Remove Copyright /d/d/d/d Apache Software Foundation from the source files > license header > - > > Key: HADOOP-10666 > URL: https://issues.apache.org/jira/browse/HADOOP-10666 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 2.4.0 >Reporter: Henry Saputra >Assignee: Henry Saputra >Priority: Minor > Fix For: 2.5.0 > > Attachments: HADOOP-10666.patch > > > Some of the files have "Copyright 2011 Apache Software Foundation" in the > license header comment. > This is not the right license header per ASF rule [1] > From the ASF header rule [1] : > "Source File Headers for Code Developed at the ASF" > "This section refers only to works submitted directly to the ASF by the > copyright owner or owner's agent." > Seems like the one without Copyright notice is used for software developed > directly by ASF which Apache Hadoop is. > When it was under external organization it does need to use the one from ASF > 2.0 license [2] which requires Copyright included. > [1] http://www.apache.org/legal/src-headers.html > [2] http://www.apache.org/licenses/LICENSE-2.0 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10711) Cleanup some extra dependencies from hadoop-auth
[
https://issues.apache.org/jira/browse/HADOOP-10711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated HADOOP-10711:
---
Attachment: HADOOP-10711.patch
The patch excludes the unnecessary dependencies. It also moves the version and
dependency management stuff to the hadoop-project pom where it normally goes
instead of the hadoop-auth pom.
> Cleanup some extra dependencies from hadoop-auth
>
>
> Key: HADOOP-10711
> URL: https://issues.apache.org/jira/browse/HADOOP-10711
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.5.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: HADOOP-10711.patch
>
>
> HADOOP-10322 added {{apacheds-kerberos-codec}} as a dependency, which brought
> in some additional dependencies.
> {noformat}
> [INFO] \-
> org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
> [INFO]+- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
> [INFO]+- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-asn1-ber:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-i18n:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-ldap-model:jar:1.0.0-M20:compile
> [INFO]| +- org.apache.mina:mina-core:jar:2.0.0-M5:compile
> [INFO]| +- antlr:antlr:jar:2.7.7:compile
> [INFO]| +- commons-lang:commons-lang:jar:2.6:compile
> [INFO]| \- commons-collections:commons-collections:jar:3.2.1:compile
> [INFO]+- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
> [INFO]\- net.sf.ehcache:ehcache-core:jar:2.4.4:compile
> {noformat}
> It looks like we don't need most of them.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10711) Cleanup some extra dependencies from hadoop-auth
[
https://issues.apache.org/jira/browse/HADOOP-10711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated HADOOP-10711:
---
Status: Patch Available (was: Open)
> Cleanup some extra dependencies from hadoop-auth
>
>
> Key: HADOOP-10711
> URL: https://issues.apache.org/jira/browse/HADOOP-10711
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.5.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: HADOOP-10711.patch
>
>
> HADOOP-10322 added {{apacheds-kerberos-codec}} as a dependency, which brought
> in some additional dependencies.
> {noformat}
> [INFO] \-
> org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
> [INFO]+- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
> [INFO]+- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-asn1-ber:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-i18n:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-ldap-model:jar:1.0.0-M20:compile
> [INFO]| +- org.apache.mina:mina-core:jar:2.0.0-M5:compile
> [INFO]| +- antlr:antlr:jar:2.7.7:compile
> [INFO]| +- commons-lang:commons-lang:jar:2.6:compile
> [INFO]| \- commons-collections:commons-collections:jar:3.2.1:compile
> [INFO]+- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
> [INFO]\- net.sf.ehcache:ehcache-core:jar:2.4.4:compile
> {noformat}
> It looks like we don't need most of them.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-8943) Support multiple group mapping providers
[ https://issues.apache.org/jira/browse/HADOOP-8943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033212#comment-14033212 ] Brandon Li commented on HADOOP-8943: The patch looks pretty nice. A few comments: 1. for the change in the public interface GroupMappingServiceProvider.java. We can use the same config key already defined in CommonCofigurationKeysPublic instead of defining a new one: + public static final String GROUP_MAPPING_CONFIG_PREFIX = "hadoop.security.group.mapping"; 2. when multiple domains are configured and LdapGroupsMapping is used multiple times for all the domains(as in the example given in core-site.xml). We may also need multiple ladap password and password file config keys for different ldap servers. Currently all ldap servers use the same configuration in hadoop.security.group.mapping.ldap.ssl.keystore.password.file and hadoop.security.group.mapping.ldap.bind.password.file. 3. you may want to change MAPPING_PROVIDERS_KEY to MAPPING_PROVIDERS_CONFIG_PREFIX to be consistent with others, like MAPPING_PROVIDER_CONFIG_PREFIX and GROUP_MAPPING_CONFIG_PREFIX. 4. please add java doc for domain param in Groups#getGroups, and doGetGroups The patch needs to be rebased too. > Support multiple group mapping providers > > > Key: HADOOP-8943 > URL: https://issues.apache.org/jira/browse/HADOOP-8943 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Kai Zheng >Assignee: Kai Zheng > Fix For: 2.5.0 > > Attachments: HADOOP-8943.patch, HADOOP-8943.patch, HADOOP-8943.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > Discussed with Natty about LdapGroupMapping, we need to improve it so that: > 1. It's possible to do different group mapping for different > users/principals. For example, AD user should go to LdapGroupMapping service > for group, but service principals such as hdfs, mapred can still use the > default one ShellBasedUnixGroupsMapping; > 2. Multiple ADs can be supported to do LdapGroupMapping; > 3. It's possible to configure what kind of users/principals (regarding > domain/realm is an option) should use which group mapping service/mechanism. > 4. It's possible to configure and combine multiple existing mapping providers > without writing codes implementing new one. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10702) KerberosAuthenticationHandler does not log the principal names correctly
[ https://issues.apache.org/jira/browse/HADOOP-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033206#comment-14033206 ] Hudson commented on HADOOP-10702: - SUCCESS: Integrated in Hadoop-trunk-Commit #5713 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5713/]) HADOOP-10702. KerberosAuthenticationHandler does not log the principal names correctly. Contributed by Benoy Antony. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1603023) * /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/KerberosAuthenticationHandler.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt > KerberosAuthenticationHandler does not log the principal names correctly > > > Key: HADOOP-10702 > URL: https://issues.apache.org/jira/browse/HADOOP-10702 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.5.0 >Reporter: Benoy Antony >Assignee: Benoy Antony >Priority: Minor > Fix For: 3.0.0, 2.5.0 > > Attachments: HADOOP-10702.patch > > > With HADOOP-10158, it is possible to load multiple principal names or all > HTTP principals in the key tab by specifying “*”. > Each principal name is logged when when the principal is loaded from key tab. > But there is a bug due to which principal name is logged each time as either > “*” or the full list of principals. > The log snippet is as below: > 2014-06-15 00:19:13,288 INFO > org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: > Login using keytab /etc/hadoop/hadoop.keytab, for principal * > 2014-06-15 00:19:13,292 INFO > org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: > Login using keytab /etc/hadoop/hadoop.keytab, for principal * > 2014-06-15 00:19:13,294 INFO > org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: > Login using keytab /etc/hadoop/hadoop.keytab, for principal * > 2014-06-15 00:19:13,295 INFO > org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: > Login using keytab /etc/hadoop/hadoop.keytab, for principal * -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10666) Remove Copyright /d/d/d/d Apache Software Foundation from the source files license header
[ https://issues.apache.org/jira/browse/HADOOP-10666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033209#comment-14033209 ] Hudson commented on HADOOP-10666: - SUCCESS: Integrated in Hadoop-trunk-Commit #5713 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5713/]) HADOOP-10666. Remove Copyright /d/d/d/d Apache Software Foundation from the source files license header. Contributed by Henry Saputra. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1603025) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-policy.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties * /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/protoc/ProtocMojo.java * /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/util/Exec.java * /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/util/FileSetUtils.java * /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/versioninfo/VersionInfoMojo.java * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/resources/distcp-default.xml * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/resources/sslConfig.xml > Remove Copyright /d/d/d/d Apache Software Foundation from the source files > license header > - > > Key: HADOOP-10666 > URL: https://issues.apache.org/jira/browse/HADOOP-10666 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 2.4.0 >Reporter: Henry Saputra >Assignee: Henry Saputra >Priority: Minor > Fix For: 2.5.0 > > Attachments: HADOOP-10666.patch > > > Some of the files have "Copyright 2011 Apache Software Foundation" in the > license header comment. > This is not the right license header per ASF rule [1] > From the ASF header rule [1] : > "Source File Headers for Code Developed at the ASF" > "This section refers only to works submitted directly to the ASF by the > copyright owner or owner's agent." > Seems like the one without Copyright notice is used for software developed > directly by ASF which Apache Hadoop is. > When it was under external organization it does need to use the one from ASF > 2.0 license [2] which requires Copyright included. > [1] http://www.apache.org/legal/src-headers.html > [2] http://www.apache.org/licenses/LICENSE-2.0 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10711) Cleanup some extra dependencies from hadoop-auth
[
https://issues.apache.org/jira/browse/HADOOP-10711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033205#comment-14033205
]
Robert Kanter commented on HADOOP-10711:
>From my testing, we only need 3 of them:
{noformat}
[INFO] \-
org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
[INFO]+- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
[INFO]+- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
[INFO]\- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
{noformat}
> Cleanup some extra dependencies from hadoop-auth
>
>
> Key: HADOOP-10711
> URL: https://issues.apache.org/jira/browse/HADOOP-10711
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.5.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>
> HADOOP-10322 added {{apacheds-kerberos-codec}} as a dependency, which brought
> in some additional dependencies.
> {noformat}
> [INFO] \-
> org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
> [INFO]+- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
> [INFO]+- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-asn1-ber:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-i18n:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-ldap-model:jar:1.0.0-M20:compile
> [INFO]| +- org.apache.mina:mina-core:jar:2.0.0-M5:compile
> [INFO]| +- antlr:antlr:jar:2.7.7:compile
> [INFO]| +- commons-lang:commons-lang:jar:2.6:compile
> [INFO]| \- commons-collections:commons-collections:jar:3.2.1:compile
> [INFO]+- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
> [INFO]\- net.sf.ehcache:ehcache-core:jar:2.4.4:compile
> {noformat}
> It looks like we don't need most of them.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10711) Cleanup some extra dependencies from hadoop-auth
Robert Kanter created HADOOP-10711:
--
Summary: Cleanup some extra dependencies from hadoop-auth
Key: HADOOP-10711
URL: https://issues.apache.org/jira/browse/HADOOP-10711
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.5.0
Reporter: Robert Kanter
Assignee: Robert Kanter
HADOOP-10322 added {{apacheds-kerberos-codec}} as a dependency, which brought
in some additional dependencies.
{noformat}
[INFO] \-
org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
[INFO]+- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
[INFO]+- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
[INFO]+- org.apache.directory.api:api-asn1-ber:jar:1.0.0-M20:compile
[INFO]+- org.apache.directory.api:api-i18n:jar:1.0.0-M20:compile
[INFO]+- org.apache.directory.api:api-ldap-model:jar:1.0.0-M20:compile
[INFO]| +- org.apache.mina:mina-core:jar:2.0.0-M5:compile
[INFO]| +- antlr:antlr:jar:2.7.7:compile
[INFO]| +- commons-lang:commons-lang:jar:2.6:compile
[INFO]| \- commons-collections:commons-collections:jar:3.2.1:compile
[INFO]+- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
[INFO]\- net.sf.ehcache:ehcache-core:jar:2.4.4:compile
{noformat}
It looks like we don't need most of them.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10666) Remove Copyright /d/d/d/d Apache Software Foundation from the source files license header
[ https://issues.apache.org/jira/browse/HADOOP-10666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-10666: - Resolution: Fixed Fix Version/s: 2.5.0 Status: Resolved (was: Patch Available) Thanks again Henry, committed to trunk and branch-2. > Remove Copyright /d/d/d/d Apache Software Foundation from the source files > license header > - > > Key: HADOOP-10666 > URL: https://issues.apache.org/jira/browse/HADOOP-10666 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 2.4.0 >Reporter: Henry Saputra >Assignee: Henry Saputra >Priority: Minor > Fix For: 2.5.0 > > Attachments: HADOOP-10666.patch > > > Some of the files have "Copyright 2011 Apache Software Foundation" in the > license header comment. > This is not the right license header per ASF rule [1] > From the ASF header rule [1] : > "Source File Headers for Code Developed at the ASF" > "This section refers only to works submitted directly to the ASF by the > copyright owner or owner's agent." > Seems like the one without Copyright notice is used for software developed > directly by ASF which Apache Hadoop is. > When it was under external organization it does need to use the one from ASF > 2.0 license [2] which requires Copyright included. > [1] http://www.apache.org/legal/src-headers.html > [2] http://www.apache.org/licenses/LICENSE-2.0 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10673) Update rpc metrics when the call throws an exception
[ https://issues.apache.org/jira/browse/HADOOP-10673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HADOOP-10673: - Attachment: HADOOP-10673-6.patch Thanks, Tsuyoshi. [~lohit] asked a good question how the increase of the number of metrics will impact metrics subsystem. The most common one might be StandbyException, which can be thrown regardless of methods. Other exceptions are likely to be specific to certain methods. Based on the current number of metrics, JMX size; number of methods, a very rough estimate can put the increase to around 10-15% for NN. So to be safe, we can aggregate by exception class name instead; that will still provide useful information (from the exception name, we can find the possible methods). Here is the updated patch. > Update rpc metrics when the call throws an exception > > > Key: HADOOP-10673 > URL: https://issues.apache.org/jira/browse/HADOOP-10673 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ming Ma >Assignee: Ming Ma > Attachments: HADOOP-10673-2.patch, HADOOP-10673-4.patch, > HADOOP-10673-5.patch, HADOOP-10673-6.patch, HADOOP-10673.patch > > > Currently RPC metrics isn't updated when the call throws an exception. We can > either update the existing metrics or have a new set of metrics in the case > of exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HADOOP-10666) Remove Copyright /d/d/d/d Apache Software Foundation from the source files license header
[ https://issues.apache.org/jira/browse/HADOOP-10666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reassigned HADOOP-10666: Assignee: Henry Saputra Hi Henry, looks good to me. I made you a Hadoop contributor and assigned to you, will commit this shortly. > Remove Copyright /d/d/d/d Apache Software Foundation from the source files > license header > - > > Key: HADOOP-10666 > URL: https://issues.apache.org/jira/browse/HADOOP-10666 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 2.4.0 >Reporter: Henry Saputra >Assignee: Henry Saputra >Priority: Minor > Attachments: HADOOP-10666.patch > > > Some of the files have "Copyright 2011 Apache Software Foundation" in the > license header comment. > This is not the right license header per ASF rule [1] > From the ASF header rule [1] : > "Source File Headers for Code Developed at the ASF" > "This section refers only to works submitted directly to the ASF by the > copyright owner or owner's agent." > Seems like the one without Copyright notice is used for software developed > directly by ASF which Apache Hadoop is. > When it was under external organization it does need to use the one from ASF > 2.0 license [2] which requires Copyright included. > [1] http://www.apache.org/legal/src-headers.html > [2] http://www.apache.org/licenses/LICENSE-2.0 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10702) KerberosAuthenticationHandler does not log the principal names correctly
[ https://issues.apache.org/jira/browse/HADOOP-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-10702: --- Target Version/s: 3.0.0, 2.5.0 Fix Version/s: 2.5.0 3.0.0 Hadoop Flags: Reviewed +1 for the patch. I committed this to trunk and branch-2. Thank you for another fix, Benoy. bq. -1 tests included. The patch doesn't appear to include any new or modified tests. This patch changed logging only, and there is no need for an additional test. > KerberosAuthenticationHandler does not log the principal names correctly > > > Key: HADOOP-10702 > URL: https://issues.apache.org/jira/browse/HADOOP-10702 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.5.0 >Reporter: Benoy Antony >Assignee: Benoy Antony >Priority: Minor > Fix For: 3.0.0, 2.5.0 > > Attachments: HADOOP-10702.patch > > > With HADOOP-10158, it is possible to load multiple principal names or all > HTTP principals in the key tab by specifying “*”. > Each principal name is logged when when the principal is loaded from key tab. > But there is a bug due to which principal name is logged each time as either > “*” or the full list of principals. > The log snippet is as below: > 2014-06-15 00:19:13,288 INFO > org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: > Login using keytab /etc/hadoop/hadoop.keytab, for principal * > 2014-06-15 00:19:13,292 INFO > org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: > Login using keytab /etc/hadoop/hadoop.keytab, for principal * > 2014-06-15 00:19:13,294 INFO > org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: > Login using keytab /etc/hadoop/hadoop.keytab, for principal * > 2014-06-15 00:19:13,295 INFO > org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: > Login using keytab /etc/hadoop/hadoop.keytab, for principal * -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10702) KerberosAuthenticationHandler does not log the principal names correctly
[ https://issues.apache.org/jira/browse/HADOOP-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-10702: --- Resolution: Fixed Status: Resolved (was: Patch Available) > KerberosAuthenticationHandler does not log the principal names correctly > > > Key: HADOOP-10702 > URL: https://issues.apache.org/jira/browse/HADOOP-10702 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.5.0 >Reporter: Benoy Antony >Assignee: Benoy Antony >Priority: Minor > Fix For: 3.0.0, 2.5.0 > > Attachments: HADOOP-10702.patch > > > With HADOOP-10158, it is possible to load multiple principal names or all > HTTP principals in the key tab by specifying “*”. > Each principal name is logged when when the principal is loaded from key tab. > But there is a bug due to which principal name is logged each time as either > “*” or the full list of principals. > The log snippet is as below: > 2014-06-15 00:19:13,288 INFO > org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: > Login using keytab /etc/hadoop/hadoop.keytab, for principal * > 2014-06-15 00:19:13,292 INFO > org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: > Login using keytab /etc/hadoop/hadoop.keytab, for principal * > 2014-06-15 00:19:13,294 INFO > org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: > Login using keytab /etc/hadoop/hadoop.keytab, for principal * > 2014-06-15 00:19:13,295 INFO > org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: > Login using keytab /etc/hadoop/hadoop.keytab, for principal * -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10690) Lack of synchronization on access to InputStream in NativeAzureFileSystem#NativeAzureFsInputStream#close()
[
https://issues.apache.org/jira/browse/HADOOP-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen He updated HADOOP-10690:
-
Target Version/s: 0.23.11, 2.5.0
Status: Patch Available (was: Open)
> Lack of synchronization on access to InputStream in
> NativeAzureFileSystem#NativeAzureFsInputStream#close()
> --
>
> Key: HADOOP-10690
> URL: https://issues.apache.org/jira/browse/HADOOP-10690
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Chen He
>Priority: Minor
> Attachments: HADOOP-10690.patch
>
>
> {code}
> public void close() throws IOException {
> in.close();
> }
> {code}
> The close() method should be protected by synchronized keyword.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10690) Lack of synchronization on access to InputStream in NativeAzureFileSystem#NativeAzureFsInputStream#close()
[
https://issues.apache.org/jira/browse/HADOOP-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen He updated HADOOP-10690:
-
Attachment: HADOOP-10690.patch
Patch submitted.
> Lack of synchronization on access to InputStream in
> NativeAzureFileSystem#NativeAzureFsInputStream#close()
> --
>
> Key: HADOOP-10690
> URL: https://issues.apache.org/jira/browse/HADOOP-10690
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
> Attachments: HADOOP-10690.patch
>
>
> {code}
> public void close() throws IOException {
> in.close();
> }
> {code}
> The close() method should be protected by synchronized keyword.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (HADOOP-10690) Lack of synchronization on access to InputStream in NativeAzureFileSystem#NativeAzureFsInputStream#close()
[
https://issues.apache.org/jira/browse/HADOOP-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen He reassigned HADOOP-10690:
Assignee: Chen He
> Lack of synchronization on access to InputStream in
> NativeAzureFileSystem#NativeAzureFsInputStream#close()
> --
>
> Key: HADOOP-10690
> URL: https://issues.apache.org/jira/browse/HADOOP-10690
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Chen He
>Priority: Minor
> Attachments: HADOOP-10690.patch
>
>
> {code}
> public void close() throws IOException {
> in.close();
> }
> {code}
> The close() method should be protected by synchronized keyword.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10690) Lack of synchronization on access to InputStream in NativeAzureFileSystem#NativeAzureFsInputStream#close()
[
https://issues.apache.org/jira/browse/HADOOP-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033036#comment-14033036
]
Chris Nauroth commented on HADOOP-10690:
The equivalent class in the HDFS client is {{DFSInputStream}}, and it has a
{{close}} method that is synchronized. I don't know that lack of synchronized
here is causing any specific problems, but I suppose we ought to add it for
maximum compatibility with HDFS semantics.
The underlying class that ends up getting used from the Azure SDK is
{{BlobInputStream}}. Reading its source, it looks like that class has a
synchronized {{close}}. Even though it's synchronized at that layer, we might
as well go ahead and add it here too.
> Lack of synchronization on access to InputStream in
> NativeAzureFileSystem#NativeAzureFsInputStream#close()
> --
>
> Key: HADOOP-10690
> URL: https://issues.apache.org/jira/browse/HADOOP-10690
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> {code}
> public void close() throws IOException {
> in.close();
> }
> {code}
> The close() method should be protected by synchronized keyword.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10710) hadoop.auth cookie is not properly constructed according to RFC2109
Alejandro Abdelnur created HADOOP-10710:
---
Summary: hadoop.auth cookie is not properly constructed according
to RFC2109
Key: HADOOP-10710
URL: https://issues.apache.org/jira/browse/HADOOP-10710
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.4.0
Reporter: Alejandro Abdelnur
It seems that HADOOP-10379 introduced a bug on how hadoop.auth cookies are
being constructed.
Before HADOOP-10379, cookies were constructed using Servlet's {{Cookie}} class
and corresponding {{HttpServletResponse}} methods. This was taking care of
setting attributes like 'Version=1' and double-quoting the cookie value if
necessary.
HADOOP-10379 changed the Cookie creation to use a {{StringBuillder}} and
setting values and attributes by hand. This is not taking care of setting
required attributes like Version and escaping the cookie value.
While this is not breaking HadoopAuth {{AuthenticatedURL}} access, it is
breaking access done using {{HtttpClient}}. I.e. Solr uses HttpClient and its
access is broken since this change.
It seems that HADOOP-10379 main objective was to set the 'secure' attribute.
Note this can be done using the {{Cookie}} API.
We should revert the cookie creation logic to use the {{Cookie}} API and take
care of the security flag via {{setSecure(boolean)}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10379) Protect authentication cookies with the HttpOnly and Secure flags
[
https://issues.apache.org/jira/browse/HADOOP-10379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032968#comment-14032968
]
Alejandro Abdelnur commented on HADOOP-10379:
-
This JIRA has introduced a regression: HADOOP-10710.
> Protect authentication cookies with the HttpOnly and Secure flags
> -
>
> Key: HADOOP-10379
> URL: https://issues.apache.org/jira/browse/HADOOP-10379
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Fix For: 2.4.0
>
> Attachments: HADOOP-10379-branch-1.000.patch, HADOOP-10379.000.patch,
> HADOOP-10379.001.patch, HADOOP-10379.002.patch
>
>
> Browser vendors have adopted proposals to enhance the security of HTTP
> cookies. For example, the server can mark a cookie as {{Secure}} so that it
> will not be transfer via plain-text HTTP protocol, and the server can mark a
> cookie as {{HttpOnly}} to prohibit the JavaScript to access that cookie.
> This jira proposes to adopt these flags in Hadoop to protect the HTTP cookie
> used for authentication purposes.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10282) Create a FairCallQueue: a multi-level call queue which schedules incoming calls and multiplexes outgoing calls
[ https://issues.apache.org/jira/browse/HADOOP-10282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032938#comment-14032938 ] Chris Li commented on HADOOP-10282: --- Hi Arapit, Check out HADOOP-10278, which allows the default LinkedBlockingQueue to be swapped with any implementor of BlockingQueue. The FairCallQueue is one such implementor of BlockingQueue. > Create a FairCallQueue: a multi-level call queue which schedules incoming > calls and multiplexes outgoing calls > -- > > Key: HADOOP-10282 > URL: https://issues.apache.org/jira/browse/HADOOP-10282 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Chris Li >Assignee: Chris Li > Attachments: HADOOP-10282.patch > > > The FairCallQueue ensures quality of service by altering the order of RPC > calls internally. > It consists of three parts: > 1. a Scheduler (`HistoryRpcScheduler` is provided) which provides a priority > number from 0 to N (0 being highest priority) > 2. a Multi-level queue (residing in `FairCallQueue`) which provides a way to > keep calls in priority order internally > 3. a Multiplexer (`WeightedRoundRobinMultiplexer` is provided) which provides > logic to control which queue to take from > Currently the Mux and Scheduler are not pluggable, but they probably should > be (up for discussion). > This is how it is used: > // Production > 1. Call is created and given to the CallQueueManager > 2. CallQueueManager requests a `put(T call)` into the `FairCallQueue` which > implements `BlockingQueue` > 3. `FairCallQueue` asks its scheduler for a scheduling decision, which is an > integer e.g. 12 > 4. `FairCallQueue` inserts Call into the 12th queue: > `queues.get(12).put(call)` > // Consumption > 1. CallQueueManager requests `take()` or `poll()` on FairCallQueue > 2. `FairCallQueue` asks its multiplexer for which queue to draw from, which > will also be an integer e.g. 2 > 3. `FairCallQueue` draws from this queue if it has an available call (or > tries other queues if it is empty) > Additional information is available in the linked JIRAs regarding the > Scheduler and Multiplexer's roles. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10565) Support IP ranges (CIDR) in proxyuser.hosts
[ https://issues.apache.org/jira/browse/HADOOP-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032935#comment-14032935 ] Benoy Antony commented on HADOOP-10565: --- Thanks [~arpitagarwal]. Were there any update ? > Support IP ranges (CIDR) in proxyuser.hosts > > > Key: HADOOP-10565 > URL: https://issues.apache.org/jira/browse/HADOOP-10565 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10565.patch, HADOOP-10565.patch > > > In some use cases, there will be many hosts from which the user can > impersonate. > This requires specifying many ips in the XML configuration. > It is cumbersome to specify and maintain long list of ips in proxyuser.hosts > The problem can be solved if we enable proxyuser.hosts to accept ip ranges in > CIDR format. > In addition, the current ip authorization involve a liner scan of the ips and > an attempt to do InetAddress.getByName() for each ip/host. > It may be beneficial to group this functionality of ip authorization by > looking up "ip addresses/host names/ip-ranges" into a separate class. This > could be reused in other usecases which require similar functionality -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10709) Reuse Filters across web apps
[ https://issues.apache.org/jira/browse/HADOOP-10709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HADOOP-10709: -- Attachment: HADOOP-10709.patch The approach taken is as follows: # _HttpServer2_ keeps a cache of Filters mapped to names. e.g. “authentication” -> AuthenticationFilter #. _NameNodeHttpServer_ is enhanced to accept a set of Authenticationfilters. This can be an AuthenticationFIlter class or a name. eg. org.apache.hadoop.hdfs.web.TokenAuthFilter,authentication 3. _AuthenticationFilter_ is enhanced to look for SKIP attribute to decide whether to skip its authentication. 4. _TokenAuthFilter_ is added which looks for delegation token in the http request. If delegation token is present, it sets the SKIP attribute in the httprequest. > Reuse Filters across web apps > - > > Key: HADOOP-10709 > URL: https://issues.apache.org/jira/browse/HADOOP-10709 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10709.patch > > > Currently, we need to define separate authentication filters for webhdfs and > general webui. This also involves defining parameters for those filters. > It will be better if one could reuse filters for web apps if desired. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10282) Create a FairCallQueue: a multi-level call queue which schedules incoming calls and multiplexes outgoing calls
[ https://issues.apache.org/jira/browse/HADOOP-10282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032924#comment-14032924 ] Arpit Agarwal commented on HADOOP-10282: Hi Chris, all the Jiras HADOOP-10279, HADOOP-10281 and HADOOP-10282 just add new files. How are they meant to plugin to the existing code? Perhaps I forgot some context on the changes so if this is described elsewhere, can you just give me a pointer? Thanks! > Create a FairCallQueue: a multi-level call queue which schedules incoming > calls and multiplexes outgoing calls > -- > > Key: HADOOP-10282 > URL: https://issues.apache.org/jira/browse/HADOOP-10282 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Chris Li >Assignee: Chris Li > Attachments: HADOOP-10282.patch > > > The FairCallQueue ensures quality of service by altering the order of RPC > calls internally. > It consists of three parts: > 1. a Scheduler (`HistoryRpcScheduler` is provided) which provides a priority > number from 0 to N (0 being highest priority) > 2. a Multi-level queue (residing in `FairCallQueue`) which provides a way to > keep calls in priority order internally > 3. a Multiplexer (`WeightedRoundRobinMultiplexer` is provided) which provides > logic to control which queue to take from > Currently the Mux and Scheduler are not pluggable, but they probably should > be (up for discussion). > This is how it is used: > // Production > 1. Call is created and given to the CallQueueManager > 2. CallQueueManager requests a `put(T call)` into the `FairCallQueue` which > implements `BlockingQueue` > 3. `FairCallQueue` asks its scheduler for a scheduling decision, which is an > integer e.g. 12 > 4. `FairCallQueue` inserts Call into the 12th queue: > `queues.get(12).put(call)` > // Consumption > 1. CallQueueManager requests `take()` or `poll()` on FairCallQueue > 2. `FairCallQueue` asks its multiplexer for which queue to draw from, which > will also be an integer e.g. 2 > 3. `FairCallQueue` draws from this queue if it has an available call (or > tries other queues if it is empty) > Additional information is available in the linked JIRAs regarding the > Scheduler and Multiplexer's roles. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10709) Reuse Filters across web apps
Benoy Antony created HADOOP-10709: - Summary: Reuse Filters across web apps Key: HADOOP-10709 URL: https://issues.apache.org/jira/browse/HADOOP-10709 Project: Hadoop Common Issue Type: Improvement Components: security Reporter: Benoy Antony Assignee: Benoy Antony Currently, we need to define separate authentication filters for webhdfs and general webui. This also involves defining parameters for those filters. It will be better if one could reuse filters for web apps if desired. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10683) Users authenticated with KERBEROS are recorded as being authenticated with SIMPLE
[ https://issues.apache.org/jira/browse/HADOOP-10683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032908#comment-14032908 ] Hudson commented on HADOOP-10683: - SUCCESS: Integrated in Hadoop-trunk-Commit #5710 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5710/]) HADOOP-10683. Users authenticated with KERBEROS are recorded as being authenticated with SIMPLE. Contributed by Benoy Antony. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1602991) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java > Users authenticated with KERBEROS are recorded as being authenticated with > SIMPLE > - > > Key: HADOOP-10683 > URL: https://issues.apache.org/jira/browse/HADOOP-10683 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.0.0, 2.4.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Fix For: 3.0.0, 2.5.0 > > Attachments: HADOOP-10683.patch > > > We have enabled kerberos authentication in our clusters, but we see the > following in the log files > 2014-06-11 11:07:05,903 INFO SecurityLogger.org.apache.hadoop.ipc.Server: > Auth successful for x...@y.com (*auth:SIMPLE*) > 2014-06-11 11:07:05,914 INFO > SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: > Authorization successful for x...@y.com (auth:KERBEROS) for > protocol=interface > This is quite confusing for administrators. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10683) Users authenticated with KERBEROS are recorded as being authenticated with SIMPLE
[ https://issues.apache.org/jira/browse/HADOOP-10683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-10683: --- Resolution: Fixed Fix Version/s: 2.5.0 3.0.0 Status: Resolved (was: Patch Available) I committed this to trunk and branch-2. Benoy, thank you for contributing the patch. > Users authenticated with KERBEROS are recorded as being authenticated with > SIMPLE > - > > Key: HADOOP-10683 > URL: https://issues.apache.org/jira/browse/HADOOP-10683 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.0.0, 2.4.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Fix For: 3.0.0, 2.5.0 > > Attachments: HADOOP-10683.patch > > > We have enabled kerberos authentication in our clusters, but we see the > following in the log files > 2014-06-11 11:07:05,903 INFO SecurityLogger.org.apache.hadoop.ipc.Server: > Auth successful for x...@y.com (*auth:SIMPLE*) > 2014-06-11 11:07:05,914 INFO > SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: > Authorization successful for x...@y.com (auth:KERBEROS) for > protocol=interface > This is quite confusing for administrators. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10683) Users authenticated with KERBEROS are recorded as being authenticated with SIMPLE
[
https://issues.apache.org/jira/browse/HADOOP-10683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-10683:
---
Target Version/s: 3.0.0, 2.5.0
Affects Version/s: 3.0.0
2.4.0
Hadoop Flags: Reviewed
+1 for the patch. Nice catch, [~benoyantony]. I reviewed call sites that use
the return value of {{UserGroupInformation#getAuthenticationMethod}}, and I
didn't see any potential problems with this change. I'll commit it.
> Users authenticated with KERBEROS are recorded as being authenticated with
> SIMPLE
> -
>
> Key: HADOOP-10683
> URL: https://issues.apache.org/jira/browse/HADOOP-10683
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HADOOP-10683.patch
>
>
> We have enabled kerberos authentication in our clusters, but we see the
> following in the log files
> 2014-06-11 11:07:05,903 INFO SecurityLogger.org.apache.hadoop.ipc.Server:
> Auth successful for x...@y.com (*auth:SIMPLE*)
> 2014-06-11 11:07:05,914 INFO
> SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager:
> Authorization successful for x...@y.com (auth:KERBEROS) for
> protocol=interface
> This is quite confusing for administrators.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10696) Add optional tags to KeyProvider Options and Metadata
[ https://issues.apache.org/jira/browse/HADOOP-10696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032808#comment-14032808 ] Mike Yoder commented on HADOOP-10696: - My "commentary" - Vormetric allows key-value pairs. KMIP is even more general; PKCS11 is similar in attributes being blobs of opaque data. Unfortunately I don't know this detail about other key managers out there. > Add optional tags to KeyProvider Options and Metadata > - > > Key: HADOOP-10696 > URL: https://issues.apache.org/jira/browse/HADOOP-10696 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HADOOP-10696.patch, HADOOP-10696.patch, > HADOOP-10696.patch > > > In addition to having an optional description, KeyProvider Options and > Metadata should support optional tags to help categorize keys. > This would be useful for visualization purposes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (HADOOP-10389) Native RPCv9 client
[
https://issues.apache.org/jira/browse/HADOOP-10389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032698#comment-14032698
]
Haohui Mai edited comment on HADOOP-10389 at 6/16/14 6:47 PM:
--
bq. If the code is so impenetrable that review is truly impossible, please
consider demonstrating the virtues of C++11 in a competing branch.
As a weekend project, I've uploaded a proof-of-concept patch in C++ for this
jira. It implements a synchronous RPC engine and the {{exists()}} call for
libhdfs.
The main point is not (and never intends to be) to demonstrate c++ is superior
to c, but to show that it is practical to using current technology to improve
code quality and to cut down the maintenance cost down the road. In my personal
opinion it is important to ensure that the code can be easily maintained by the
community. The patch demonstrates a viable way to approach this goal.
Just a couple points to highlight:
# Simpler implementation. Thanks to the standard libraries, there is no need to
put things like RB trees, hash tables, linked lists into the patch and to ask
for reviews. Though not fully equivalent, the patch is an order of magnitude
smaller in terms of size.
# Automatic resource management. Explicit resource management, (e.g.,
{{malloc()}} / {{free()}}, and closing sockets) are no longer required. The
life time of resource matches the scope of the code. It is a systematic way to
avoid resource leaks.
# Stronger claims from the type systems. Modern language allows the code to be
written in a mostly type-safe way, where the type system is able to show that
the majority of the code is safe. There is only one cast required in the code
compared to many (each in the linked list) in the old implementation. New
constructs like {{std::array}} also allow integrate bounds check in the code to
help prevent buffer overflows.
Just to reiterate, by no means I'm trying to claim that the current patch is
perfect and free of bugs. People myself included make mistakes all the time. A
modern tool, however, can help people to catch the mistakes at the beginning of
the development cycle and to avoid them. Thoughts?
was (Author: wheat9):
bq. If the code is so impenetrable that review is truly impossible, please
consider demonstrating the virtues of C++11 in a competing branch.
As a weekend project, I've uploaded a proof-of-concept patch in C++ for this
jira. It implements a synchronous RPC engine and the {{exists()}} call for
libhdfs.
The main point is not (and never intends to be) to demonstrate c++ is superior
to c, but to show that it is practical to using current technology to improve
code quality and to cut down the maintenance cost down the road. In my personal
opinion it is important to ensure that the code can be easily maintained by the
community. The patch demonstrates a viable way to approach this goal.
Just a couple points to highlight:
# Simpler implementation. Thanks to the standard libraries, there is no need to
put things like RB trees, hash tables, linked lists into the patch and to ask
for reviews. Though not fully equivalent, the patch is an order of magnitude
smaller in terms of size.
# Automatic resource management. Explicit resource management, (e.g.,
{{malloc()}} / {{free()}}, and closing sockets) are no longer required. The
life time of resource matches the scope of the code. It is a systematic way to
avoid resource leaks.
# Stronger claims from the type systems. Modern language allows the code to be
written in a mostly type-safe way, where the type system is able to show that
the majority of the code is safe. There is only one cast required in the code
compared to many (each in the linked list) in the old implementation. New
constructs like {{std::array}} also allow integrate bounds check in the code to
help prevent buffer overflows.
Just to reiterate, by no means I'm trying to claim that the current patch is
perfect and free of bugs. People myself included make mistakes all the time. A
modern tool, however, can help people to catch the mistakes at the beginning of
the development cycle and to avoid them. I believe that this is what good
software engineering practice should do. I don't see why this is a
philosophical debate on which language is better (though I can be happily
convinced in either way), and why writing safer, and easier to approach
correctness can be out-of-the-scope during development.
> Native RPCv9 client
> ---
>
> Key: HADOOP-10389
> URL: https://issues.apache.org/jira/browse/HADOOP-10389
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: HADOOP-10388
>Reporter: Binglin Chang
>Assignee: Colin Patrick McCabe
> Attachments: HADOOP-10388.001.patch,
> HADOOP-10389-alternative.000.patch, HADOOP-10389.00
[jira] [Commented] (HADOOP-10696) Add optional tags to KeyProvider Options and Metadata
[ https://issues.apache.org/jira/browse/HADOOP-10696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032728#comment-14032728 ] Andrew Wang commented on HADOOP-10696: -- Code-wise this looks good, I just have the same question as Mike about using KV pairs (which seems more general and future-proof). Could we get some commentary on which key managers use tags vs. KV pairs? It'd be unfortunate to have to revisit this after the API is released. > Add optional tags to KeyProvider Options and Metadata > - > > Key: HADOOP-10696 > URL: https://issues.apache.org/jira/browse/HADOOP-10696 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HADOOP-10696.patch, HADOOP-10696.patch, > HADOOP-10696.patch > > > In addition to having an optional description, KeyProvider Options and > Metadata should support optional tags to help categorize keys. > This would be useful for visualization purposes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10389) Native RPCv9 client
[
https://issues.apache.org/jira/browse/HADOOP-10389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032698#comment-14032698
]
Haohui Mai commented on HADOOP-10389:
-
bq. If the code is so impenetrable that review is truly impossible, please
consider demonstrating the virtues of C++11 in a competing branch.
As a weekend project, I've uploaded a proof-of-concept patch in C++ for this
jira. It implements a synchronous RPC engine and the {{exists()}} call for
libhdfs.
The main point is not (and never intends to be) to demonstrate c++ is superior
to c, but to show that it is practical to using current technology to improve
code quality and to cut down the maintenance cost down the road. In my personal
opinion it is important to ensure that the code can be easily maintained by the
community. The patch demonstrates a viable way to approach this goal.
Just a couple points to highlight:
# Simpler implementation. Thanks to the standard libraries, there is no need to
put things like RB trees, hash tables, linked lists into the patch and to ask
for reviews. Though not fully equivalent, the patch is an order of magnitude
smaller in terms of size.
# Automatic resource management. Explicit resource management, (e.g.,
{{malloc()}} / {{free()}}, and closing sockets) are no longer required. The
life time of resource matches the scope of the code. It is a systematic way to
avoid resource leaks.
# Stronger claims from the type systems. Modern language allows the code to be
written in a mostly type-safe way, where the type system is able to show that
the majority of the code is safe. There is only one cast required in the code
compared to many (each in the linked list) in the old implementation. New
constructs like {{std::array}} also allow integrate bounds check in the code to
help prevent buffer overflows.
Just to reiterate, by no means I'm trying to claim that the current patch is
perfect and free of bugs. People myself included make mistakes all the time. A
modern tool, however, can help people to catch the mistakes at the beginning of
the development cycle and to avoid them. I believe that this is what good
software engineering practice should do. I don't see why this is a
philosophical debate on which language is better (though I can be happily
convinced in either way), and why writing safer, and easier to approach
correctness can be out-of-the-scope during development.
> Native RPCv9 client
> ---
>
> Key: HADOOP-10389
> URL: https://issues.apache.org/jira/browse/HADOOP-10389
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: HADOOP-10388
>Reporter: Binglin Chang
>Assignee: Colin Patrick McCabe
> Attachments: HADOOP-10388.001.patch,
> HADOOP-10389-alternative.000.patch, HADOOP-10389.002.patch,
> HADOOP-10389.004.patch, HADOOP-10389.005.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10657) Have RetryInvocationHandler log failover attempt at INFO level
[ https://issues.apache.org/jira/browse/HADOOP-10657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032695#comment-14032695 ] Hudson commented on HADOOP-10657: - FAILURE: Integrated in Hadoop-trunk-Commit #5709 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5709/]) HADOOP-10657. Have RetryInvocationHandler log failover attempt at INFO level. Contributed by Ming Ma. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1602941) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryInvocationHandler.java > Have RetryInvocationHandler log failover attempt at INFO level > -- > > Key: HADOOP-10657 > URL: https://issues.apache.org/jira/browse/HADOOP-10657 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ming Ma >Assignee: Ming Ma > Fix For: 2.5.0 > > Attachments: HADOOP-10657-2.patch, HADOOP-10657.patch > > > RetryInovcationHandler uses worthLogging flag to decide if it will do > logging. worthLogging will be false for first fails over given > invocationFailoverCount is zero. That addresses the log noise where the > second-listed NN is active. > For other failover scenarios, it will be useful to log the error message at > info level for analysis purpose. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10657) Have RetryInvocationHandler log failover attempt at INFO level
[ https://issues.apache.org/jira/browse/HADOOP-10657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HADOOP-10657: --- Resolution: Fixed Fix Version/s: 2.5.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the contribution, [~mingma]! I've committed this to trunk and branch-2. > Have RetryInvocationHandler log failover attempt at INFO level > -- > > Key: HADOOP-10657 > URL: https://issues.apache.org/jira/browse/HADOOP-10657 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ming Ma >Assignee: Ming Ma > Fix For: 2.5.0 > > Attachments: HADOOP-10657-2.patch, HADOOP-10657.patch > > > RetryInovcationHandler uses worthLogging flag to decide if it will do > logging. worthLogging will be false for first fails over given > invocationFailoverCount is zero. That addresses the log noise where the > second-listed NN is active. > For other failover scenarios, it will be useful to log the error message at > info level for analysis purpose. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10389) Native RPCv9 client
[ https://issues.apache.org/jira/browse/HADOOP-10389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HADOOP-10389: Attachment: HADOOP-10389-alternative.000.patch > Native RPCv9 client > --- > > Key: HADOOP-10389 > URL: https://issues.apache.org/jira/browse/HADOOP-10389 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-10388 >Reporter: Binglin Chang >Assignee: Colin Patrick McCabe > Attachments: HADOOP-10388.001.patch, > HADOOP-10389-alternative.000.patch, HADOOP-10389.002.patch, > HADOOP-10389.004.patch, HADOOP-10389.005.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10704) Corrupt Files are not moved to bad_files in LocalFileSystem
[
https://issues.apache.org/jira/browse/HADOOP-10704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suraj Nayak M updated HADOOP-10704:
---
Description:
A file is created using LocalFileSystem. When the file is corrupted/changed and
tried to read, it throws ChecksumException as the CRC does not match. But this
process does not move the affected file to "bad_files" directory.
Code :
public void testReportChecksumFailureOnCorruption() throws IOException {
String sampleData = "My Test Data. This will be corrupted soon";
FileSystem fs = LocalFileSystem.get(new Configuration());
Path rootPath = new Path(TEST_ROOT_DIR);
FSDataOutputStream fsOutStream = fs.create(new Path(rootPath
+ "/testFile.txt"));
ByteArrayInputStream inStream = new ByteArrayInputStream(
sampleData.getBytes());
IOUtils.copyBytes(inStream, fsOutStream, 4096, false);
fsOutStream.close();
inStream.close();
// Corrupt the file
File corruptFile = new File(TEST_ROOT_DIR + "/testFile.txt");
FileOutputStream fOutCorruptingStream = new FileOutputStream(
corruptFile, true);
fOutCorruptingStream.write(". This is bad data!".getBytes());
fOutCorruptingStream.close();
// Read the corrupted file using LocalFS
FileSystem readLFS = LocalFileSystem.get(new Configuration());
FSDataInputStream fsInStream = readLFS.open(new
Path(TEST_ROOT_DIR
+ "/testFile.txt"));
try {
byte[] b = new byte[10];
fsInStream.read(b, 0, 10);
} catch (Exception e) {
if (corruptFile.exists()) {
// Should not exit before moving the corrupt
file to bad_files
assertFalse(true);
} else {
// Should exit after moving the corrupt file to
bad_files
assertTrue(true);
}
}
fsInStream.close();
}
was:A file is created using LocalFileSystem. When the file is
corrupted/changed and tried to read, it throws ChecksumException as the CRC
does not match. But this process does not move the affected file to "bad_files"
directory.
> Corrupt Files are not moved to bad_files in LocalFileSystem
> ---
>
> Key: HADOOP-10704
> URL: https://issues.apache.org/jira/browse/HADOOP-10704
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Reporter: Suraj Nayak M
>Priority: Minor
>
> A file is created using LocalFileSystem. When the file is corrupted/changed
> and tried to read, it throws ChecksumException as the CRC does not match. But
> this process does not move the affected file to "bad_files" directory.
> Code :
> public void testReportChecksumFailureOnCorruption() throws IOException {
> String sampleData = "My Test Data. This will be corrupted soon";
> FileSystem fs = LocalFileSystem.get(new Configuration());
> Path rootPath = new Path(TEST_ROOT_DIR);
> FSDataOutputStream fsOutStream = fs.create(new Path(rootPath
> + "/testFile.txt"));
> ByteArrayInputStream inStream = new ByteArrayInputStream(
> sampleData.getBytes());
> IOUtils.copyBytes(inStream, fsOutStream, 4096, false);
> fsOutStream.close();
> inStream.close();
> // Corrupt the file
> File corruptFile = new File(TEST_ROOT_DIR + "/testFile.txt");
> FileOutputStream fOutCorruptingStream = new FileOutputStream(
> corruptFile, true);
> fOutCorruptingStream.write(". This is bad data!".getBytes());
> fOutCorruptingStream.close();
> // Read the corrupted file using LocalFS
> FileSystem readLFS = LocalFileSystem.get(new Configuration());
> FSDataInputStream fsInStream = readLFS.open(new
> Path(TEST_ROOT_DIR
> + "/testFile.txt"));
> try {
> byte[] b = new byte[10];
> fsInStream.read(b, 0, 10);
> } catch (Exception e) {
> if (corruptFile.exists()) {
> // Should not exit before moving the corrupt
> file to bad_files
> assertFalse(true);
>
[jira] [Commented] (HADOOP-10704) Corrupt Files are not moved to bad_files in LocalFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-10704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032479#comment-14032479 ] Suraj Nayak M commented on HADOOP-10704: Removed Patch as TestFSInputChecker Test case was failing. > Corrupt Files are not moved to bad_files in LocalFileSystem > --- > > Key: HADOOP-10704 > URL: https://issues.apache.org/jira/browse/HADOOP-10704 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: Suraj Nayak M >Priority: Minor > > A file is created using LocalFileSystem. When the file is corrupted/changed > and tried to read, it throws ChecksumException as the CRC does not match. But > this process does not move the affected file to "bad_files" directory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10704) Corrupt Files are not moved to bad_files in LocalFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-10704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suraj Nayak M updated HADOOP-10704: --- Attachment: (was: HADOOP-10704.patch) > Corrupt Files are not moved to bad_files in LocalFileSystem > --- > > Key: HADOOP-10704 > URL: https://issues.apache.org/jira/browse/HADOOP-10704 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: Suraj Nayak M >Priority: Minor > > A file is created using LocalFileSystem. When the file is corrupted/changed > and tried to read, it throws ChecksumException as the CRC does not match. But > this process does not move the affected file to "bad_files" directory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-10708) support bzip2 in python avro tool
[ https://issues.apache.org/jira/browse/HADOOP-10708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eustache resolved HADOOP-10708. --- Resolution: Invalid wrong component > support bzip2 in python avro tool > - > > Key: HADOOP-10708 > URL: https://issues.apache.org/jira/browse/HADOOP-10708 > Project: Hadoop Common > Issue Type: Improvement > Components: tools >Reporter: Eustache >Priority: Minor > Labels: avro > > The Python tool to decode avro files is currently missing support for bzip2 > compression. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10708) support bzip2 in python avro tool
Eustache created HADOOP-10708: - Summary: support bzip2 in python avro tool Key: HADOOP-10708 URL: https://issues.apache.org/jira/browse/HADOOP-10708 Project: Hadoop Common Issue Type: Improvement Components: tools Reporter: Eustache Priority: Minor The Python tool to decode avro files is currently missing support for bzip2 compression. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10707) support bzip2 in python avro tool
Eustache created HADOOP-10707: - Summary: support bzip2 in python avro tool Key: HADOOP-10707 URL: https://issues.apache.org/jira/browse/HADOOP-10707 Project: Hadoop Common Issue Type: Improvement Components: tools Reporter: Eustache Priority: Minor The Python tool to decode avro files is currently missing support for bzip2 compression. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10604) CryptoFileSystem decorator using xAttrs and KeyProvider
[
https://issues.apache.org/jira/browse/HADOOP-10604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu updated HADOOP-10604:
Attachment: HADOOP-10604.1.patch
Hi Uma, the new patch includes update for your comments.
{quote}
typo. Doen't -> Doesn't ?
{quote}
Right, let’s modify it.
{quote}
Seems like we are not allowing nested encryption zones, but cfs will take
client side configuration, one client would have configured one dir and other
client could configure the sub dir of it. In this case how would we avoid
nested encryption zones to configure from the checks?
{quote}
Nice, I consider this more later, and we can support nested encryption zones.
So let’s remove the restriction of nested encryption zones. Typically
encryption zones are used to stored sensitive data, user should be aware of the
configuration.
{quote}
decodeCFSURI is doing the some special decoding stuff for replacing @ with ://
etc. But there is no encode method and I think it’s done directly in
getAuthority, instead can we make like encode and decode method?
{quote}
I try to do as your suggest, and the url is specified by user, in
{{getAuthority}} we just need to get the authority, so I think we don’t need
{{encode}} method.
{quote}
Also is it good to document about how to create ezs in other fs? ( I mean to
tell the info about what is the qualification to consider as ez? ex if
underlying fs is hdfs, HdfsAdmin has api to creae EZs)
{quote}
If using {{CryptoFileSystem}}, the encryption zone and corresponding key name
are configured in configuration file, it will not utilize underlying fs
command, such as DFSAdmin.
> CryptoFileSystem decorator using xAttrs and KeyProvider
> ---
>
> Key: HADOOP-10604
> URL: https://issues.apache.org/jira/browse/HADOOP-10604
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134)
>Reporter: Alejandro Abdelnur
>Assignee: Yi Liu
> Fix For: fs-encryption (HADOOP-10150 and HDFS-6134)
>
> Attachments: HADOOP-10604.1.patch, HADOOP-10604.patch
>
>
> A FileSystem implementation that wraps an existing filesystem and provides
> encryption. It will require the underlying filesystem to support xAttrs. It
> will use the KeyProvider API to retrieve encryption keys.
> This is mostly the work in the patch HADOOP-10150 minus the crypto streams
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10705) Fix namenode-rpc-unit warning reported by memory leak check tool(valgrind)
[ https://issues.apache.org/jira/browse/HADOOP-10705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenwu Peng updated HADOOP-10705: Attachment: vargrind.log Attach the vargrind log of namenode-rpc-unit > Fix namenode-rpc-unit warning reported by memory leak check tool(valgrind) > --- > > Key: HADOOP-10705 > URL: https://issues.apache.org/jira/browse/HADOOP-10705 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-10388 >Reporter: Wenwu Peng > Attachments: vargrind.log > > > Rum valgrind to check memory leak for namenode-rpc.unit. > There are many warning, need to fix it. > valgrind --tool=memcheck --leak-check=full --show-reachable=yes > ./namenode-rpc-unit > ==10462== Memcheck, a memory error detector > ==10462== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al. > ==10462== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info > ==10462== Command: ./namenode-rpc-unit > ==10462== > ==10462== HEAP SUMMARY: > ==10462== in use at exit: 428 bytes in 12 blocks > ==10462== total heap usage: 91 allocs, 79 frees, 16,056 bytes allocated > ==10462== > ==10462== 16 bytes in 1 blocks are indirectly lost in loss record 1 of 12 > ==10462==at 0x4C2B6CD: malloc (in > /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) > ==10462==by 0x557EB99: __nss_lookup_function (nsswitch.c:456) > ==10462==by 0x604863E: ??? > ==10462==by 0x553744C: getpwuid_r@@GLIBC_2.2.5 (getXXbyYY_r.c:256) > ==10462==by 0x42681E: geteuid_string (user.c:67) > ==10462==by 0x425ABD: main (namenode-rpc-unit.c:71) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10668) TestZKFailoverControllerStress#testExpireBackAndForth occasionally fails
[
https://issues.apache.org/jira/browse/HADOOP-10668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032269#comment-14032269
]
Binglin Chang commented on HADOOP-10668:
The test failed again in HDFS-5574
https://builds.apache.org/job/PreCommit-HDFS-Build/7127//testReport/org.apache.hadoop.ha/TestZKFailoverControllerStress/testExpireBackAndForth/
> TestZKFailoverControllerStress#testExpireBackAndForth occasionally fails
>
>
> Key: HADOOP-10668
> URL: https://issues.apache.org/jira/browse/HADOOP-10668
> Project: Hadoop Common
> Issue Type: Test
>Reporter: Ted Yu
>Priority: Minor
> Labels: test
>
> From
> https://builds.apache.org/job/PreCommit-HADOOP-Build/4018//testReport/org.apache.hadoop.ha/TestZKFailoverControllerStress/testExpireBackAndForth/
> :
> {code}
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
> at org.apache.zookeeper.server.DataTree.getData(DataTree.java:648)
> at org.apache.zookeeper.server.ZKDatabase.getData(ZKDatabase.java:371)
> at
> org.apache.hadoop.ha.MiniZKFCCluster.expireActiveLockHolder(MiniZKFCCluster.java:199)
> at
> org.apache.hadoop.ha.MiniZKFCCluster.expireAndVerifyFailover(MiniZKFCCluster.java:234)
> at
> org.apache.hadoop.ha.TestZKFailoverControllerStress.testExpireBackAndForth(TestZKFailoverControllerStress.java:84)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10706) fix some bug related to hrpc_sync_ctx
[
https://issues.apache.org/jira/browse/HADOOP-10706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated HADOOP-10706:
---
Attachment: HADOOP-10706.v1.patch
> fix some bug related to hrpc_sync_ctx
> -
>
> Key: HADOOP-10706
> URL: https://issues.apache.org/jira/browse/HADOOP-10706
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Binglin Chang
>Assignee: Binglin Chang
> Attachments: HADOOP-10706.v1.patch
>
>
> 1.
> {code}
> memset(&ctx, 0, sizeof(ctx));
> return ctx;
> {code}
> Doing this will alway make return value to 0
> 2.
> hrpc_release_sync_ctx should changed to hrpc_proxy_release_sync_ctx, all the
> functions in this .h/.c file follow this rule
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10640) Implement Namenode RPCs in HDFS native client
[ https://issues.apache.org/jira/browse/HADOOP-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032247#comment-14032247 ] Wenwu Peng commented on HADOOP-10640: - bq.Wenwu, Binglin, are you testing the latest version (v3) ? It includes this fix: Hi Colin, Still get OutOfMemoryException in latest version(v4) ,Could anyone take a look? OS: Ubuntu 12.04.1 hdfsBuilderConnect: ndfs failed to connect: org.apache.hadoop.native.HadoopCore.OutOfMemoryException: cnn_get_server_defaults: failed to allocate sync_ctx (error 12) org.apache.hadoop.native.HadoopCore.OutOfMemoryException: cnn_get_server_defaults: failed to allocate sync_ctxerror: did not expect '0': '/home/hary/hadoop-common/hadoop-native-core/src/main/native/test/fs/test_libhdfs_meta_ops.c at line 60' > Implement Namenode RPCs in HDFS native client > - > > Key: HADOOP-10640 > URL: https://issues.apache.org/jira/browse/HADOOP-10640 > Project: Hadoop Common > Issue Type: Sub-task > Components: native >Affects Versions: HADOOP-10388 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: HADOOP-10388 > > Attachments: HADOOP-10640-pnative.001.patch, > HADOOP-10640-pnative.002.patch, HADOOP-10640-pnative.003.patch, > HADOOP-10640-pnative.004.patch > > > Implement the parts of libhdfs that just involve making RPCs to the Namenode, > such as mkdir, rename, etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10706) fix some bug related to hrpc_sync_ctx
Binglin Chang created HADOOP-10706:
--
Summary: fix some bug related to hrpc_sync_ctx
Key: HADOOP-10706
URL: https://issues.apache.org/jira/browse/HADOOP-10706
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Binglin Chang
Assignee: Binglin Chang
1.
{code}
memset(&ctx, 0, sizeof(ctx));
return ctx;
{code}
Doing this will alway make return value to 0
2.
hrpc_release_sync_ctx should changed to hrpc_proxy_release_sync_ctx, all the
functions in this .h/.c file follow this rule
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10699) Fix build native library on mac osx
[
https://issues.apache.org/jira/browse/HADOOP-10699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032209#comment-14032209
]
Hadoop QA commented on HADOOP-10699:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650515/HADOOP-10699-common.v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4073//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4073//console
This message is automatically generated.
> Fix build native library on mac osx
> ---
>
> Key: HADOOP-10699
> URL: https://issues.apache.org/jira/browse/HADOOP-10699
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Kirill A. Korinskiy
>Assignee: Binglin Chang
> Attachments: HADOOP-10699-common.v3.patch,
> HADOOP-9648-native-osx.1.0.4.patch, HADOOP-9648-native-osx.1.1.2.patch,
> HADOOP-9648-native-osx.1.2.0.patch,
> HADOOP-9648-native-osx.2.0.5-alpha-rc1.patch, HADOOP-9648.v2.patch
>
>
> Some patches for fixing build a hadoop native library on os x 10.7/10.8.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10705) Fix namenode-rpc-unit warning reported by memory leak check tool(valgrind)
[ https://issues.apache.org/jira/browse/HADOOP-10705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenwu Peng updated HADOOP-10705: Issue Type: Sub-task (was: Bug) Parent: HADOOP-10388 > Fix namenode-rpc-unit warning reported by memory leak check tool(valgrind) > --- > > Key: HADOOP-10705 > URL: https://issues.apache.org/jira/browse/HADOOP-10705 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-10388 >Reporter: Wenwu Peng > > Rum valgrind to check memory leak for namenode-rpc.unit. > There are many warning, need to fix it. > valgrind --tool=memcheck --leak-check=full --show-reachable=yes > ./namenode-rpc-unit > ==10462== Memcheck, a memory error detector > ==10462== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al. > ==10462== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info > ==10462== Command: ./namenode-rpc-unit > ==10462== > ==10462== HEAP SUMMARY: > ==10462== in use at exit: 428 bytes in 12 blocks > ==10462== total heap usage: 91 allocs, 79 frees, 16,056 bytes allocated > ==10462== > ==10462== 16 bytes in 1 blocks are indirectly lost in loss record 1 of 12 > ==10462==at 0x4C2B6CD: malloc (in > /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) > ==10462==by 0x557EB99: __nss_lookup_function (nsswitch.c:456) > ==10462==by 0x604863E: ??? > ==10462==by 0x553744C: getpwuid_r@@GLIBC_2.2.5 (getXXbyYY_r.c:256) > ==10462==by 0x42681E: geteuid_string (user.c:67) > ==10462==by 0x425ABD: main (namenode-rpc-unit.c:71) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10705) Fix namenode-rpc-unit warning reported by memory leak check tool(valgrind)
Wenwu Peng created HADOOP-10705: --- Summary: Fix namenode-rpc-unit warning reported by memory leak check tool(valgrind) Key: HADOOP-10705 URL: https://issues.apache.org/jira/browse/HADOOP-10705 Project: Hadoop Common Issue Type: Bug Affects Versions: HADOOP-10388 Reporter: Wenwu Peng Rum valgrind to check memory leak for namenode-rpc.unit. There are many warning, need to fix it. valgrind --tool=memcheck --leak-check=full --show-reachable=yes ./namenode-rpc-unit ==10462== Memcheck, a memory error detector ==10462== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al. ==10462== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info ==10462== Command: ./namenode-rpc-unit ==10462== ==10462== HEAP SUMMARY: ==10462== in use at exit: 428 bytes in 12 blocks ==10462== total heap usage: 91 allocs, 79 frees, 16,056 bytes allocated ==10462== ==10462== 16 bytes in 1 blocks are indirectly lost in loss record 1 of 12 ==10462==at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==10462==by 0x557EB99: __nss_lookup_function (nsswitch.c:456) ==10462==by 0x604863E: ??? ==10462==by 0x553744C: getpwuid_r@@GLIBC_2.2.5 (getXXbyYY_r.c:256) ==10462==by 0x42681E: geteuid_string (user.c:67) ==10462==by 0x425ABD: main (namenode-rpc-unit.c:71) -- This message was sent by Atlassian JIRA (v6.2#6252)
