common-issues@hadoop.apache.org
[ https://issues.apache.org/jira/browse/HADOOP-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034884#comment-14034884 ] Wenwu Peng commented on HADOOP-10636: - Thank a lot Colin and Binglin for review > Native Hadoop Client:add unit test case for call&client_id > -- > > Key: HADOOP-10636 > URL: https://issues.apache.org/jira/browse/HADOOP-10636 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-10388 >Reporter: Wenwu Peng >Assignee: Wenwu Peng > Fix For: HADOOP-10388 > > Attachments: HADOOP-10636-pnative.001.patch, > HADOOP-10636-pnative.002.patch, HADOOP-10636-pnative.003.patch, > HADOOP-10636-pnative.004.patch, HADOOP-10636-pnative.005.patch, > HADOOP-10636-pnative.006.patch, HADOOP-10636-pnative.007.patch, > HADOOP-10636-pnative.008-commit.patch, HADOOP-10636-pnative.008.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10282) Create a FairCallQueue: a multi-level call queue which schedules incoming calls and multiplexes outgoing calls
[
https://issues.apache.org/jira/browse/HADOOP-10282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034877#comment-14034877
]
Arpit Agarwal commented on HADOOP-10282:
Hi Chris,
I reviewed this patch before its dependencies. My comments below:
# Initial question - FairCallQueue will not be on by default, correct?
# Do you need a configuration prefix ({{ns}}) or can you use a hardcoded
setting name? If the name is constructed dynamically it makes it difficult to
find usages in the code.
# Will these settings be documented and if so then where?
# {{FairCallQueue#put}} violates the contract of {{BlockingQueue#put}}, which
states that put must wait indefinitely for space to become available.
# {{FairCallQueue#put}} - can we wraparound {{priorityLevel}} to zero?
# {{FairCallQueue#getFirstNonEmptyQueue}} - javadoc should be _returns the
first non-empty queue_.
# I am curious why do you need a Reentrant lock? If it is not acquired
recursively can it be replaced with a different synchronization primitive?
# Javadoc on {{#take}} also violates the contract of its overridden method
which states that it _Retrieves and removes the head of this queue, waiting if
necessary until an element becomes available_. However it looks like it does
not actually return null in practice. So perhaps we just need to fix the
Javadoc?
# I am also unsureabout this comment - _As such, one should prefer poll() over
take()_. There is no such expectation of {{BlockingQueue#take}}. Is the comment
necessary?
# {{FairCallQueue#iterator}} - should it just throw {{NotImplementedException}}?
# Nitpick: _spurriously_ -> _spuriously_
I am still reviewing TestFairCallQueue.java.
> Create a FairCallQueue: a multi-level call queue which schedules incoming
> calls and multiplexes outgoing calls
> --
>
> Key: HADOOP-10282
> URL: https://issues.apache.org/jira/browse/HADOOP-10282
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Chris Li
>Assignee: Chris Li
> Attachments: HADOOP-10282.patch
>
>
> The FairCallQueue ensures quality of service by altering the order of RPC
> calls internally.
> It consists of three parts:
> 1. a Scheduler (`HistoryRpcScheduler` is provided) which provides a priority
> number from 0 to N (0 being highest priority)
> 2. a Multi-level queue (residing in `FairCallQueue`) which provides a way to
> keep calls in priority order internally
> 3. a Multiplexer (`WeightedRoundRobinMultiplexer` is provided) which provides
> logic to control which queue to take from
> Currently the Mux and Scheduler are not pluggable, but they probably should
> be (up for discussion).
> This is how it is used:
> // Production
> 1. Call is created and given to the CallQueueManager
> 2. CallQueueManager requests a `put(T call)` into the `FairCallQueue` which
> implements `BlockingQueue`
> 3. `FairCallQueue` asks its scheduler for a scheduling decision, which is an
> integer e.g. 12
> 4. `FairCallQueue` inserts Call into the 12th queue:
> `queues.get(12).put(call)`
> // Consumption
> 1. CallQueueManager requests `take()` or `poll()` on FairCallQueue
> 2. `FairCallQueue` asks its multiplexer for which queue to draw from, which
> will also be an integer e.g. 2
> 3. `FairCallQueue` draws from this queue if it has an available call (or
> tries other queues if it is empty)
> Additional information is available in the linked JIRAs regarding the
> Scheduler and Multiplexer's roles.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10707) support bzip2 in python avro tool
[ https://issues.apache.org/jira/browse/HADOOP-10707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034865#comment-14034865 ] Eustache commented on HADOOP-10707: --- Got it, sorry :( > support bzip2 in python avro tool > - > > Key: HADOOP-10707 > URL: https://issues.apache.org/jira/browse/HADOOP-10707 > Project: Hadoop Common > Issue Type: Improvement > Components: tools >Reporter: Eustache >Priority: Minor > Labels: avro > > The Python tool to decode avro files is currently missing support for bzip2 > compression. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10711) Cleanup some extra dependencies from hadoop-auth
[
https://issues.apache.org/jira/browse/HADOOP-10711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034854#comment-14034854
]
Steve Loughran commented on HADOOP-10711:
-
+1: we don't need more dependencies
> Cleanup some extra dependencies from hadoop-auth
>
>
> Key: HADOOP-10711
> URL: https://issues.apache.org/jira/browse/HADOOP-10711
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.5.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: HADOOP-10711_branch-2.patch,
> HADOOP-10711_branch-2.patch, HADOOP-10711_trunk.patch,
> HADOOP-10711_trunk.patch
>
>
> HADOOP-10322 added {{apacheds-kerberos-codec}} as a dependency, which brought
> in some additional dependencies.
> {noformat}
> [INFO] \-
> org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
> [INFO]+- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
> [INFO]+- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-asn1-ber:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-i18n:jar:1.0.0-M20:compile
> [INFO]+- org.apache.directory.api:api-ldap-model:jar:1.0.0-M20:compile
> [INFO]| +- org.apache.mina:mina-core:jar:2.0.0-M5:compile
> [INFO]| +- antlr:antlr:jar:2.7.7:compile
> [INFO]| +- commons-lang:commons-lang:jar:2.6:compile
> [INFO]| \- commons-collections:commons-collections:jar:3.2.1:compile
> [INFO]+- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
> [INFO]\- net.sf.ehcache:ehcache-core:jar:2.4.4:compile
> {noformat}
> It looks like we don't need most of them.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-9361) Strictly define the expected behavior of filesystem APIs and write tests to verify compliance
[
https://issues.apache.org/jira/browse/HADOOP-9361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-9361:
---
Status: Patch Available (was: Open)
> Strictly define the expected behavior of filesystem APIs and write tests to
> verify compliance
> -
>
> Key: HADOOP-9361
> URL: https://issues.apache.org/jira/browse/HADOOP-9361
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, test
>Affects Versions: 2.4.0, 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-9361-001.patch, HADOOP-9361-002.patch,
> HADOOP-9361-003.patch, HADOOP-9361-004.patch, HADOOP-9361-005.patch,
> HADOOP-9361-006.patch, HADOOP-9361-007.patch, HADOOP-9361-008.patch,
> HADOOP-9361-009.patch, HADOOP-9361-011.patch, HADOOP-9361-012.patch,
> HADOOP-9361-013.patch, HADOOP-9361-014.patch, HADOOP-9361-015.patch,
> HADOOP-9361-016.patch, HADOOP-9361.awang-addendum.patch
>
>
> {{FileSystem}} and {{FileContract}} aren't tested rigorously enough -while
> HDFS gets tested downstream, other filesystems, such as blobstore bindings,
> don't.
> The only tests that are common are those of {{FileSystemContractTestBase}},
> which HADOOP-9258 shows is incomplete.
> I propose
> # writing more tests which clarify expected behavior
> # testing operations in the interface being in their own JUnit4 test classes,
> instead of one big test suite.
> # Having each FS declare via a properties file what behaviors they offer,
> such as atomic-rename, atomic-delete, umask, immediate-consistency -test
> methods can downgrade to skipped test cases if a feature is missing.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-9361) Strictly define the expected behavior of filesystem APIs and write tests to verify compliance
[
https://issues.apache.org/jira/browse/HADOOP-9361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-9361:
---
Attachment: HADOOP-9361-016.patch
add code changes recommended by andrew
# flag is rejects-seek-past-eof, negated and defaults to true
# tests renamed to {{Test${FS}Contract{$Operation}}}, e.g
{{TestHDFSContractAppend}}.
# also: tagged openstack swift as returning false on missing source file on a
rename(), as it catches the FNFE and downgrades it for consistency with HDFS.
> Strictly define the expected behavior of filesystem APIs and write tests to
> verify compliance
> -
>
> Key: HADOOP-9361
> URL: https://issues.apache.org/jira/browse/HADOOP-9361
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, test
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-9361-001.patch, HADOOP-9361-002.patch,
> HADOOP-9361-003.patch, HADOOP-9361-004.patch, HADOOP-9361-005.patch,
> HADOOP-9361-006.patch, HADOOP-9361-007.patch, HADOOP-9361-008.patch,
> HADOOP-9361-009.patch, HADOOP-9361-011.patch, HADOOP-9361-012.patch,
> HADOOP-9361-013.patch, HADOOP-9361-014.patch, HADOOP-9361-015.patch,
> HADOOP-9361-016.patch, HADOOP-9361.awang-addendum.patch
>
>
> {{FileSystem}} and {{FileContract}} aren't tested rigorously enough -while
> HDFS gets tested downstream, other filesystems, such as blobstore bindings,
> don't.
> The only tests that are common are those of {{FileSystemContractTestBase}},
> which HADOOP-9258 shows is incomplete.
> I propose
> # writing more tests which clarify expected behavior
> # testing operations in the interface being in their own JUnit4 test classes,
> instead of one big test suite.
> # Having each FS declare via a properties file what behaviors they offer,
> such as atomic-rename, atomic-delete, umask, immediate-consistency -test
> methods can downgrade to skipped test cases if a feature is missing.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-9361) Strictly define the expected behavior of filesystem APIs and write tests to verify compliance
[
https://issues.apache.org/jira/browse/HADOOP-9361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-9361:
---
Status: Open (was: Patch Available)
> Strictly define the expected behavior of filesystem APIs and write tests to
> verify compliance
> -
>
> Key: HADOOP-9361
> URL: https://issues.apache.org/jira/browse/HADOOP-9361
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, test
>Affects Versions: 2.4.0, 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-9361-001.patch, HADOOP-9361-002.patch,
> HADOOP-9361-003.patch, HADOOP-9361-004.patch, HADOOP-9361-005.patch,
> HADOOP-9361-006.patch, HADOOP-9361-007.patch, HADOOP-9361-008.patch,
> HADOOP-9361-009.patch, HADOOP-9361-011.patch, HADOOP-9361-012.patch,
> HADOOP-9361-013.patch, HADOOP-9361-014.patch, HADOOP-9361-015.patch,
> HADOOP-9361.awang-addendum.patch
>
>
> {{FileSystem}} and {{FileContract}} aren't tested rigorously enough -while
> HDFS gets tested downstream, other filesystems, such as blobstore bindings,
> don't.
> The only tests that are common are those of {{FileSystemContractTestBase}},
> which HADOOP-9258 shows is incomplete.
> I propose
> # writing more tests which clarify expected behavior
> # testing operations in the interface being in their own JUnit4 test classes,
> instead of one big test suite.
> # Having each FS declare via a properties file what behaviors they offer,
> such as atomic-rename, atomic-delete, umask, immediate-consistency -test
> methods can downgrade to skipped test cases if a feature is missing.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10590) ServiceAuthorizationManager is not threadsafe
[ https://issues.apache.org/jira/browse/HADOOP-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034841#comment-14034841 ] Hudson commented on HADOOP-10590: - FAILURE: Integrated in Hadoop-trunk-Commit #5721 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5721/]) HADOOP-10590. ServiceAuthorizationManager is not threadsafe. (Contributed by Benoy Antony) (vinayakumarb: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1603356) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/authorize/ServiceAuthorizationManager.java > ServiceAuthorizationManager is not threadsafe > -- > > Key: HADOOP-10590 > URL: https://issues.apache.org/jira/browse/HADOOP-10590 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.4.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Fix For: 2.5.0 > > Attachments: HADOOP-10590.patch, performance-test-without-rpc.patch, > performancetest.patch > > > The mutators in ServiceAuthorizationManager are synchronized. The accessors > are not synchronized. > This results in visibility issues when ServiceAuthorizationManager's state > is accessed from different threads. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10590) ServiceAuthorizationManager is not threadsafe
[ https://issues.apache.org/jira/browse/HADOOP-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034839#comment-14034839 ] Vinayakumar B commented on HADOOP-10590: Committed to trunk and branch-2. > ServiceAuthorizationManager is not threadsafe > -- > > Key: HADOOP-10590 > URL: https://issues.apache.org/jira/browse/HADOOP-10590 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.4.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Fix For: 2.5.0 > > Attachments: HADOOP-10590.patch, performance-test-without-rpc.patch, > performancetest.patch > > > The mutators in ServiceAuthorizationManager are synchronized. The accessors > are not synchronized. > This results in visibility issues when ServiceAuthorizationManager's state > is accessed from different threads. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-10590) ServiceAuthorizationManager is not threadsafe
[ https://issues.apache.org/jira/browse/HADOOP-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B resolved HADOOP-10590. Resolution: Fixed Fix Version/s: 2.5.0 Hadoop Flags: Reviewed > ServiceAuthorizationManager is not threadsafe > -- > > Key: HADOOP-10590 > URL: https://issues.apache.org/jira/browse/HADOOP-10590 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.4.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Fix For: 2.5.0 > > Attachments: HADOOP-10590.patch, performance-test-without-rpc.patch, > performancetest.patch > > > The mutators in ServiceAuthorizationManager are synchronized. The accessors > are not synchronized. > This results in visibility issues when ServiceAuthorizationManager's state > is accessed from different threads. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10590) ServiceAuthorizationManager is not threadsafe
[ https://issues.apache.org/jira/browse/HADOOP-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034836#comment-14034836 ] Vinayakumar B commented on HADOOP-10590: Thanks [~benoyantony] for the finding and the patch. +1 from my side as well. Thanks [~daryn] for the review. I will commit the patch soon. > ServiceAuthorizationManager is not threadsafe > -- > > Key: HADOOP-10590 > URL: https://issues.apache.org/jira/browse/HADOOP-10590 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.4.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10590.patch, performance-test-without-rpc.patch, > performancetest.patch > > > The mutators in ServiceAuthorizationManager are synchronized. The accessors > are not synchronized. > This results in visibility issues when ServiceAuthorizationManager's state > is accessed from different threads. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10710) hadoop.auth cookie is not properly constructed according to RFC2109
[
https://issues.apache.org/jira/browse/HADOOP-10710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034827#comment-14034827
]
Hadoop QA commented on HADOOP-10710:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650960/HADOOP-10710.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4093//console
This message is automatically generated.
> hadoop.auth cookie is not properly constructed according to RFC2109
> ---
>
> Key: HADOOP-10710
> URL: https://issues.apache.org/jira/browse/HADOOP-10710
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.0
>Reporter: Alejandro Abdelnur
>Assignee: Juan Yu
> Attachments: HADOOP-10710.001.patch
>
>
> It seems that HADOOP-10379 introduced a bug on how hadoop.auth cookies are
> being constructed.
> Before HADOOP-10379, cookies were constructed using Servlet's {{Cookie}}
> class and corresponding {{HttpServletResponse}} methods. This was taking care
> of setting attributes like 'Version=1' and double-quoting the cookie value if
> necessary.
> HADOOP-10379 changed the Cookie creation to use a {{StringBuillder}} and
> setting values and attributes by hand. This is not taking care of setting
> required attributes like Version and escaping the cookie value.
> While this is not breaking HadoopAuth {{AuthenticatedURL}} access, it is
> breaking access done using {{HtttpClient}}. I.e. Solr uses HttpClient and its
> access is broken since this change.
> It seems that HADOOP-10379 main objective was to set the 'secure' attribute.
> Note this can be done using the {{Cookie}} API.
> We should revert the cookie creation logic to use the {{Cookie}} API and take
> care of the security flag via {{setSecure(boolean)}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10530) Make hadoop trunk build on Java7+ only
[
https://issues.apache.org/jira/browse/HADOOP-10530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034815#comment-14034815
]
Alejandro Abdelnur commented on HADOOP-10530:
-
Before we commit this, please lets recap the discussion we had in the mailing
list, what we agreed on, and the guidelines we will follow.
> Make hadoop trunk build on Java7+ only
> --
>
> Key: HADOOP-10530
> URL: https://issues.apache.org/jira/browse/HADOOP-10530
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 3.0.0, 2.4.0
> Environment: Java 1.7+
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-10530-001.patch, HADOOP-10530-002.patch
>
>
> As discussed on hadoop-common, hadoop 3 is envisaged to be Java7+ *only*
> -this JIRA covers switching the build for this
> # maven enforcer plugin to set Java version = {{[1.7)}}
> # compiler to set language to java 1.7
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10710) hadoop.auth cookie is not properly constructed according to RFC2109
[
https://issues.apache.org/jira/browse/HADOOP-10710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Juan Yu updated HADOOP-10710:
-
Status: Patch Available (was: In Progress)
> hadoop.auth cookie is not properly constructed according to RFC2109
> ---
>
> Key: HADOOP-10710
> URL: https://issues.apache.org/jira/browse/HADOOP-10710
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.0
>Reporter: Alejandro Abdelnur
>Assignee: Juan Yu
> Attachments: HADOOP-10710.001.patch
>
>
> It seems that HADOOP-10379 introduced a bug on how hadoop.auth cookies are
> being constructed.
> Before HADOOP-10379, cookies were constructed using Servlet's {{Cookie}}
> class and corresponding {{HttpServletResponse}} methods. This was taking care
> of setting attributes like 'Version=1' and double-quoting the cookie value if
> necessary.
> HADOOP-10379 changed the Cookie creation to use a {{StringBuillder}} and
> setting values and attributes by hand. This is not taking care of setting
> required attributes like Version and escaping the cookie value.
> While this is not breaking HadoopAuth {{AuthenticatedURL}} access, it is
> breaking access done using {{HtttpClient}}. I.e. Solr uses HttpClient and its
> access is broken since this change.
> It seems that HADOOP-10379 main objective was to set the 'secure' attribute.
> Note this can be done using the {{Cookie}} API.
> We should revert the cookie creation logic to use the {{Cookie}} API and take
> care of the security flag via {{setSecure(boolean)}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10710) hadoop.auth cookie is not properly constructed according to RFC2109
[
https://issues.apache.org/jira/browse/HADOOP-10710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Juan Yu updated HADOOP-10710:
-
Attachment: HADOOP-10710.001.patch
I use HttpCookie class to get the cookie string so it will be quoted properly.
then still manually construct the whole cookie so we could keep both secure and
httponly flag. Cookie class doesn't support HttpOnly flag.
I think there is a bug in HttpCookie class, it doesn't remove double quote for
empty token when calling HttpCookie#getValue() so I handle empty token
separately.
> hadoop.auth cookie is not properly constructed according to RFC2109
> ---
>
> Key: HADOOP-10710
> URL: https://issues.apache.org/jira/browse/HADOOP-10710
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.0
>Reporter: Alejandro Abdelnur
>Assignee: Juan Yu
> Attachments: HADOOP-10710.001.patch
>
>
> It seems that HADOOP-10379 introduced a bug on how hadoop.auth cookies are
> being constructed.
> Before HADOOP-10379, cookies were constructed using Servlet's {{Cookie}}
> class and corresponding {{HttpServletResponse}} methods. This was taking care
> of setting attributes like 'Version=1' and double-quoting the cookie value if
> necessary.
> HADOOP-10379 changed the Cookie creation to use a {{StringBuillder}} and
> setting values and attributes by hand. This is not taking care of setting
> required attributes like Version and escaping the cookie value.
> While this is not breaking HadoopAuth {{AuthenticatedURL}} access, it is
> breaking access done using {{HtttpClient}}. I.e. Solr uses HttpClient and its
> access is broken since this change.
> It seems that HADOOP-10379 main objective was to set the 'secure' attribute.
> Note this can be done using the {{Cookie}} API.
> We should revert the cookie creation logic to use the {{Cookie}} API and take
> care of the security flag via {{setSecure(boolean)}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Work started] (HADOOP-10710) hadoop.auth cookie is not properly constructed according to RFC2109
[
https://issues.apache.org/jira/browse/HADOOP-10710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HADOOP-10710 started by Juan Yu.
> hadoop.auth cookie is not properly constructed according to RFC2109
> ---
>
> Key: HADOOP-10710
> URL: https://issues.apache.org/jira/browse/HADOOP-10710
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.0
>Reporter: Alejandro Abdelnur
>Assignee: Juan Yu
>
> It seems that HADOOP-10379 introduced a bug on how hadoop.auth cookies are
> being constructed.
> Before HADOOP-10379, cookies were constructed using Servlet's {{Cookie}}
> class and corresponding {{HttpServletResponse}} methods. This was taking care
> of setting attributes like 'Version=1' and double-quoting the cookie value if
> necessary.
> HADOOP-10379 changed the Cookie creation to use a {{StringBuillder}} and
> setting values and attributes by hand. This is not taking care of setting
> required attributes like Version and escaping the cookie value.
> While this is not breaking HadoopAuth {{AuthenticatedURL}} access, it is
> breaking access done using {{HtttpClient}}. I.e. Solr uses HttpClient and its
> access is broken since this change.
> It seems that HADOOP-10379 main objective was to set the 'secure' attribute.
> Note this can be done using the {{Cookie}} API.
> We should revert the cookie creation logic to use the {{Cookie}} API and take
> care of the security flag via {{setSecure(boolean)}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
common-issues@hadoop.apache.org
[ https://issues.apache.org/jira/browse/HADOOP-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang resolved HADOOP-10636. Resolution: Fixed Fix Version/s: HADOOP-10388 > Native Hadoop Client:add unit test case for call&client_id > -- > > Key: HADOOP-10636 > URL: https://issues.apache.org/jira/browse/HADOOP-10636 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-10388 >Reporter: Wenwu Peng >Assignee: Wenwu Peng > Fix For: HADOOP-10388 > > Attachments: HADOOP-10636-pnative.001.patch, > HADOOP-10636-pnative.002.patch, HADOOP-10636-pnative.003.patch, > HADOOP-10636-pnative.004.patch, HADOOP-10636-pnative.005.patch, > HADOOP-10636-pnative.006.patch, HADOOP-10636-pnative.007.patch, > HADOOP-10636-pnative.008-commit.patch, HADOOP-10636-pnative.008.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
common-issues@hadoop.apache.org
[ https://issues.apache.org/jira/browse/HADOOP-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated HADOOP-10636: --- Attachment: HADOOP-10636-pnative.008-commit.patch committed. Thanks, Wenwu. > Native Hadoop Client:add unit test case for call&client_id > -- > > Key: HADOOP-10636 > URL: https://issues.apache.org/jira/browse/HADOOP-10636 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-10388 >Reporter: Wenwu Peng >Assignee: Wenwu Peng > Attachments: HADOOP-10636-pnative.001.patch, > HADOOP-10636-pnative.002.patch, HADOOP-10636-pnative.003.patch, > HADOOP-10636-pnative.004.patch, HADOOP-10636-pnative.005.patch, > HADOOP-10636-pnative.006.patch, HADOOP-10636-pnative.007.patch, > HADOOP-10636-pnative.008-commit.patch, HADOOP-10636-pnative.008.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
common-issues@hadoop.apache.org
[ https://issues.apache.org/jira/browse/HADOOP-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034770#comment-14034770 ] Binglin Chang commented on HADOOP-10636: lgtm, +1, with minor formatting changes > Native Hadoop Client:add unit test case for call&client_id > -- > > Key: HADOOP-10636 > URL: https://issues.apache.org/jira/browse/HADOOP-10636 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-10388 >Reporter: Wenwu Peng >Assignee: Wenwu Peng > Attachments: HADOOP-10636-pnative.001.patch, > HADOOP-10636-pnative.002.patch, HADOOP-10636-pnative.003.patch, > HADOOP-10636-pnative.004.patch, HADOOP-10636-pnative.005.patch, > HADOOP-10636-pnative.006.patch, HADOOP-10636-pnative.007.patch, > HADOOP-10636-pnative.008.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10705) Fix namenode-rpc-unit warning reported by memory leak check tool(valgrind)
[ https://issues.apache.org/jira/browse/HADOOP-10705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034726#comment-14034726 ] Wenwu Peng commented on HADOOP-10705: - Thanks Colin 's detail explanation. Try the in other Linux distro(CentOS 6.4) ,the valgrind report about getpwuid_r go away, however, there is a report about uv__loop_init still exist, need to care about? ==24085== 128 bytes in 1 blocks are definitely lost in loss record 18 of 20 ==24085==at 0x4A069EE: malloc (vg_replace_malloc.c:270) ==24085==by 0x4A06B62: realloc (vg_replace_malloc.c:662) ==24085==by 0x4C2BD6A: uv__io_start (core.c:679) ==24085==by 0x4C31868: uv_signal_init (signal.c:225) ==24085==by 0x4C3000E: uv__loop_init (loop.c:136) ==24085==by 0x4275EF: hrpc_reactor_create (reactor.c:183) ==24085==by 0x426829: hrpc_messenger_create (messenger.c:83) ==24085==by 0x42590F: main (namenode-rpc-unit.c:115) ==24085== > Fix namenode-rpc-unit warning reported by memory leak check tool(valgrind) > --- > > Key: HADOOP-10705 > URL: https://issues.apache.org/jira/browse/HADOOP-10705 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-10388 >Reporter: Wenwu Peng > Attachments: vargrind.log > > > Rum valgrind to check memory leak for namenode-rpc.unit. > There are many warning, need to fix it. > valgrind --tool=memcheck --leak-check=full --show-reachable=yes > ./namenode-rpc-unit > ==24085== > ==24085== HEAP SUMMARY: > ==24085== in use at exit: 1,216 bytes in 20 blocks > ==24085== total heap usage: 89 allocs, 69 frees, 12,222 bytes allocated > ==24085== > ==24085== 128 bytes in 1 blocks are definitely lost in loss record 18 of 20 > ==24085==at 0x4A069EE: malloc (vg_replace_malloc.c:270) > ==24085==by 0x4A06B62: realloc (vg_replace_malloc.c:662) > ==24085==by 0x4C2BD6A: uv__io_start (core.c:679) > ==24085==by 0x4C31868: uv_signal_init (signal.c:225) > ==24085==by 0x4C3000E: uv__loop_init (loop.c:136) > ==24085==by 0x4275EF: hrpc_reactor_create (reactor.c:183) > ==24085==by 0x426829: hrpc_messenger_create (messenger.c:83) > ==24085==by 0x42590F: main (namenode-rpc-unit.c:115) > ==24085== > ==24085== LEAK SUMMARY: > ==24085==definitely lost: 128 bytes in 1 blocks > ==24085==indirectly lost: 0 bytes in 0 blocks > ==24085== possibly lost: 0 bytes in 0 blocks > ==24085==still reachable: 1,088 bytes in 19 blocks > ==24085== suppressed: 0 bytes in 0 blocks > ==24085== Reachable blocks (those to which a pointer was found) are not shown. > ==24085== To see them, rerun with: --leak-check=full --show-reachable=yes > ==24085== > ==24085== For counts of detected and suppressed errors, rerun with: -v > ==24085== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 6 from 6) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10705) Fix namenode-rpc-unit warning reported by memory leak check tool(valgrind)
[ https://issues.apache.org/jira/browse/HADOOP-10705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenwu Peng updated HADOOP-10705: Description: Rum valgrind to check memory leak for namenode-rpc.unit. There are many warning, need to fix it. valgrind --tool=memcheck --leak-check=full --show-reachable=yes ./namenode-rpc-unit ==24085== ==24085== HEAP SUMMARY: ==24085== in use at exit: 1,216 bytes in 20 blocks ==24085== total heap usage: 89 allocs, 69 frees, 12,222 bytes allocated ==24085== ==24085== 128 bytes in 1 blocks are definitely lost in loss record 18 of 20 ==24085==at 0x4A069EE: malloc (vg_replace_malloc.c:270) ==24085==by 0x4A06B62: realloc (vg_replace_malloc.c:662) ==24085==by 0x4C2BD6A: uv__io_start (core.c:679) ==24085==by 0x4C31868: uv_signal_init (signal.c:225) ==24085==by 0x4C3000E: uv__loop_init (loop.c:136) ==24085==by 0x4275EF: hrpc_reactor_create (reactor.c:183) ==24085==by 0x426829: hrpc_messenger_create (messenger.c:83) ==24085==by 0x42590F: main (namenode-rpc-unit.c:115) ==24085== ==24085== LEAK SUMMARY: ==24085==definitely lost: 128 bytes in 1 blocks ==24085==indirectly lost: 0 bytes in 0 blocks ==24085== possibly lost: 0 bytes in 0 blocks ==24085==still reachable: 1,088 bytes in 19 blocks ==24085== suppressed: 0 bytes in 0 blocks ==24085== Reachable blocks (those to which a pointer was found) are not shown. ==24085== To see them, rerun with: --leak-check=full --show-reachable=yes ==24085== ==24085== For counts of detected and suppressed errors, rerun with: -v ==24085== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 6 from 6) was: Rum valgrind to check memory leak for namenode-rpc.unit. There are many warning, need to fix it. valgrind --tool=memcheck --leak-check=full --show-reachable=yes ./namenode-rpc-unit ==10462== Memcheck, a memory error detector ==10462== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al. ==10462== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info ==10462== Command: ./namenode-rpc-unit ==10462== ==10462== HEAP SUMMARY: ==10462== in use at exit: 428 bytes in 12 blocks ==10462== total heap usage: 91 allocs, 79 frees, 16,056 bytes allocated ==10462== ==10462== 16 bytes in 1 blocks are indirectly lost in loss record 1 of 12 ==10462==at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==10462==by 0x557EB99: __nss_lookup_function (nsswitch.c:456) ==10462==by 0x604863E: ??? ==10462==by 0x553744C: getpwuid_r@@GLIBC_2.2.5 (getXXbyYY_r.c:256) ==10462==by 0x42681E: geteuid_string (user.c:67) ==10462==by 0x425ABD: main (namenode-rpc-unit.c:71) > Fix namenode-rpc-unit warning reported by memory leak check tool(valgrind) > --- > > Key: HADOOP-10705 > URL: https://issues.apache.org/jira/browse/HADOOP-10705 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-10388 >Reporter: Wenwu Peng > Attachments: vargrind.log > > > Rum valgrind to check memory leak for namenode-rpc.unit. > There are many warning, need to fix it. > valgrind --tool=memcheck --leak-check=full --show-reachable=yes > ./namenode-rpc-unit > ==24085== > ==24085== HEAP SUMMARY: > ==24085== in use at exit: 1,216 bytes in 20 blocks > ==24085== total heap usage: 89 allocs, 69 frees, 12,222 bytes allocated > ==24085== > ==24085== 128 bytes in 1 blocks are definitely lost in loss record 18 of 20 > ==24085==at 0x4A069EE: malloc (vg_replace_malloc.c:270) > ==24085==by 0x4A06B62: realloc (vg_replace_malloc.c:662) > ==24085==by 0x4C2BD6A: uv__io_start (core.c:679) > ==24085==by 0x4C31868: uv_signal_init (signal.c:225) > ==24085==by 0x4C3000E: uv__loop_init (loop.c:136) > ==24085==by 0x4275EF: hrpc_reactor_create (reactor.c:183) > ==24085==by 0x426829: hrpc_messenger_create (messenger.c:83) > ==24085==by 0x42590F: main (namenode-rpc-unit.c:115) > ==24085== > ==24085== LEAK SUMMARY: > ==24085==definitely lost: 128 bytes in 1 blocks > ==24085==indirectly lost: 0 bytes in 0 blocks > ==24085== possibly lost: 0 bytes in 0 blocks > ==24085==still reachable: 1,088 bytes in 19 blocks > ==24085== suppressed: 0 bytes in 0 blocks > ==24085== Reachable blocks (those to which a pointer was found) are not shown. > ==24085== To see them, rerun with: --leak-check=full --show-reachable=yes > ==24085== > ==24085== For counts of detected and suppressed errors, rerun with: -v > ==24085== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 6 from 6) -- This message was sent by Atlassian JIRA (v6.2#6252)
common-issues@hadoop.apache.org
[ https://issues.apache.org/jira/browse/HADOOP-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenwu Peng updated HADOOP-10636: Attachment: HADOOP-10636-pnative.008.patch Thanks Colin 's nice and comments. :) Version 8 address the comments. > Native Hadoop Client:add unit test case for call&client_id > -- > > Key: HADOOP-10636 > URL: https://issues.apache.org/jira/browse/HADOOP-10636 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HADOOP-10388 >Reporter: Wenwu Peng >Assignee: Wenwu Peng > Attachments: HADOOP-10636-pnative.001.patch, > HADOOP-10636-pnative.002.patch, HADOOP-10636-pnative.003.patch, > HADOOP-10636-pnative.004.patch, HADOOP-10636-pnative.005.patch, > HADOOP-10636-pnative.006.patch, HADOOP-10636-pnative.007.patch, > HADOOP-10636-pnative.008.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10530) Make hadoop trunk build on Java7+ only
[
https://issues.apache.org/jira/browse/HADOOP-10530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034697#comment-14034697
]
Steve Loughran commented on HADOOP-10530:
-
Andew: yes. Branch-2 will always have the right to backport this patch to adopt
new features, but we can't let the java version policy of branch 2 stop us
upgrading trunk
> Make hadoop trunk build on Java7+ only
> --
>
> Key: HADOOP-10530
> URL: https://issues.apache.org/jira/browse/HADOOP-10530
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 3.0.0, 2.4.0
> Environment: Java 1.7+
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-10530-001.patch, HADOOP-10530-002.patch
>
>
> As discussed on hadoop-common, hadoop 3 is envisaged to be Java7+ *only*
> -this JIRA covers switching the build for this
> # maven enforcer plugin to set Java version = {{[1.7)}}
> # compiler to set language to java 1.7
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10589) NativeS3FileSystem throw NullPointerException when the file is empty
[ https://issues.apache.org/jira/browse/HADOOP-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034657#comment-14034657 ] Steve Loughran commented on HADOOP-10589: - HADOOP-10457 had found this, but this JIRA found it in production and has more text > NativeS3FileSystem throw NullPointerException when the file is empty > > > Key: HADOOP-10589 > URL: https://issues.apache.org/jira/browse/HADOOP-10589 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.2.0 >Reporter: shuisheng wei >Assignee: Steve Loughran > > An empty file in the s3 path. > NativeS3FsInputStream dose not check the InputStream . > 2014-05-06 20:29:26,961 INFO [main] > org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: 4 forwarded 0 rows > 2014-05-06 20:29:26,961 INFO [main] > org.apache.hadoop.hive.ql.exec.GroupByOperator: 3 Close done > 2014-05-06 20:29:26,961 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 2 Close done > 2014-05-06 20:29:26,961 INFO [main] > org.apache.hadoop.hive.ql.exec.FilterOperator: 1 Close done > 2014-05-06 20:29:26,961 INFO [main] > org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 Close done > 2014-05-06 20:29:26,961 INFO [main] > org.apache.hadoop.hive.ql.exec.MapOperator: 5 Close done > 2014-05-06 20:29:26,961 INFO [main] > org.apache.hadoop.hive.ql.exec.mr.ExecMapper: ExecMapper: processed 0 rows: > used memory = 602221488 > 2014-05-06 20:29:26,964 WARN [main] org.apache.hadoop.mapred.YarnChild: > Exception running child : java.lang.NullPointerException > at > org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.close(NativeS3FileSystem.java:147) > at java.io.BufferedInputStream.close(BufferedInputStream.java:472) > at java.io.FilterInputStream.close(FilterInputStream.java:181) > at org.apache.hadoop.util.LineReader.close(LineReader.java:150) > at > org.apache.hadoop.mapred.LineRecordReader.close(LineRecordReader.java:244) > at > org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doClose(CombineHiveRecordReader.java:72) > at > org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.close(HiveContextAwareRecordReader.java:96) > at > org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.close(HadoopShimsSecure.java:248) > at > org.apache.hadoop.mapred.MapTask$TrackedRecordReader.close(MapTask.java:209) > at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1950) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:445) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) > 2014-05-06 20:29:26,970 INFO [main] org.apache.hadoop.mapred.Task: Runnning > cleanup for the task -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10457) S3N NPEs if you do a read() after a seek() past the EOF
[ https://issues.apache.org/jira/browse/HADOOP-10457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-10457: Summary: S3N NPEs if you do a read() after a seek() past the EOF (was: S3N NPEs if you do a read() after a seek()) > S3N NPEs if you do a read() after a seek() past the EOF > --- > > Key: HADOOP-10457 > URL: https://issues.apache.org/jira/browse/HADOOP-10457 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 3.0.0, 2.4.0 >Reporter: Steve Loughran >Assignee: Allen Wittenauer >Priority: Minor > > if you do a seek past the EOF of an S3n file > # it doesn't throw any exception > # on the next read, you get to see a stack trace -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-10457) S3N NPEs if you do a read() after a seek() past the EOF
[ https://issues.apache.org/jira/browse/HADOOP-10457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-10457. - Resolution: Duplicate > S3N NPEs if you do a read() after a seek() past the EOF > --- > > Key: HADOOP-10457 > URL: https://issues.apache.org/jira/browse/HADOOP-10457 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 3.0.0, 2.4.0 >Reporter: Steve Loughran >Assignee: Allen Wittenauer >Priority: Minor > > if you do a seek past the EOF of an S3n file > # it doesn't throw any exception > # on the next read, you get to see a stack trace -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10589) NativeS3FileSystem throw NullPointerException when the file is empty
[ https://issues.apache.org/jira/browse/HADOOP-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-10589: Release Note: (was: Duplicate of HADOOP-10533, though the stack trace is more up to date on this one) > NativeS3FileSystem throw NullPointerException when the file is empty > > > Key: HADOOP-10589 > URL: https://issues.apache.org/jira/browse/HADOOP-10589 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.2.0 >Reporter: shuisheng wei >Assignee: Steve Loughran > > An empty file in the s3 path. > NativeS3FsInputStream dose not check the InputStream . > 2014-05-06 20:29:26,961 INFO [main] > org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: 4 forwarded 0 rows > 2014-05-06 20:29:26,961 INFO [main] > org.apache.hadoop.hive.ql.exec.GroupByOperator: 3 Close done > 2014-05-06 20:29:26,961 INFO [main] > org.apache.hadoop.hive.ql.exec.SelectOperator: 2 Close done > 2014-05-06 20:29:26,961 INFO [main] > org.apache.hadoop.hive.ql.exec.FilterOperator: 1 Close done > 2014-05-06 20:29:26,961 INFO [main] > org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 Close done > 2014-05-06 20:29:26,961 INFO [main] > org.apache.hadoop.hive.ql.exec.MapOperator: 5 Close done > 2014-05-06 20:29:26,961 INFO [main] > org.apache.hadoop.hive.ql.exec.mr.ExecMapper: ExecMapper: processed 0 rows: > used memory = 602221488 > 2014-05-06 20:29:26,964 WARN [main] org.apache.hadoop.mapred.YarnChild: > Exception running child : java.lang.NullPointerException > at > org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.close(NativeS3FileSystem.java:147) > at java.io.BufferedInputStream.close(BufferedInputStream.java:472) > at java.io.FilterInputStream.close(FilterInputStream.java:181) > at org.apache.hadoop.util.LineReader.close(LineReader.java:150) > at > org.apache.hadoop.mapred.LineRecordReader.close(LineRecordReader.java:244) > at > org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doClose(CombineHiveRecordReader.java:72) > at > org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.close(HiveContextAwareRecordReader.java:96) > at > org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.close(HadoopShimsSecure.java:248) > at > org.apache.hadoop.mapred.MapTask$TrackedRecordReader.close(MapTask.java:209) > at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1950) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:445) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) > 2014-05-06 20:29:26,970 INFO [main] org.apache.hadoop.mapred.Task: Runnning > cleanup for the task -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[
https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034650#comment-14034650
]
Daryn Sharp commented on HADOOP-8989:
-
Do you actually need the changes to {{Command}} and {{CommandFactory}} to add a
new command?
I'm not sure the path handling/expansion is consistent with the rest of the
commands. Find is a bit odd in that paths come before all options, but I think
there has to be a cleaner way to implement the parsing.
> hadoop dfs -find feature
>
>
> Key: HADOOP-8989
> URL: https://issues.apache.org/jira/browse/HADOOP-8989
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Marco Nicosia
>Assignee: Jonathan Allen
> Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
> HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
> HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
> HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
> HADOOP-8989.patch
>
>
> Both sysadmins and users make frequent use of the unix 'find' command, but
> Hadoop has no correlate. Without this, users are writing scripts which make
> heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs
> -lsr is somewhat taxing on the NameNode, and a really slow experience on the
> client side. Possibly an in-NameNode find operation would be only a bit more
> taxing on the NameNode, but significantly faster from the client's point of
> view?
> The minimum set of options I can think of which would make a Hadoop find
> command generally useful is (in priority order):
> * -type (file or directory, for now)
> * -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments)
> * -print0 (for piping to xargs -0)
> * -depth
> * -owner/-group (and -nouser/-nogroup)
> * -name (allowing for shell pattern, or even regex?)
> * -perm
> * -size
> One possible special case, but could possibly be really cool if it ran from
> within the NameNode:
> * -delete
> The "hadoop dfs -lsr | hadoop dfs -rm" cycle is really, really slow.
> Lower priority, some people do use operators, mostly to execute -or searches
> such as:
> * find / \(-nouser -or -nogroup\)
> Finally, I thought I'd include a link to the [Posix spec for
> find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html]
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-9705) FsShell cp -p does not preserve directory attibutes
[
https://issues.apache.org/jira/browse/HADOOP-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034609#comment-14034609
]
Hadoop QA commented on HADOOP-9705:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650873/HADOOP-9705.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4090//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4090//console
This message is automatically generated.
> FsShell cp -p does not preserve directory attibutes
> ---
>
> Key: HADOOP-9705
> URL: https://issues.apache.org/jira/browse/HADOOP-9705
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Stephen Chu
>Assignee: Akira AJISAKA
> Attachments: HADOOP-9705.2.patch, HADOOP-9705.3.patch,
> HADOOP-9705.4.patch, HADOOP-9705.5.patch, HADOOP-9705.patch
>
>
> HADOOP-9338 added the -p flag to preserve file attributes when copying.
> However, cp -p does not preserve directory attributes. It'd be useful to add
> this functionality.
> For example, the following shows that the modified time is not preserved
> {code}
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -mkdir
> /user/schu/testDir1
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu/
> Found 1 items
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -cp -p
> /user/schu/testDir1 /user/schu/testDir2
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu
> Found 2 items
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:35 /user/schu/testDir2
> [schu@hdfs-snapshots-1 ~]$
> {code}
> The preserve logic is in CommandWithDestination#copyFileToTarget, which is
> only called with files.
> {code}
> protected void processPath(PathData src, PathData dst) throws IOException {
> if (src.stat.isSymlink()) {
> // TODO: remove when FileContext is supported, this needs to either
>
>
> // copy the symlink or deref the symlink
>
>
> throw new PathOperationException(src.toString());
> } else if (src.stat.isFile()) {
> copyFileToTarget(src, dst);
> } else if (src.stat.isDirectory() && !isRecursive()) {
> throw new PathIsDirectoryException(src.toString());
> }
> }
> {code}
> {code}
> /**
>
>
>* Copies the source file to the target.
>
>
>* @param src item to copy
>
>
>* @param target where to copy the item
>
>
>* @throws IOException if copy fails
>
>
>*/
> protected void copyFileToTarget(PathData src, PathData target) throws
> IOException {
> src.fs.setVerifyChecksum(verifyChecksum);
> if
[jira] [Commented] (HADOOP-10716) Cannot use more than 1 har filesystem
[
https://issues.apache.org/jira/browse/HADOOP-10716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034595#comment-14034595
]
Hadoop QA commented on HADOOP-10716:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650904/HADOOP-10716.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4092//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4092//console
This message is automatically generated.
> Cannot use more than 1 har filesystem
> -
>
> Key: HADOOP-10716
> URL: https://issues.apache.org/jira/browse/HADOOP-10716
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf, fs
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Daryn Sharp
>Priority: Critical
> Attachments: HADOOP-10716.patch
>
>
> Filesystems are cached purely on scheme + authority. Har filesystems
> actually need further differentiation based on path to the har file itself.
> For this reason, the fs cache used to be explicitly disable for har via
> "fs.har.impl.cache.disable" in core-default.xml.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10389) Native RPCv9 client
[
https://issues.apache.org/jira/browse/HADOOP-10389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034585#comment-14034585
]
Haohui Mai commented on HADOOP-10389:
-
Thanks [~decster] for the input. This could be another direction to go. Some of
my thoughts:
bq. 1. centos does not have enough support for c+11, c+11 is not generally
available yet
The POC patch uses a few syntactic sugars in c++11, which can be easily
implemented in C++TR1 (Note that the current native code is using
{{std::shared_ptr}} from tr1 already). The tougher part is that the private
APIs passes the {{std::error_code}} objects for error conditions, it can be
fixed by passing int or {{boost::system::error_code}} instead.
bq. 2. remain libhdfs compatibility, since libhdfs is written in c, we might
just continue using c as well
If the ultimate goal is to implement the libhdfs interface, there should be no
c++ symbols exported in the library, thus there is really no compatibility
issue from this regard. As a result, c++ remains as an implementation detail
only.
bq. 1. the protobuf-c library is just not so reliable as official protobuf
library which is maintained and verified by google and many other
companies/projects, I read some of the protobuf-c code, it uses a reflection
style implementation to do serializing/deserializing, so performance, security,
compatibility may all at risk. I see https://github.com/protobuf-c/protobuf-c
only have 92 stars.
I think this is a legit concern. It might be better to depend only on widely
deployed, well-tested libraries.
> Native RPCv9 client
> ---
>
> Key: HADOOP-10389
> URL: https://issues.apache.org/jira/browse/HADOOP-10389
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: HADOOP-10388
>Reporter: Binglin Chang
>Assignee: Colin Patrick McCabe
> Attachments: HADOOP-10388.001.patch,
> HADOOP-10389-alternative.000.patch, HADOOP-10389.002.patch,
> HADOOP-10389.004.patch, HADOOP-10389.005.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Comment Edited] (HADOOP-10389) Native RPCv9 client
[
https://issues.apache.org/jira/browse/HADOOP-10389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034560#comment-14034560
]
Haohui Mai edited comment on HADOOP-10389 at 6/17/14 11:05 PM:
---
bq. didn't integrate the existing JNIFS code, are not parsing the config XMLs,
don't handle errors, etc. etc.
Correct me if I'm wrong -- based on my understanding the original patch in
HADOOP-10369 implements none of them above. Both implementation can benefit
from other jiras (e.g. HADOOP-10446).
bq. ..., because you didn't implement generation of type-safe RPC wrappers (you
just wrote out everything manually in HdfsImpl::Exists),
This is implemented as C++ code in the original patch. It can be brought in if
it is required.
For what is worth, do you agree that the code implements comparable
functionality with much smaller code base?
bq. boost::asio doesn't provide equivalent functionality to libuv. It does wrap
a lot of network stuff, but doesn't handle things like condition variables,
signals, etc. etc., ..., For example, the boost condition variable actually can
let your thread wake up without recovering the lock (it then has to sleep
again).
I'm unsure why this is relevant as I don't see the current code uses condition
variables and signals. My guess is that you might need them to implement RPC
with timeout. I'm yet to be convinced that this is required in the c++
implementation, as {{std::future}} provides a much better abstraction for that.
bq. It uses exceptions, which are problematic.
This is not a fundamental part of the API. The current implementation uses the
exception-free variants of the library calls whenever possible. The
{{std::system_error}} object carries the error conditions.
The current implementation of the rpc-send path, however, might throw
{{bad_alloc}}. It can be fixed by limiting the maximum size of the outgoing
buffer.
I say it is safe to define {{BOOOST_NO_EXCEPTIONS}} and compile the code with
{{-fno-exceptions}}.
bq. Yes, you can use boost for some of that stuff, but the boost versions are
often not that good.
Boost in general has a versioning problem which is really bad for libraries. If
your library links against version X of boost, any application linking to you
needs to use the same version... otherwise bad stuff happens. You could link it
statically, but it might be too big for that to work (I've never tried).
The only thing required is {{boost::system::error}}, which have a fairly small
foolprints (~70K on Mac OS X). The size is comparable to the size of
{{protobuf-c}} (~75K on Mac OS X).
bq. To me this is not that interesting. There are a lot of clients already out
there in various languages... C, C++, even golang.
In my opinion the same argument applies to any clients, none of them is
superior to the others. Everybody can write their own clients. Contributions
are welcome, but for a community takes over the code and to maintain it, it is
important to ensure that the code can be easily changed, reviewed, and
maintained by other contributors.
This key point, however, remains unaddressed in the the current branch.
bq. If you're interested in something even smaller and more minimal, you can
check out this: https://github.com/toddlipcon/libhrpc/ Though I think that that
is for an out-of-date version of hadoop RPC at this point.
Given the fact that many hadoop RPC clients happily sit on github, and the
protocol remains pretty stable, is it more appropriate to host this project on
github if you don't have time to address the comments?
was (Author: wheat9):
bq. didn't integrate the existing JNIFS code, are not parsing the config XMLs,
don't handle errors, etc. etc.
Correct me if I'm wrong -- based on my understanding the original patch in
HADOOP-10369 implements none of them above. Both implementation can benefit
from other jiras (e.g. HADOOP-10446).
bq. ..., because you didn't implement generation of type-safe RPC wrappers (you
just wrote out everything manually in HdfsImpl::Exists),
This is implemented as C++ code in the original patch. It can be brought in if
it is required.
For what is worth, do you agree that the code implements comparable
functionality with much smaller code base?
bq. boost::asio doesn't provide equivalent functionality to libuv. It does wrap
a lot of network stuff, but doesn't handle things like condition variables,
signals, etc. etc., ..., For example, the boost condition variable actually can
let your thread wake up without recovering the lock (it then has to sleep
again).
I'm unsure why this is relevant as I don't see the current code uses condition
variables and signals. My guess is that you might need them to implement RPC
with timeout. I'm yet to be convinced that this is required in the c++
implementation, as {{std::future}} provides a much better abstraction for that.
[jira] [Commented] (HADOOP-10389) Native RPCv9 client
[
https://issues.apache.org/jira/browse/HADOOP-10389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034560#comment-14034560
]
Haohui Mai commented on HADOOP-10389:
-
bq. didn't integrate the existing JNIFS code, are not parsing the config XMLs,
don't handle errors, etc. etc.
Correct me if I'm wrong -- based on my understanding the original patch in
HADOOP-10369 implements none of them above. Both implementation can benefit
from other jiras (e.g. HADOOP-10446).
bq. ..., because you didn't implement generation of type-safe RPC wrappers (you
just wrote out everything manually in HdfsImpl::Exists),
This is implemented as C++ code in the original patch. It can be brought in if
it is required.
For what is worth, do you agree that the code implements comparable
functionality with much smaller code base?
bq. boost::asio doesn't provide equivalent functionality to libuv. It does wrap
a lot of network stuff, but doesn't handle things like condition variables,
signals, etc. etc., ..., For example, the boost condition variable actually can
let your thread wake up without recovering the lock (it then has to sleep
again).
I'm unsure why this is relevant as I don't see the current code uses condition
variables and signals. My guess is that you might need them to implement RPC
with timeout. I'm yet to be convinced that this is required in the c++
implementation, as {{std::future}} provides a much better abstraction for that.
bq. It uses exceptions, which are problematic.
This is not a fundamental part of the API. The current implementation uses the
exception-free variants of the library calls whenever possible. The
{{std::system_error}} object carries the error conditions.
The current implementation of the rpc-send path, however, might throw
{{bad_alloc}}. It can be fixed by limiting the maximum size of the outgoing
buffer.
I say it is safe to define {{BOOOST_NO_EXCEPTIONS}} and compile the code with
{{-fno-exceptions}}.
Yes, you can use boost for some of that stuff, but the boost versions are often
not that good.
Boost in general has a versioning problem which is really bad for libraries. If
your library links against version X of boost, any application linking to you
needs to use the same version... otherwise bad stuff happens. You could link it
statically, but it might be too big for that to work (I've never tried).
bq. The only thing required is {{boost::system::error}}, which have a fairly
small foolprints (~70K on Mac OS X). The size is comparable to the size of
{{protobuf-c}} (~75K on Mac OS X).
bq. To me this is not that interesting. There are a lot of clients already out
there in various languages... C, C++, even golang.
In my opinion the same argument applies to any clients, none of them is
superior to the others. Everybody can write their own clients. Contributions
are welcome, but for a community takes over the code and to maintain it, it is
important to ensure that the code can be easily changed, reviewed, and
maintained by other contributors.
This key point, however, remains unaddressed in the the current branch.
bq. If you're interested in something even smaller and more minimal, you can
check out this: https://github.com/toddlipcon/libhrpc/ Though I think that that
is for an out-of-date version of hadoop RPC at this point.
Given the fact that many hadoop RPC clients happily sit on github, and the
protocol remains pretty stable, is it more appropriate to host this project on
github if you don't have time to address the comments?
> Native RPCv9 client
> ---
>
> Key: HADOOP-10389
> URL: https://issues.apache.org/jira/browse/HADOOP-10389
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: HADOOP-10388
>Reporter: Binglin Chang
>Assignee: Colin Patrick McCabe
> Attachments: HADOOP-10388.001.patch,
> HADOOP-10389-alternative.000.patch, HADOOP-10389.002.patch,
> HADOOP-10389.004.patch, HADOOP-10389.005.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10530) Make hadoop trunk build on Java7+ only
[
https://issues.apache.org/jira/browse/HADOOP-10530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034545#comment-14034545
]
Hadoop QA commented on HADOOP-10530:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650891/HADOOP-10530-002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4091//console
This message is automatically generated.
> Make hadoop trunk build on Java7+ only
> --
>
> Key: HADOOP-10530
> URL: https://issues.apache.org/jira/browse/HADOOP-10530
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 3.0.0, 2.4.0
> Environment: Java 1.7+
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-10530-001.patch, HADOOP-10530-002.patch
>
>
> As discussed on hadoop-common, hadoop 3 is envisaged to be Java7+ *only*
> -this JIRA covers switching the build for this
> # maven enforcer plugin to set Java version = {{[1.7)}}
> # compiler to set language to java 1.7
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10530) Make hadoop trunk build on Java7+ only
[
https://issues.apache.org/jira/browse/HADOOP-10530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034519#comment-14034519
]
Andrew Wang commented on HADOOP-10530:
--
Are we going to start allowing JDK7-only APIs and libraries to be used in
trunk? I think last time this was brought up on the mailing list, there were
concerns about backports to branch-2 becoming more difficult if branch-2 and
trunk started diverging because of things like this.
> Make hadoop trunk build on Java7+ only
> --
>
> Key: HADOOP-10530
> URL: https://issues.apache.org/jira/browse/HADOOP-10530
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 3.0.0, 2.4.0
> Environment: Java 1.7+
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-10530-001.patch, HADOOP-10530-002.patch
>
>
> As discussed on hadoop-common, hadoop 3 is envisaged to be Java7+ *only*
> -this JIRA covers switching the build for this
> # maven enforcer plugin to set Java version = {{[1.7)}}
> # compiler to set language to java 1.7
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[
https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034491#comment-14034491
]
Akira AJISAKA commented on HADOOP-8989:
---
Thanks for updating the patch! Mostly looks good. I verified -print0 option and
the following command worked well.
{code}
# hdfs dfs -find /user/root/ -name "file*" -print0 | xargs -0 hdfs dfs -rm
{code}
Minor comment:
{code}
private static String[] HELP =
{ "Finds all files that match the specified expression and "
+ "applies selected actions to them." };
{code}
Would you change the code like other commands? i.e.
{code}
private static String[] HELP =
{ "Finds all files that match the specified expression and",
"applies selected actions to them." };
{code}
In addition, can you please include a document in the patch for keeping
consistency between source code and document? (i.e. Would you split
HADOOP-10580?)
[~umamaheswararao], do you have any comments on this patch? I'll appreciate
your reviews.
> hadoop dfs -find feature
>
>
> Key: HADOOP-8989
> URL: https://issues.apache.org/jira/browse/HADOOP-8989
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Marco Nicosia
>Assignee: Jonathan Allen
> Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
> HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
> HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
> HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
> HADOOP-8989.patch
>
>
> Both sysadmins and users make frequent use of the unix 'find' command, but
> Hadoop has no correlate. Without this, users are writing scripts which make
> heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs
> -lsr is somewhat taxing on the NameNode, and a really slow experience on the
> client side. Possibly an in-NameNode find operation would be only a bit more
> taxing on the NameNode, but significantly faster from the client's point of
> view?
> The minimum set of options I can think of which would make a Hadoop find
> command generally useful is (in priority order):
> * -type (file or directory, for now)
> * -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments)
> * -print0 (for piping to xargs -0)
> * -depth
> * -owner/-group (and -nouser/-nogroup)
> * -name (allowing for shell pattern, or even regex?)
> * -perm
> * -size
> One possible special case, but could possibly be really cool if it ran from
> within the NameNode:
> * -delete
> The "hadoop dfs -lsr | hadoop dfs -rm" cycle is really, really slow.
> Lower priority, some people do use operators, mostly to execute -or searches
> such as:
> * find / \(-nouser -or -nogroup\)
> Finally, I thought I'd include a link to the [Posix spec for
> find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html]
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10716) Cannot use more than 1 har filesystem
[ https://issues.apache.org/jira/browse/HADOOP-10716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HADOOP-10716: Attachment: HADOOP-10716.patch Added a config fs.har.impl.disable.cache in core-default.xml. > Cannot use more than 1 har filesystem > - > > Key: HADOOP-10716 > URL: https://issues.apache.org/jira/browse/HADOOP-10716 > Project: Hadoop Common > Issue Type: Bug > Components: conf, fs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Priority: Critical > Attachments: HADOOP-10716.patch > > > Filesystems are cached purely on scheme + authority. Har filesystems > actually need further differentiation based on path to the har file itself. > For this reason, the fs cache used to be explicitly disable for har via > "fs.har.impl.cache.disable" in core-default.xml. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10716) Cannot use more than 1 har filesystem
[ https://issues.apache.org/jira/browse/HADOOP-10716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HADOOP-10716: Status: Patch Available (was: Open) Thanks Daryn for helping me on the fix. Tested on cluster. > Cannot use more than 1 har filesystem > - > > Key: HADOOP-10716 > URL: https://issues.apache.org/jira/browse/HADOOP-10716 > Project: Hadoop Common > Issue Type: Bug > Components: conf, fs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Priority: Critical > Attachments: HADOOP-10716.patch > > > Filesystems are cached purely on scheme + authority. Har filesystems > actually need further differentiation based on path to the har file itself. > For this reason, the fs cache used to be explicitly disable for har via > "fs.har.impl.cache.disable" in core-default.xml. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10530) Make hadoop trunk build on Java7+ only
[
https://issues.apache.org/jira/browse/HADOOP-10530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-10530:
Affects Version/s: 2.4.0
Status: Patch Available (was: Open)
> Make hadoop trunk build on Java7+ only
> --
>
> Key: HADOOP-10530
> URL: https://issues.apache.org/jira/browse/HADOOP-10530
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 2.4.0, 3.0.0
> Environment: Java 1.7+
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-10530-001.patch, HADOOP-10530-002.patch
>
>
> As discussed on hadoop-common, hadoop 3 is envisaged to be Java7+ *only*
> -this JIRA covers switching the build for this
> # maven enforcer plugin to set Java version = {{[1.7)}}
> # compiler to set language to java 1.7
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10530) Make hadoop trunk build on Java7+ only
[
https://issues.apache.org/jira/browse/HADOOP-10530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034446#comment-14034446
]
Haohui Mai commented on HADOOP-10530:
-
The patch looks good to me. +1.
Built hdfs on both Mac OS X and CentOS with Java 7. Work as expected.
Built with Java 6 on CentOS. Failed as expected.
Ran a few tests with hdfs and it looked pretty good to me.
This should be a low risk change since (1) it is going into trunk only, and (2)
Hadoop on Java 7 has been proved in production (based on my understanding
several company has been supporting Hadoop on Java 7 for a while).
This only thing left is that we need to make sure that Jenkins moves to Java 7
before this patch goes into the trunk.
> Make hadoop trunk build on Java7+ only
> --
>
> Key: HADOOP-10530
> URL: https://issues.apache.org/jira/browse/HADOOP-10530
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 3.0.0
> Environment: Java 1.7+
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-10530-001.patch, HADOOP-10530-002.patch
>
>
> As discussed on hadoop-common, hadoop 3 is envisaged to be Java7+ *only*
> -this JIRA covers switching the build for this
> # maven enforcer plugin to set Java version = {{[1.7)}}
> # compiler to set language to java 1.7
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10530) Make hadoop trunk build on Java7+ only
[
https://issues.apache.org/jira/browse/HADOOP-10530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-10530:
Attachment: HADOOP-10530-002.patch
Patch rebased against trunk; updates BUILDING.txt
> Make hadoop trunk build on Java7+ only
> --
>
> Key: HADOOP-10530
> URL: https://issues.apache.org/jira/browse/HADOOP-10530
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 3.0.0
> Environment: Java 1.7+
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-10530-001.patch, HADOOP-10530-002.patch
>
>
> As discussed on hadoop-common, hadoop 3 is envisaged to be Java7+ *only*
> -this JIRA covers switching the build for this
> # maven enforcer plugin to set Java version = {{[1.7)}}
> # compiler to set language to java 1.7
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10530) Make hadoop trunk build on Java7+ only
[
https://issues.apache.org/jira/browse/HADOOP-10530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-10530:
Status: Open (was: Patch Available)
> Make hadoop trunk build on Java7+ only
> --
>
> Key: HADOOP-10530
> URL: https://issues.apache.org/jira/browse/HADOOP-10530
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 3.0.0
> Environment: Java 1.7+
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-10530-001.patch, HADOOP-10530-002.patch
>
>
> As discussed on hadoop-common, hadoop 3 is envisaged to be Java7+ *only*
> -this JIRA covers switching the build for this
> # maven enforcer plugin to set Java version = {{[1.7)}}
> # compiler to set language to java 1.7
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10076) Update tomcat/jasper dependency to version 7
[ https://issues.apache.org/jira/browse/HADOOP-10076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034415#comment-14034415 ] Haohui Mai commented on HADOOP-10076: - This patch might be slightly out of dated since jspc is no longer required in trunk. > Update tomcat/jasper dependency to version 7 > > > Key: HADOOP-10076 > URL: https://issues.apache.org/jira/browse/HADOOP-10076 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.2.0 >Reporter: Robert Rati > Attachments: HADOOP-10076.patch > > > Tomcat 6 is no longer maintained. Update to use tomcat/jasper 7. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10716) Cannot use more than 1 har filesystem
Daryn Sharp created HADOOP-10716: Summary: Cannot use more than 1 har filesystem Key: HADOOP-10716 URL: https://issues.apache.org/jira/browse/HADOOP-10716 Project: Hadoop Common Issue Type: Bug Components: conf, fs Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Priority: Critical Filesystems are cached purely on scheme + authority. Har filesystems actually need further differentiation based on path to the har file itself. For this reason, the fs cache used to be explicitly disable for har via "fs.har.impl.cache.disable" in core-default.xml. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10716) Cannot use more than 1 har filesystem
[ https://issues.apache.org/jira/browse/HADOOP-10716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034410#comment-14034410 ] Daryn Sharp commented on HADOOP-10716: -- The config key appears to have been erroneously removed when the fs service loader was added. > Cannot use more than 1 har filesystem > - > > Key: HADOOP-10716 > URL: https://issues.apache.org/jira/browse/HADOOP-10716 > Project: Hadoop Common > Issue Type: Bug > Components: conf, fs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Priority: Critical > > Filesystems are cached purely on scheme + authority. Har filesystems > actually need further differentiation based on path to the har file itself. > For this reason, the fs cache used to be explicitly disable for har via > "fs.har.impl.cache.disable" in core-default.xml. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10703) Duplicate filter instances are created during HttpServer2 initialization
[ https://issues.apache.org/jira/browse/HADOOP-10703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034358#comment-14034358 ] Benoy Antony commented on HADOOP-10703: --- I quickly scanned the servlet spec, but could not find the related text. [~tucu00], could you please point me to the text ? What's the benefit in creating duplicate filter instances in this use case ? > Duplicate filter instances are created during HttpServer2 initialization > > > Key: HADOOP-10703 > URL: https://issues.apache.org/jira/browse/HADOOP-10703 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.4.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10703.patch, > multiple-authenticationfilter-inits.log > > > The HttpServer2.defineFilter creates a Filter instance for each context. By > default, there are 3 contexts. > So there will be 3 separate AuthenticationFilter instances and corresponding > AuthenticationHandler instances. This also results in 3 separate > initializations of AuthenticationHandler. > The log file illustrating this repeated initialization is attached. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10681) Remove synchronized blocks from SnappyCodec and ZlibCodec buffering
[
https://issues.apache.org/jira/browse/HADOOP-10681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034342#comment-14034342
]
Hadoop QA commented on HADOOP-10681:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650430/HADOOP-10681.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4089//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4089//console
This message is automatically generated.
> Remove synchronized blocks from SnappyCodec and ZlibCodec buffering
> ---
>
> Key: HADOOP-10681
> URL: https://issues.apache.org/jira/browse/HADOOP-10681
> Project: Hadoop Common
> Issue Type: Bug
> Components: performance
>Affects Versions: 2.2.0, 2.4.0, 2.5.0
>Reporter: Gopal V
>Assignee: Gopal V
> Labels: perfomance
> Attachments: HADOOP-10681.1.patch, compress-cmpxchg-small.png,
> perf-top-spill-merge.png, snappy-perf-unsync.png
>
>
> The current implementation of SnappyCompressor spends more time within the
> java loop of copying from the user buffer into the direct buffer allocated to
> the compressor impl, than the time it takes to compress the buffers.
> !perf-top-spill-merge.png!
> The bottleneck was found to be java monitor code inside SnappyCompressor.
> The methods are neatly inlined by the JIT into the parent caller
> (BlockCompressorStream::write), which unfortunately does not flatten out the
> synchronized blocks.
> !compress-cmpxchg-small.png!
> The loop does a write of small byte[] buffers (each IFile key+value).
> I counted approximately 6 monitor enter/exit blocks per k-v pair written.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-9705) FsShell cp -p does not preserve directory attibutes
[
https://issues.apache.org/jira/browse/HADOOP-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-9705:
--
Attachment: HADOOP-9705.5.patch
Rebased the patch.
> FsShell cp -p does not preserve directory attibutes
> ---
>
> Key: HADOOP-9705
> URL: https://issues.apache.org/jira/browse/HADOOP-9705
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Stephen Chu
>Assignee: Akira AJISAKA
> Attachments: HADOOP-9705.2.patch, HADOOP-9705.3.patch,
> HADOOP-9705.4.patch, HADOOP-9705.5.patch, HADOOP-9705.patch
>
>
> HADOOP-9338 added the -p flag to preserve file attributes when copying.
> However, cp -p does not preserve directory attributes. It'd be useful to add
> this functionality.
> For example, the following shows that the modified time is not preserved
> {code}
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -mkdir
> /user/schu/testDir1
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu/
> Found 1 items
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -cp -p
> /user/schu/testDir1 /user/schu/testDir2
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu
> Found 2 items
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:35 /user/schu/testDir2
> [schu@hdfs-snapshots-1 ~]$
> {code}
> The preserve logic is in CommandWithDestination#copyFileToTarget, which is
> only called with files.
> {code}
> protected void processPath(PathData src, PathData dst) throws IOException {
> if (src.stat.isSymlink()) {
> // TODO: remove when FileContext is supported, this needs to either
>
>
> // copy the symlink or deref the symlink
>
>
> throw new PathOperationException(src.toString());
> } else if (src.stat.isFile()) {
> copyFileToTarget(src, dst);
> } else if (src.stat.isDirectory() && !isRecursive()) {
> throw new PathIsDirectoryException(src.toString());
> }
> }
> {code}
> {code}
> /**
>
>
>* Copies the source file to the target.
>
>
>* @param src item to copy
>
>
>* @param target where to copy the item
>
>
>* @throws IOException if copy fails
>
>
>*/
> protected void copyFileToTarget(PathData src, PathData target) throws
> IOException {
> src.fs.setVerifyChecksum(verifyChecksum);
> if (src != null) {
> throw new PathExistsException("hi");
> }
> InputStream in = null;
> try {
> in = src.fs.open(src.path);
> copyStreamToTarget(in, target);
> if(preserve) {
> target.fs.setTimes(
> target.path,
> src.stat.getModificationTime(),
> src.stat.getAccessTime());
> target.fs.setOwner(
> target.path,
> src.stat.getOwner(),
> src.stat.getGroup());
> target.fs.setPermission(
> target.path,
> src.stat.getPermission());
> System.out.println("Preserving");
> if (src.fs.equals(target.fs)) {
> System.out.println("Same filesystems");
> src.fs.preserveAttributes(src.path, target.path);
> }
> throw new IOException("hi");
> }
> } finally {
> IOUtils.closeStream(in);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10307) Support multiple Authentication mechanisms for HTTP
[
https://issues.apache.org/jira/browse/HADOOP-10307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034288#comment-14034288
]
Hadoop QA commented on HADOOP-10307:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650862/HADOOP-10307.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-auth.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4088//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4088//console
This message is automatically generated.
> Support multiple Authentication mechanisms for HTTP
> ---
>
> Key: HADOOP-10307
> URL: https://issues.apache.org/jira/browse/HADOOP-10307
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.2.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HADOOP-10307.patch, HADOOP-10307.patch,
> HADOOP-10307.patch
>
>
> Currently it is possible to specify a custom Authentication Handler for HTTP
> authentication.
> We have a requirement to support multiple mechanisms to authenticate HTTP
> access.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10703) Duplicate filter instances are created during HttpServer2 initialization
[ https://issues.apache.org/jira/browse/HADOOP-10703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034277#comment-14034277 ] Alejandro Abdelnur commented on HADOOP-10703: - By servlet spec it is expected that servlets/filters instances are not shared among webapps. If we want to respect that semantics the current behavior is correct, each context (webapp) should have its own instance. > Duplicate filter instances are created during HttpServer2 initialization > > > Key: HADOOP-10703 > URL: https://issues.apache.org/jira/browse/HADOOP-10703 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.4.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10703.patch, > multiple-authenticationfilter-inits.log > > > The HttpServer2.defineFilter creates a Filter instance for each context. By > default, there are 3 contexts. > So there will be 3 separate AuthenticationFilter instances and corresponding > AuthenticationHandler instances. This also results in 3 separate > initializations of AuthenticationHandler. > The log file illustrating this repeated initialization is attached. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10681) Remove synchronized blocks from SnappyCodec and ZlibCodec buffering
[ https://issues.apache.org/jira/browse/HADOOP-10681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HADOOP-10681: - Release Note: Remove unnecessary synchronized blocks from Snappy/Zlib codecs. Status: Patch Available (was: In Progress) > Remove synchronized blocks from SnappyCodec and ZlibCodec buffering > --- > > Key: HADOOP-10681 > URL: https://issues.apache.org/jira/browse/HADOOP-10681 > Project: Hadoop Common > Issue Type: Bug > Components: performance >Affects Versions: 2.4.0, 2.2.0, 2.5.0 >Reporter: Gopal V >Assignee: Gopal V > Labels: perfomance > Attachments: HADOOP-10681.1.patch, compress-cmpxchg-small.png, > perf-top-spill-merge.png, snappy-perf-unsync.png > > > The current implementation of SnappyCompressor spends more time within the > java loop of copying from the user buffer into the direct buffer allocated to > the compressor impl, than the time it takes to compress the buffers. > !perf-top-spill-merge.png! > The bottleneck was found to be java monitor code inside SnappyCompressor. > The methods are neatly inlined by the JIT into the parent caller > (BlockCompressorStream::write), which unfortunately does not flatten out the > synchronized blocks. > !compress-cmpxchg-small.png! > The loop does a write of small byte[] buffers (each IFile key+value). > I counted approximately 6 monitor enter/exit blocks per k-v pair written. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034281#comment-14034281
]
Sumit Kumar commented on HADOOP-9565:
-
I see some interface like this in hadoop-azure codebase
https://github.com/apache/hadoop-common/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterface.java
> Add a Blobstore interface to add to blobstore FileSystems
> -
>
> Key: HADOOP-9565
> URL: https://issues.apache.org/jira/browse/HADOOP-9565
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 2.0.4-alpha
>Reporter: Steve Loughran
>Priority: Minor
>
> We can make the fact that some {{FileSystem}} implementations are really
> blobstores, with different atomicity and consistency guarantees, by adding a
> {{Blobstore}} interface to add to them.
> This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
> all blobstores implement at server-side copy operation as a substitute for
> rename.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10703) Duplicate filter instances are created during HttpServer2 initialization
[ https://issues.apache.org/jira/browse/HADOOP-10703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034255#comment-14034255 ] Daryn Sharp commented on HADOOP-10703: -- I think it looks ok, but someone with more jetty knowledge should confirm there's no unexpected side-effects. > Duplicate filter instances are created during HttpServer2 initialization > > > Key: HADOOP-10703 > URL: https://issues.apache.org/jira/browse/HADOOP-10703 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.4.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10703.patch, > multiple-authenticationfilter-inits.log > > > The HttpServer2.defineFilter creates a Filter instance for each context. By > default, there are 3 contexts. > So there will be 3 separate AuthenticationFilter instances and corresponding > AuthenticationHandler instances. This also results in 3 separate > initializations of AuthenticationHandler. > The log file illustrating this repeated initialization is attached. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10307) Support multiple Authentication mechanisms for HTTP
[ https://issues.apache.org/jira/browse/HADOOP-10307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HADOOP-10307: -- Attachment: HADOOP-10307.patch Attaching the patch which sets the authentication type as "composite" in the tokens generated via _CompositeAuthenticationHandler_. > Support multiple Authentication mechanisms for HTTP > --- > > Key: HADOOP-10307 > URL: https://issues.apache.org/jira/browse/HADOOP-10307 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.2.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10307.patch, HADOOP-10307.patch, > HADOOP-10307.patch > > > Currently it is possible to specify a custom Authentication Handler for HTTP > authentication. > We have a requirement to support multiple mechanisms to authenticate HTTP > access. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10400) Incorporate new S3A FileSystem implementation
[
https://issues.apache.org/jira/browse/HADOOP-10400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034219#comment-14034219
]
Steve Loughran commented on HADOOP-10400:
-
I've just added a patch for HADOOP-10373 which creates a new module for
hadoop-amazon support
# all the new filesystem code must go in there, to keep it and its dependencies
isolated
# you can just incorporate that patch and move your code over, to have one
unified patch (i.e. no need to depend on 10373 being checked in)
# core-site.xml can still retain the settings for the new FS
# we'll need documentation in hadoop-amazon/src/main/site/markdown (or
site/apt if you really prefer)
# the tests are designed to only work if the file test/resources/auth-keys.xml
is present. We can mark that as svnignore, gitignore and have the tests load it
in. You can then use Test* as the pattern for tests, and be confident that if
the -amazon tests run, they are really running. Look at the azure and openstack
examples here.
Having had a quick look at the code
# please mark as {{final}} fields that are fixed in the constructor
# That test {{FileSystemContractBaseTest.testMkdirsWithUmask()}} ? just skip it
in your subclass; no need to patch the root test
> Incorporate new S3A FileSystem implementation
> -
>
> Key: HADOOP-10400
> URL: https://issues.apache.org/jira/browse/HADOOP-10400
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/s3
>Affects Versions: 2.4.0
>Reporter: Jordan Mendelson
>Assignee: Jordan Mendelson
> Attachments: HADOOP-10400-1.patch, HADOOP-10400-2.patch,
> HADOOP-10400-3.patch, HADOOP-10400-4.patch, HADOOP-10400-5.patch
>
>
> The s3native filesystem has a number of limitations (some of which were
> recently fixed by HADOOP-9454). This patch adds an s3a filesystem which uses
> the aws-sdk instead of the jets3t library. There are a number of improvements
> over s3native including:
> - Parallel copy (rename) support (dramatically speeds up commits on large
> files)
> - AWS S3 explorer compatible empty directories files "xyz/" instead of
> "xyz_$folder$" (reduces littering)
> - Ignores s3native created _$folder$ files created by s3native and other S3
> browsing utilities
> - Supports multiple output buffer dirs to even out IO when uploading files
> - Supports IAM role-based authentication
> - Allows setting a default canned ACL for uploads (public, private, etc.)
> - Better error recovery handling
> - Should handle input seeks without having to download the whole file (used
> for splits a lot)
> This code is a copy of https://github.com/Aloisius/hadoop-s3a with patches to
> various pom files to get it to build against trunk. I've been using 0.0.1 in
> production with CDH 4 for several months and CDH 5 for a few days. The
> version here is 0.0.2 which changes around some keys to hopefully bring the
> key name style more inline with the rest of hadoop 2.x.
> *Tunable parameters:*
> fs.s3a.access.key - Your AWS access key ID (omit for role authentication)
> fs.s3a.secret.key - Your AWS secret key (omit for role authentication)
> fs.s3a.connection.maximum - Controls how many parallel connections
> HttpClient spawns (default: 15)
> fs.s3a.connection.ssl.enabled - Enables or disables SSL connections to S3
> (default: true)
> fs.s3a.attempts.maximum - How many times we should retry commands on
> transient errors (default: 10)
> fs.s3a.connection.timeout - Socket connect timeout (default: 5000)
> fs.s3a.paging.maximum - How many keys to request from S3 when doing
> directory listings at a time (default: 5000)
> fs.s3a.multipart.size - How big (in bytes) to split a upload or copy
> operation up into (default: 104857600)
> fs.s3a.multipart.threshold - Until a file is this large (in bytes), use
> non-parallel upload (default: 2147483647)
> fs.s3a.acl.default - Set a canned ACL on newly created/copied objects
> (private | public-read | public-read-write | authenticated-read |
> log-delivery-write | bucket-owner-read | bucket-owner-full-control)
> fs.s3a.multipart.purge - True if you want to purge existing multipart
> uploads that may not have been completed/aborted correctly (default: false)
> fs.s3a.multipart.purge.age - Minimum age in seconds of multipart uploads
> to purge (default: 86400)
> fs.s3a.buffer.dir - Comma separated list of directories that will be used
> to buffer file writes out of (default: uses ${hadoop.tmp.dir}/s3a )
> *Caveats*:
> Hadoop uses a standard output committer which uploads files as
> filename.COPYING before renaming them. This can cause unnecessary performance
> issues with S3 because it does not have a rename operation and S3 already
> verifies uploads against an md5 that the driver
[jira] [Commented] (HADOOP-10590) ServiceAuthorizationManager is not threadsafe
[ https://issues.apache.org/jira/browse/HADOOP-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034217#comment-14034217 ] Benoy Antony commented on HADOOP-10590: --- Thanks [~daryn]. [~vinayrpet] , Could you please review and commit as well ? > ServiceAuthorizationManager is not threadsafe > -- > > Key: HADOOP-10590 > URL: https://issues.apache.org/jira/browse/HADOOP-10590 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.4.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10590.patch, performance-test-without-rpc.patch, > performancetest.patch > > > The mutators in ServiceAuthorizationManager are synchronized. The accessors > are not synchronized. > This results in visibility issues when ServiceAuthorizationManager's state > is accessed from different threads. -- This message was sent by Atlassian JIRA (v6.2#6252)
common-issues@hadoop.apache.org
[
https://issues.apache.org/jira/browse/HADOOP-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034212#comment-14034212
]
Colin Patrick McCabe commented on HADOOP-10636:
---
In version 7, need to fix the same problem in this function:
{code}
+void hrpc_call_deliver_err_test_cb(struct hrpc_response *resp,
+struct hadoop_err *err, void *cb_data)
+{
+uv_sem_t *sem = cb_data;
+
EXPECT_STR_EQ("org.apache.hadoop.native.HadoopCore.InvalidRequestException: "
+"test hadoop_lerr_alloc", hadoop_err_msg(err));
+EXPECT_NULL(resp);
{code}
+1 once that's addressed
> Native Hadoop Client:add unit test case for call&client_id
> --
>
> Key: HADOOP-10636
> URL: https://issues.apache.org/jira/browse/HADOOP-10636
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: HADOOP-10388
>Reporter: Wenwu Peng
>Assignee: Wenwu Peng
> Attachments: HADOOP-10636-pnative.001.patch,
> HADOOP-10636-pnative.002.patch, HADOOP-10636-pnative.003.patch,
> HADOOP-10636-pnative.004.patch, HADOOP-10636-pnative.005.patch,
> HADOOP-10636-pnative.006.patch, HADOOP-10636-pnative.007.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10373) move s3 logic to own replaceable jar, hadoop-aws
[
https://issues.apache.org/jira/browse/HADOOP-10373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-10373:
Attachment: HADOOP-10373-001.patch
create stub project {{hadoop-tools/hadoop-amazon}}
# declares a dependency on the amazon library
# is a peer of hadoop-openstack and hadoop-azure
There is no code here, that is for HADOOP-10400 - which can go on top of this
> move s3 logic to own replaceable jar, hadoop-aws
>
>
> Key: HADOOP-10373
> URL: https://issues.apache.org/jira/browse/HADOOP-10373
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Affects Versions: 2.3.0
>Reporter: Steve Loughran
> Fix For: 2.5.0
>
> Attachments: HADOOP-10373-001.patch
>
>
> After HADOOP-9565 adds a marker interface for blobstores, move s3 & s3n into
> their own hadoop-amazon library
> # keeps the S3 dependencies out of the standard hadoop client dependency
> graph.
> # lets people switch this for alternative implementations.
> feature #2 would let you swap over to another s3n impl (e.g. amazon's)
> without rebuilding everything
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10701) NFS should not validate the access premission only based on the user's primary group
[ https://issues.apache.org/jira/browse/HADOOP-10701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034176#comment-14034176 ] Premchandra Preetham Kukillaya commented on HADOOP-10701: - If i am given an option, Yes i would vote for other option (List of GID received from LDAP/OS) for just making the deployment at Cloud very easy. Also i dont want limitation of 15 GID. > NFS should not validate the access premission only based on the user's > primary group > > > Key: HADOOP-10701 > URL: https://issues.apache.org/jira/browse/HADOOP-10701 > Project: Hadoop Common > Issue Type: Bug > Components: nfs >Affects Versions: 2.3.0 >Reporter: Premchandra Preetham Kukillaya >Assignee: Harsh J > Attachments: HADOOP-10701.patch, HADOOP-10701.patch, > Hadoop_NFS_BugCreation_Steps.docx > > > The bug is while accessing NFS Mounted File System the permission is always > validated based on the primary Unix group the user is associated with and > Secondary Unix groups are ignored. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10701) NFS should not validate the access premission only based on the user's primary group
[ https://issues.apache.org/jira/browse/HADOOP-10701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HADOOP-10701: Summary: NFS should not validate the access premission only based on the user's primary group (was: NFS should not only validate the access premission based on the user's primary group) > NFS should not validate the access premission only based on the user's > primary group > > > Key: HADOOP-10701 > URL: https://issues.apache.org/jira/browse/HADOOP-10701 > Project: Hadoop Common > Issue Type: Bug > Components: nfs >Affects Versions: 2.3.0 >Reporter: Premchandra Preetham Kukillaya >Assignee: Harsh J > Attachments: HADOOP-10701.patch, HADOOP-10701.patch, > Hadoop_NFS_BugCreation_Steps.docx > > > The bug is while accessing NFS Mounted File System the permission is always > validated based on the primary Unix group the user is associated with and > Secondary Unix groups are ignored. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10701) NFS should not only validate the access premission based on the user's primary group
[ https://issues.apache.org/jira/browse/HADOOP-10701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HADOOP-10701: Summary: NFS should not only validate the access premission based on the user's primary group (was: NFS should not only validate the access premission based on the primary group the user is associated with) > NFS should not only validate the access premission based on the user's > primary group > > > Key: HADOOP-10701 > URL: https://issues.apache.org/jira/browse/HADOOP-10701 > Project: Hadoop Common > Issue Type: Bug > Components: nfs >Affects Versions: 2.3.0 >Reporter: Premchandra Preetham Kukillaya >Assignee: Harsh J > Attachments: HADOOP-10701.patch, HADOOP-10701.patch, > Hadoop_NFS_BugCreation_Steps.docx > > > The bug is while accessing NFS Mounted File System the permission is always > validated based on the primary Unix group the user is associated with and > Secondary Unix groups are ignored. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10701) NFS should not only validate the access premission based on the primary group the user is associated with
[ https://issues.apache.org/jira/browse/HADOOP-10701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HADOOP-10701: Summary: NFS should not only validate the access premission based on the primary group the user is associated with (was: The bug is while accessing NFS Mounted File System the permission is always validated based on the primary Unix group the user is associated with and Secondary Unix groups are ignored.) > NFS should not only validate the access premission based on the primary group > the user is associated with > - > > Key: HADOOP-10701 > URL: https://issues.apache.org/jira/browse/HADOOP-10701 > Project: Hadoop Common > Issue Type: Bug > Components: nfs >Affects Versions: 2.3.0 >Reporter: Premchandra Preetham Kukillaya >Assignee: Harsh J > Attachments: HADOOP-10701.patch, HADOOP-10701.patch, > Hadoop_NFS_BugCreation_Steps.docx > > > The bug is while accessing NFS Mounted File System the permission is always > validated based on the primary Unix group the user is associated with and > Secondary Unix groups are ignored. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10701) The bug is while accessing NFS Mounted File System the permission is always validated based on the primary Unix group the user is associated with and Secondary Unix
[ https://issues.apache.org/jira/browse/HADOOP-10701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034162#comment-14034162 ] Harsh J commented on HADOOP-10701: -- Thanks for giving it a look - I just reused what was already being stored by the AUTH_SYS. The other way could be done as well. > The bug is while accessing NFS Mounted File System the permission is always > validated based on the primary Unix group the user is associated with and > Secondary Unix groups are ignored. > - > > Key: HADOOP-10701 > URL: https://issues.apache.org/jira/browse/HADOOP-10701 > Project: Hadoop Common > Issue Type: Bug > Components: nfs >Affects Versions: 2.3.0 >Reporter: Premchandra Preetham Kukillaya >Assignee: Harsh J > Attachments: HADOOP-10701.patch, HADOOP-10701.patch, > Hadoop_NFS_BugCreation_Steps.docx > > > The bug is while accessing NFS Mounted File System the permission is always > validated based on the primary Unix group the user is associated with and > Secondary Unix groups are ignored. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10660) GraphiteSink should implement Closeable
[
https://issues.apache.org/jira/browse/HADOOP-10660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034143#comment-14034143
]
Hadoop QA commented on HADOOP-10660:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650835/HADOOP-10660-v6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4087//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4087//console
This message is automatically generated.
> GraphiteSink should implement Closeable
> ---
>
> Key: HADOOP-10660
> URL: https://issues.apache.org/jira/browse/HADOOP-10660
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Chen He
> Attachments: HADOOP-10660-v2.patch, HADOOP-10660-v3.patch,
> HADOOP-10660-v6.patch, HADOOP-10660.patch, hadoop-10660-v4.txt,
> hadoop-10660-v5.txt
>
>
> GraphiteSink wraps OutputStreamWriter around socket's output stream.
> Currently the socket is never closed.
> GraphiteSink should implement Closeable such that MetricsSystem can close the
> socket when it is stopped.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10701) The bug is while accessing NFS Mounted File System the permission is always validated based on the primary Unix group the user is associated with and Secondary Unix gr
[ https://issues.apache.org/jira/browse/HADOOP-10701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Premchandra Preetham Kukillaya updated HADOOP-10701: Description: The bug is while accessing NFS Mounted File System the permission is always validated based on the primary Unix group the user is associated with and Secondary Unix groups are ignored. > The bug is while accessing NFS Mounted File System the permission is always > validated based on the primary Unix group the user is associated with and > Secondary Unix groups are ignored. > - > > Key: HADOOP-10701 > URL: https://issues.apache.org/jira/browse/HADOOP-10701 > Project: Hadoop Common > Issue Type: Bug > Components: nfs >Affects Versions: 2.3.0 >Reporter: Premchandra Preetham Kukillaya >Assignee: Harsh J > Attachments: HADOOP-10701.patch, HADOOP-10701.patch, > Hadoop_NFS_BugCreation_Steps.docx > > > The bug is while accessing NFS Mounted File System the permission is always > validated based on the primary Unix group the user is associated with and > Secondary Unix groups are ignored. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9991) Fix up Hadoop Poms for enforced dependencies, roll up JARs to latest versions
[ https://issues.apache.org/jira/browse/HADOOP-9991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034122#comment-14034122 ] Steve Loughran commented on HADOOP-9991: HADOOP-9555 updates ZK to 3.4.6 > Fix up Hadoop Poms for enforced dependencies, roll up JARs to latest versions > - > > Key: HADOOP-9991 > URL: https://issues.apache.org/jira/browse/HADOOP-9991 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.1.1-beta, 2.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: hadoop-9991-v1.txt > > > If you try using Hadoop downstream with a classpath shared with HBase and > Accumulo, you soon discover how messy the dependencies are. > Hadoop's side of this problem is > # not being up to date with some of the external releases of common JARs > # not locking down/excluding inconsistent versions of artifacts provided down > the dependency graph -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10701) The bug is while accessing NFS Mounted File System the permission is always validated based on the primary Unix group the user is associated with and Secondary Unix
[ https://issues.apache.org/jira/browse/HADOOP-10701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034116#comment-14034116 ] Premchandra Preetham Kukillaya commented on HADOOP-10701: - @Harsh, Thanks for comming up with Fix . But here are my questions 1) As per the fix you are using the List of GID's sent by NFS Client to NFS Proxy. But in this approach are disadvantages like 1) max of 16 GID/ UNIX ID 2) Requires integration of Client machine with LDAP and its a big pain. 2) Why the List of GID for each UID is not retrieved from OS/LDAP instead of list of List of GID's sent by NFS Client to NFS Proxy. Doing so it will eliminate the need for Client machines to integrate with LDAP. > The bug is while accessing NFS Mounted File System the permission is always > validated based on the primary Unix group the user is associated with and > Secondary Unix groups are ignored. > - > > Key: HADOOP-10701 > URL: https://issues.apache.org/jira/browse/HADOOP-10701 > Project: Hadoop Common > Issue Type: Bug > Components: nfs >Affects Versions: 2.3.0 >Reporter: Premchandra Preetham Kukillaya >Assignee: Harsh J > Attachments: HADOOP-10701.patch, HADOOP-10701.patch, > Hadoop_NFS_BugCreation_Steps.docx > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10660) GraphiteSink should implement Closeable
[
https://issues.apache.org/jira/browse/HADOOP-10660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034087#comment-14034087
]
Hadoop QA commented on HADOOP-10660:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650831/hadoop-10660-v5.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4086//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4086//console
This message is automatically generated.
> GraphiteSink should implement Closeable
> ---
>
> Key: HADOOP-10660
> URL: https://issues.apache.org/jira/browse/HADOOP-10660
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Chen He
> Attachments: HADOOP-10660-v2.patch, HADOOP-10660-v3.patch,
> HADOOP-10660-v6.patch, HADOOP-10660.patch, hadoop-10660-v4.txt,
> hadoop-10660-v5.txt
>
>
> GraphiteSink wraps OutputStreamWriter around socket's output stream.
> Currently the socket is never closed.
> GraphiteSink should implement Closeable such that MetricsSystem can close the
> socket when it is stopped.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10714) AmazonS3Client.deleteObjects() need to be limited to 1000 entries per call
[ https://issues.apache.org/jira/browse/HADOOP-10714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034083#comment-14034083 ] Steve Loughran commented on HADOOP-10714: - * can this just be merged into a new HADOOP-10400 patch ... as that isn't checked in yet it should just be updated * scale tests are good -for the swift stuff we have some scalable ones which you can tune off the test config file. This lets you run smaller tests over slower links. File size can be kept low for better performance. Tests for a large set of files should # verify that the results of a directory listing is complete # try a rename() (as this has a delete inside) # do the delete() > AmazonS3Client.deleteObjects() need to be limited to 1000 entries per call > -- > > Key: HADOOP-10714 > URL: https://issues.apache.org/jira/browse/HADOOP-10714 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.5.0 >Reporter: David S. Wang >Assignee: David S. Wang >Priority: Critical > Labels: s3 > Attachments: HADOOP-10714-1.patch > > > In the patch for HADOOP-10400, calls to AmazonS3Client.deleteObjects() need > to have the number of entries at 1000 or below. Otherwise we get a Malformed > XML error similar to: > com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS > Service: Amazon S3, AWS Request ID: 6626AD56A3C76F5B, AWS Error Code: > MalformedXML, AWS Error Message: The XML you provided was not well-formed or > did not validate against our published schema, S3 Extended Request ID: > DOt6C+Y84mGSoDuaQTCo33893VaoKGEVC3y1k2zFIQRm+AJkFH2mTyrDgnykSL+v > at > com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798) > at > com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421) > at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) > at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) > at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3480) > at > com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.java:1739) > at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:388) > at > org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:829) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:874) > at > org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:878) > Note that this is mentioned in the AWS documentation: > http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html > "The Multi-Object Delete request contains a list of up to 1000 keys that you > want to delete. In the XML, you provide the object key names, and optionally, > version IDs if you want to delete a specific version of the object from a > versioning-enabled bucket. For each key, Amazon S3….” > Thanks to Matteo Bertozzi and Rahul Bhartia from AWS for identifying the > problem. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10714) AmazonS3Client.deleteObjects() need to be limited to 1000 entries per call
[ https://issues.apache.org/jira/browse/HADOOP-10714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-10714: Target Version/s: 2.5.0 (was: 2.4.1) Affects Version/s: (was: 2.4.0) 2.5.0 > AmazonS3Client.deleteObjects() need to be limited to 1000 entries per call > -- > > Key: HADOOP-10714 > URL: https://issues.apache.org/jira/browse/HADOOP-10714 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.5.0 >Reporter: David S. Wang >Assignee: David S. Wang >Priority: Critical > Labels: s3 > Attachments: HADOOP-10714-1.patch > > > In the patch for HADOOP-10400, calls to AmazonS3Client.deleteObjects() need > to have the number of entries at 1000 or below. Otherwise we get a Malformed > XML error similar to: > com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS > Service: Amazon S3, AWS Request ID: 6626AD56A3C76F5B, AWS Error Code: > MalformedXML, AWS Error Message: The XML you provided was not well-formed or > did not validate against our published schema, S3 Extended Request ID: > DOt6C+Y84mGSoDuaQTCo33893VaoKGEVC3y1k2zFIQRm+AJkFH2mTyrDgnykSL+v > at > com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798) > at > com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421) > at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) > at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) > at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3480) > at > com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.java:1739) > at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:388) > at > org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:829) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:874) > at > org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:878) > Note that this is mentioned in the AWS documentation: > http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html > "The Multi-Object Delete request contains a list of up to 1000 keys that you > want to delete. In the XML, you provide the object key names, and optionally, > version IDs if you want to delete a specific version of the object from a > versioning-enabled bucket. For each key, Amazon S3….” > Thanks to Matteo Bertozzi and Rahul Bhartia from AWS for identifying the > problem. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9705) FsShell cp -p does not preserve directory attibutes
[
https://issues.apache.org/jira/browse/HADOOP-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034056#comment-14034056
]
Chris Nauroth commented on HADOOP-9705:
---
Hi, [~ajisakaa]. I came across this issue while reviewing HADOOP-10557 for the
new cp -pa option that preserves ACLs. Do you want to rebase this patch and
make sure it works with ACLs too? I can help with code review and committing
it. Thanks!
> FsShell cp -p does not preserve directory attibutes
> ---
>
> Key: HADOOP-9705
> URL: https://issues.apache.org/jira/browse/HADOOP-9705
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Stephen Chu
>Assignee: Akira AJISAKA
> Attachments: HADOOP-9705.2.patch, HADOOP-9705.3.patch,
> HADOOP-9705.4.patch, HADOOP-9705.patch
>
>
> HADOOP-9338 added the -p flag to preserve file attributes when copying.
> However, cp -p does not preserve directory attributes. It'd be useful to add
> this functionality.
> For example, the following shows that the modified time is not preserved
> {code}
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -mkdir
> /user/schu/testDir1
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu/
> Found 1 items
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -cp -p
> /user/schu/testDir1 /user/schu/testDir2
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu
> Found 2 items
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:35 /user/schu/testDir2
> [schu@hdfs-snapshots-1 ~]$
> {code}
> The preserve logic is in CommandWithDestination#copyFileToTarget, which is
> only called with files.
> {code}
> protected void processPath(PathData src, PathData dst) throws IOException {
> if (src.stat.isSymlink()) {
> // TODO: remove when FileContext is supported, this needs to either
>
>
> // copy the symlink or deref the symlink
>
>
> throw new PathOperationException(src.toString());
> } else if (src.stat.isFile()) {
> copyFileToTarget(src, dst);
> } else if (src.stat.isDirectory() && !isRecursive()) {
> throw new PathIsDirectoryException(src.toString());
> }
> }
> {code}
> {code}
> /**
>
>
>* Copies the source file to the target.
>
>
>* @param src item to copy
>
>
>* @param target where to copy the item
>
>
>* @throws IOException if copy fails
>
>
>*/
> protected void copyFileToTarget(PathData src, PathData target) throws
> IOException {
> src.fs.setVerifyChecksum(verifyChecksum);
> if (src != null) {
> throw new PathExistsException("hi");
> }
> InputStream in = null;
> try {
> in = src.fs.open(src.path);
> copyStreamToTarget(in, target);
> if(preserve) {
> target.fs.setTimes(
> target.path,
> src.stat.getModificationTime(),
> src.stat.getAccessTime());
> target.fs.setOwner(
> target.path,
> src.stat.getOwner(),
> src.stat.getGroup());
> target.fs.setPermission(
> target.path,
> src.stat.getPermission());
> System.out.println("Preserving");
> if (src.fs.equals(target.fs)) {
> System.out.println("Same filesystems");
> src.fs.preserveAttributes(src.path, target.path);
> }
> throw new IOException("hi");
> }
> } finally {
> IOUtils.closeStream(in);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-9705) FsShell cp -p does not preserve directory attibutes
[
https://issues.apache.org/jira/browse/HADOOP-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034059#comment-14034059
]
Akira AJISAKA commented on HADOOP-9705:
---
I'll rebase the patch and make sure it works with all the attributes by
tomorrow. Thanks!
> FsShell cp -p does not preserve directory attibutes
> ---
>
> Key: HADOOP-9705
> URL: https://issues.apache.org/jira/browse/HADOOP-9705
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Stephen Chu
>Assignee: Akira AJISAKA
> Attachments: HADOOP-9705.2.patch, HADOOP-9705.3.patch,
> HADOOP-9705.4.patch, HADOOP-9705.patch
>
>
> HADOOP-9338 added the -p flag to preserve file attributes when copying.
> However, cp -p does not preserve directory attributes. It'd be useful to add
> this functionality.
> For example, the following shows that the modified time is not preserved
> {code}
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -mkdir
> /user/schu/testDir1
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu/
> Found 1 items
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -cp -p
> /user/schu/testDir1 /user/schu/testDir2
> [schu@hdfs-snapshots-1 ~]$ $HADOOP_HOME/bin/hdfs dfs -ls /user/schu
> Found 2 items
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:25 /user/schu/testDir1
> drwxr-xr-x - schu supergroup 0 2013-07-07 20:35 /user/schu/testDir2
> [schu@hdfs-snapshots-1 ~]$
> {code}
> The preserve logic is in CommandWithDestination#copyFileToTarget, which is
> only called with files.
> {code}
> protected void processPath(PathData src, PathData dst) throws IOException {
> if (src.stat.isSymlink()) {
> // TODO: remove when FileContext is supported, this needs to either
>
>
> // copy the symlink or deref the symlink
>
>
> throw new PathOperationException(src.toString());
> } else if (src.stat.isFile()) {
> copyFileToTarget(src, dst);
> } else if (src.stat.isDirectory() && !isRecursive()) {
> throw new PathIsDirectoryException(src.toString());
> }
> }
> {code}
> {code}
> /**
>
>
>* Copies the source file to the target.
>
>
>* @param src item to copy
>
>
>* @param target where to copy the item
>
>
>* @throws IOException if copy fails
>
>
>*/
> protected void copyFileToTarget(PathData src, PathData target) throws
> IOException {
> src.fs.setVerifyChecksum(verifyChecksum);
> if (src != null) {
> throw new PathExistsException("hi");
> }
> InputStream in = null;
> try {
> in = src.fs.open(src.path);
> copyStreamToTarget(in, target);
> if(preserve) {
> target.fs.setTimes(
> target.path,
> src.stat.getModificationTime(),
> src.stat.getAccessTime());
> target.fs.setOwner(
> target.path,
> src.stat.getOwner(),
> src.stat.getGroup());
> target.fs.setPermission(
> target.path,
> src.stat.getPermission());
> System.out.println("Preserving");
> if (src.fs.equals(target.fs)) {
> System.out.println("Same filesystems");
> src.fs.preserveAttributes(src.path, target.path);
> }
> throw new IOException("hi");
> }
> } finally {
> IOUtils.closeStream(in);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10557) FsShell -cp -pa option for preserving extended ACLs
[ https://issues.apache.org/jira/browse/HADOOP-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034055#comment-14034055 ] Hudson commented on HADOOP-10557: - FAILURE: Integrated in Hadoop-trunk-Commit #5718 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5718/]) HADOOP-10557. FsShell -cp -pa option for preserving extended ACLs. Contributed by Akira Ajisaka. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1603222) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/FileSystemShell.apt.vm * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/resources/testConf.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java > FsShell -cp -pa option for preserving extended ACLs > --- > > Key: HADOOP-10557 > URL: https://issues.apache.org/jira/browse/HADOOP-10557 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 2.4.0 >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA > Fix For: 3.0.0, 2.5.0 > > Attachments: HADOOP-10557.2.patch, HADOOP-10557.3.patch, > HADOOP-10557.4.patch, HADOOP-10557.5.patch, HADOOP-10557.patch > > > This issue tracks enhancing FsShell cp to add a new command-line option (-pa) > for preserving extended ACLs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10557) FsShell -cp -pa option for preserving extended ACLs
[ https://issues.apache.org/jira/browse/HADOOP-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034054#comment-14034054 ] Akira AJISAKA commented on HADOOP-10557: Thanks [~cnauroth] for testing! > FsShell -cp -pa option for preserving extended ACLs > --- > > Key: HADOOP-10557 > URL: https://issues.apache.org/jira/browse/HADOOP-10557 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 2.4.0 >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA > Fix For: 3.0.0, 2.5.0 > > Attachments: HADOOP-10557.2.patch, HADOOP-10557.3.patch, > HADOOP-10557.4.patch, HADOOP-10557.5.patch, HADOOP-10557.patch > > > This issue tracks enhancing FsShell cp to add a new command-line option (-pa) > for preserving extended ACLs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10660) GraphiteSink should implement Closeable
[ https://issues.apache.org/jira/browse/HADOOP-10660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen He updated HADOOP-10660: - Attachment: HADOOP-10660-v6.patch remove an unused variable "ArgumentCaptor argument" > GraphiteSink should implement Closeable > --- > > Key: HADOOP-10660 > URL: https://issues.apache.org/jira/browse/HADOOP-10660 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ted Yu >Assignee: Chen He > Attachments: HADOOP-10660-v2.patch, HADOOP-10660-v3.patch, > HADOOP-10660-v6.patch, HADOOP-10660.patch, hadoop-10660-v4.txt, > hadoop-10660-v5.txt > > > GraphiteSink wraps OutputStreamWriter around socket's output stream. > Currently the socket is never closed. > GraphiteSink should implement Closeable such that MetricsSystem can close the > socket when it is stopped. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10557) FsShell -cp -pa option for preserving extended ACLs
[ https://issues.apache.org/jira/browse/HADOOP-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-10557: --- Resolution: Fixed Fix Version/s: 2.5.0 3.0.0 Status: Resolved (was: Patch Available) I have committed the patch to trunk and branch-2. Akira, thank you for contributing the patch. Gera, thank you for providing code review feedback. > FsShell -cp -pa option for preserving extended ACLs > --- > > Key: HADOOP-10557 > URL: https://issues.apache.org/jira/browse/HADOOP-10557 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 2.4.0 >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA > Fix For: 3.0.0, 2.5.0 > > Attachments: HADOOP-10557.2.patch, HADOOP-10557.3.patch, > HADOOP-10557.4.patch, HADOOP-10557.5.patch, HADOOP-10557.patch > > > This issue tracks enhancing FsShell cp to add a new command-line option (-pa) > for preserving extended ACLs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10557) FsShell -cp -pa option for preserving extended ACLs
[ https://issues.apache.org/jira/browse/HADOOP-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-10557: --- Component/s: fs Hadoop Flags: Reviewed +1 for the patch. Thanks for incorporating the feedback. In addition to the automated tests here, I started a 2.3.0 cluster and ran a trunk client build with this change. I verified that there are no compatibility problems, and I verified that the command fails as expected when doing a cp -pa across different file systems and the target file system doesn't support ACLs. [~ajisakaa], while testing I noticed that ACLs were not preserved on directories. Then, I found HADOOP-9705 reporting the bug for all attributes. I'll resume conversation about that on the other jira. I'll commit this patch now. > FsShell -cp -pa option for preserving extended ACLs > --- > > Key: HADOOP-10557 > URL: https://issues.apache.org/jira/browse/HADOOP-10557 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 2.4.0 >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA > Attachments: HADOOP-10557.2.patch, HADOOP-10557.3.patch, > HADOOP-10557.4.patch, HADOOP-10557.5.patch, HADOOP-10557.patch > > > This issue tracks enhancing FsShell cp to add a new command-line option (-pa) > for preserving extended ACLs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10557) FsShell -cp -p does not preserve extended ACLs
[ https://issues.apache.org/jira/browse/HADOOP-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-10557: --- Issue Type: Improvement (was: Bug) > FsShell -cp -p does not preserve extended ACLs > -- > > Key: HADOOP-10557 > URL: https://issues.apache.org/jira/browse/HADOOP-10557 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.4.0 >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA > Attachments: HADOOP-10557.2.patch, HADOOP-10557.3.patch, > HADOOP-10557.4.patch, HADOOP-10557.5.patch, HADOOP-10557.patch > > > This issue tracks enhancing FsShell cp to add a new command-line option (-pa) > for preserving extended ACLs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10557) FsShell -cp -pa option for preserving extended ACLs
[ https://issues.apache.org/jira/browse/HADOOP-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-10557: --- Summary: FsShell -cp -pa option for preserving extended ACLs (was: FsShell -cp -p does not preserve extended ACLs) > FsShell -cp -pa option for preserving extended ACLs > --- > > Key: HADOOP-10557 > URL: https://issues.apache.org/jira/browse/HADOOP-10557 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.4.0 >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA > Attachments: HADOOP-10557.2.patch, HADOOP-10557.3.patch, > HADOOP-10557.4.patch, HADOOP-10557.5.patch, HADOOP-10557.patch > > > This issue tracks enhancing FsShell cp to add a new command-line option (-pa) > for preserving extended ACLs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10697) KMS DelegationToken support
[ https://issues.apache.org/jira/browse/HADOOP-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034035#comment-14034035 ] Alejandro Abdelnur commented on HADOOP-10697: - correct, this JIRA to to enable that. > KMS DelegationToken support > --- > > Key: HADOOP-10697 > URL: https://issues.apache.org/jira/browse/HADOOP-10697 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > > Add DelegationToken support to KMS as per discussion in HDFS-6134. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10660) GraphiteSink should implement Closeable
[ https://issues.apache.org/jira/browse/HADOOP-10660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HADOOP-10660: Attachment: hadoop-10660-v5.txt Patch v5 fixes compilation issue w.r.t. closing socket. > GraphiteSink should implement Closeable > --- > > Key: HADOOP-10660 > URL: https://issues.apache.org/jira/browse/HADOOP-10660 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ted Yu >Assignee: Chen He > Attachments: HADOOP-10660-v2.patch, HADOOP-10660-v3.patch, > HADOOP-10660.patch, hadoop-10660-v4.txt, hadoop-10660-v5.txt > > > GraphiteSink wraps OutputStreamWriter around socket's output stream. > Currently the socket is never closed. > GraphiteSink should implement Closeable such that MetricsSystem can close the > socket when it is stopped. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10590) ServiceAuthorizationManager is not threadsafe
[ https://issues.apache.org/jira/browse/HADOOP-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034014#comment-14034014 ] Daryn Sharp commented on HADOOP-10590: -- +1 I suppose a 2% degradation in a microbench for a method called once per connection is worth the correctness. > ServiceAuthorizationManager is not threadsafe > -- > > Key: HADOOP-10590 > URL: https://issues.apache.org/jira/browse/HADOOP-10590 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.4.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-10590.patch, performance-test-without-rpc.patch, > performancetest.patch > > > The mutators in ServiceAuthorizationManager are synchronized. The accessors > are not synchronized. > This results in visibility issues when ServiceAuthorizationManager's state > is accessed from different threads. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10707) support bzip2 in python avro tool
[ https://issues.apache.org/jira/browse/HADOOP-10707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034012#comment-14034012 ] Harsh J commented on HADOOP-10707: -- This is not a HADOOP project issue, but an AVRO one. As my comment noted, you have filed it with the correct project at AVRO-1527. Therefore, I've marked this as Invalid. Sorry for any ambiguity. You had marked a similar mistake invalid yesterday BTW: HADOOP-10708 > support bzip2 in python avro tool > - > > Key: HADOOP-10707 > URL: https://issues.apache.org/jira/browse/HADOOP-10707 > Project: Hadoop Common > Issue Type: Improvement > Components: tools >Reporter: Eustache >Priority: Minor > Labels: avro > > The Python tool to decode avro files is currently missing support for bzip2 > compression. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10660) GraphiteSink should implement Closeable
[
https://issues.apache.org/jira/browse/HADOOP-10660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034006#comment-14034006
]
Hadoop QA commented on HADOOP-10660:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650825/hadoop-10660-v4.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4085//console
This message is automatically generated.
> GraphiteSink should implement Closeable
> ---
>
> Key: HADOOP-10660
> URL: https://issues.apache.org/jira/browse/HADOOP-10660
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Chen He
> Attachments: HADOOP-10660-v2.patch, HADOOP-10660-v3.patch,
> HADOOP-10660.patch, hadoop-10660-v4.txt
>
>
> GraphiteSink wraps OutputStreamWriter around socket's output stream.
> Currently the socket is never closed.
> GraphiteSink should implement Closeable such that MetricsSystem can close the
> socket when it is stopped.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10701) The bug is while accessing NFS Mounted File System the permission is always validated based on the primary Unix group the user is associated with and Secondary Unix
[
https://issues.apache.org/jira/browse/HADOOP-10701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034005#comment-14034005
]
Hadoop QA commented on HADOOP-10701:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650820/HADOOP-10701.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4084//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/4084//console
This message is automatically generated.
> The bug is while accessing NFS Mounted File System the permission is always
> validated based on the primary Unix group the user is associated with and
> Secondary Unix groups are ignored.
> -
>
> Key: HADOOP-10701
> URL: https://issues.apache.org/jira/browse/HADOOP-10701
> Project: Hadoop Common
> Issue Type: Bug
> Components: nfs
>Affects Versions: 2.3.0
>Reporter: Premchandra Preetham Kukillaya
>Assignee: Harsh J
> Attachments: HADOOP-10701.patch, HADOOP-10701.patch,
> Hadoop_NFS_BugCreation_Steps.docx
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10707) support bzip2 in python avro tool
[
https://issues.apache.org/jira/browse/HADOOP-10707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033998#comment-14033998
]
Eustache commented on HADOOP-10707:
---
Example stack trace:
"""
Traceback (most recent call last):
File "../tools/avro2json.py", line 10, in
df_reader = datafile.DataFileReader(avrofile, io.DatumReader())
File "/Users/eustache/anaconda/lib/python2.7/site-packages/avro/datafile.py",
line 240, in __init__
raise DataFileException('Unknown codec: %s.' % self.codec)
avro.datafile.DataFileException: Unknown codec: bzip2.
"""
> support bzip2 in python avro tool
> -
>
> Key: HADOOP-10707
> URL: https://issues.apache.org/jira/browse/HADOOP-10707
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools
>Reporter: Eustache
>Priority: Minor
> Labels: avro
>
> The Python tool to decode avro files is currently missing support for bzip2
> compression.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10683) Users authenticated with KERBEROS are recorded as being authenticated with SIMPLE
[
https://issues.apache.org/jira/browse/HADOOP-10683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033999#comment-14033999
]
Chris Nauroth commented on HADOOP-10683:
[~daryn], thanks for taking a look. I slightly prefer the patch from Benoy
that has been committed already. The new method better communicates the intent
of the call site and doesn't create the object in a temporary incorrect state
with {{AuthenticationMethod#SIMPLE}}. If you feel strongly about it though, I
won't object to the change you suggested.
> Users authenticated with KERBEROS are recorded as being authenticated with
> SIMPLE
> -
>
> Key: HADOOP-10683
> URL: https://issues.apache.org/jira/browse/HADOOP-10683
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Fix For: 3.0.0, 2.5.0
>
> Attachments: HADOOP-10683.patch
>
>
> We have enabled kerberos authentication in our clusters, but we see the
> following in the log files
> 2014-06-11 11:07:05,903 INFO SecurityLogger.org.apache.hadoop.ipc.Server:
> Auth successful for x...@y.com (*auth:SIMPLE*)
> 2014-06-11 11:07:05,914 INFO
> SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager:
> Authorization successful for x...@y.com (auth:KERBEROS) for
> protocol=interface
> This is quite confusing for administrators.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10697) KMS DelegationToken support
[ https://issues.apache.org/jira/browse/HADOOP-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033993#comment-14033993 ] Owen O'Malley commented on HADOOP-10697: As I mentioned on the other jira, we should have the client get the tokens before submitting the application. > KMS DelegationToken support > --- > > Key: HADOOP-10697 > URL: https://issues.apache.org/jira/browse/HADOOP-10697 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > > Add DelegationToken support to KMS as per discussion in HDFS-6134. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10707) support bzip2 in python avro tool
[ https://issues.apache.org/jira/browse/HADOOP-10707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033992#comment-14033992 ] Eustache commented on HADOOP-10707: --- Hi Harsh, How is it invalid ? could you elaborate please ? As of 1.7.6 the python tool doesn't support bzip2... while snappy for instance is supported. > support bzip2 in python avro tool > - > > Key: HADOOP-10707 > URL: https://issues.apache.org/jira/browse/HADOOP-10707 > Project: Hadoop Common > Issue Type: Improvement > Components: tools >Reporter: Eustache >Priority: Minor > Labels: avro > > The Python tool to decode avro files is currently missing support for bzip2 > compression. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-10707) support bzip2 in python avro tool
[ https://issues.apache.org/jira/browse/HADOOP-10707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HADOOP-10707. -- Resolution: Invalid Moved to AVRO-1527 > support bzip2 in python avro tool > - > > Key: HADOOP-10707 > URL: https://issues.apache.org/jira/browse/HADOOP-10707 > Project: Hadoop Common > Issue Type: Improvement > Components: tools >Reporter: Eustache >Priority: Minor > Labels: avro > > The Python tool to decode avro files is currently missing support for bzip2 > compression. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10660) GraphiteSink should implement Closeable
[ https://issues.apache.org/jira/browse/HADOOP-10660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HADOOP-10660: Attachment: hadoop-10660-v4.txt Patch v4 incorporates Ravi's test which passes. > GraphiteSink should implement Closeable > --- > > Key: HADOOP-10660 > URL: https://issues.apache.org/jira/browse/HADOOP-10660 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ted Yu >Assignee: Chen He > Attachments: HADOOP-10660-v2.patch, HADOOP-10660-v3.patch, > HADOOP-10660.patch, hadoop-10660-v4.txt > > > GraphiteSink wraps OutputStreamWriter around socket's output stream. > Currently the socket is never closed. > GraphiteSink should implement Closeable such that MetricsSystem can close the > socket when it is stopped. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10683) Users authenticated with KERBEROS are recorded as being authenticated with SIMPLE
[ https://issues.apache.org/jira/browse/HADOOP-10683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033979#comment-14033979 ] Daryn Sharp commented on HADOOP-10683: -- Instead of creating yet another UGI method, why not just have Server call the existing setAuthenticationMethod? > Users authenticated with KERBEROS are recorded as being authenticated with > SIMPLE > - > > Key: HADOOP-10683 > URL: https://issues.apache.org/jira/browse/HADOOP-10683 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.0.0, 2.4.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Fix For: 3.0.0, 2.5.0 > > Attachments: HADOOP-10683.patch > > > We have enabled kerberos authentication in our clusters, but we see the > following in the log files > 2014-06-11 11:07:05,903 INFO SecurityLogger.org.apache.hadoop.ipc.Server: > Auth successful for x...@y.com (*auth:SIMPLE*) > 2014-06-11 11:07:05,914 INFO > SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: > Authorization successful for x...@y.com (auth:KERBEROS) for > protocol=interface > This is quite confusing for administrators. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10701) The bug is while accessing NFS Mounted File System the permission is always validated based on the primary Unix group the user is associated with and Secondary Unix gr
[
https://issues.apache.org/jira/browse/HADOOP-10701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated HADOOP-10701:
-
Attachment: HADOOP-10701.patch
The findbugs warnings are related, copied below:
{code}
EI org.apache.hadoop.oncrpc.security.CredentialsSys.getAuxGIDs() may
expose internal representation by returning CredentialsSys.mAuxGIDs
EI2 org.apache.hadoop.oncrpc.security.CredentialsSys.setAuxGIDs(int[]) may
expose internal representation by storing an externally mutable object into
CredentialsSys.mAuxGIDs
{code}
But I think returning a clone of the array each request would end up being
expensive. We could make it into a list (unnecessary overhead again), or just
ignore these medium level warnings. I've gone with the latter.
> The bug is while accessing NFS Mounted File System the permission is always
> validated based on the primary Unix group the user is associated with and
> Secondary Unix groups are ignored.
> -
>
> Key: HADOOP-10701
> URL: https://issues.apache.org/jira/browse/HADOOP-10701
> Project: Hadoop Common
> Issue Type: Bug
> Components: nfs
>Affects Versions: 2.3.0
>Reporter: Premchandra Preetham Kukillaya
>Assignee: Harsh J
> Attachments: HADOOP-10701.patch, HADOOP-10701.patch,
> Hadoop_NFS_BugCreation_Steps.docx
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10683) Users authenticated with KERBEROS are recorded as being authenticated with SIMPLE
[ https://issues.apache.org/jira/browse/HADOOP-10683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033907#comment-14033907 ] Hudson commented on HADOOP-10683: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1804 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1804/]) HADOOP-10683. Users authenticated with KERBEROS are recorded as being authenticated with SIMPLE. Contributed by Benoy Antony. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1602991) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java > Users authenticated with KERBEROS are recorded as being authenticated with > SIMPLE > - > > Key: HADOOP-10683 > URL: https://issues.apache.org/jira/browse/HADOOP-10683 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.0.0, 2.4.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > Fix For: 3.0.0, 2.5.0 > > Attachments: HADOOP-10683.patch > > > We have enabled kerberos authentication in our clusters, but we see the > following in the log files > 2014-06-11 11:07:05,903 INFO SecurityLogger.org.apache.hadoop.ipc.Server: > Auth successful for x...@y.com (*auth:SIMPLE*) > 2014-06-11 11:07:05,914 INFO > SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: > Authorization successful for x...@y.com (auth:KERBEROS) for > protocol=interface > This is quite confusing for administrators. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10666) Remove Copyright /d/d/d/d Apache Software Foundation from the source files license header
[ https://issues.apache.org/jira/browse/HADOOP-10666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033915#comment-14033915 ] Hudson commented on HADOOP-10666: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1804 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1804/]) HADOOP-10666. Remove Copyright /d/d/d/d Apache Software Foundation from the source files license header. Contributed by Henry Saputra. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1603025) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-policy.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties * /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/protoc/ProtocMojo.java * /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/util/Exec.java * /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/util/FileSetUtils.java * /hadoop/common/trunk/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/versioninfo/VersionInfoMojo.java * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/resources/distcp-default.xml * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/resources/sslConfig.xml > Remove Copyright /d/d/d/d Apache Software Foundation from the source files > license header > - > > Key: HADOOP-10666 > URL: https://issues.apache.org/jira/browse/HADOOP-10666 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 2.4.0 >Reporter: Henry Saputra >Assignee: Henry Saputra >Priority: Minor > Fix For: 2.5.0 > > Attachments: HADOOP-10666.patch > > > Some of the files have "Copyright 2011 Apache Software Foundation" in the > license header comment. > This is not the right license header per ASF rule [1] > From the ASF header rule [1] : > "Source File Headers for Code Developed at the ASF" > "This section refers only to works submitted directly to the ASF by the > copyright owner or owner's agent." > Seems like the one without Copyright notice is used for software developed > directly by ASF which Apache Hadoop is. > When it was under external organization it does need to use the one from ASF > 2.0 license [2] which requires Copyright included. > [1] http://www.apache.org/legal/src-headers.html > [2] http://www.apache.org/licenses/LICENSE-2.0 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10702) KerberosAuthenticationHandler does not log the principal names correctly
[ https://issues.apache.org/jira/browse/HADOOP-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033910#comment-14033910 ] Hudson commented on HADOOP-10702: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1804 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1804/]) HADOOP-10702. KerberosAuthenticationHandler does not log the principal names correctly. Contributed by Benoy Antony. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1603023) * /hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/KerberosAuthenticationHandler.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt > KerberosAuthenticationHandler does not log the principal names correctly > > > Key: HADOOP-10702 > URL: https://issues.apache.org/jira/browse/HADOOP-10702 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.5.0 >Reporter: Benoy Antony >Assignee: Benoy Antony >Priority: Minor > Fix For: 3.0.0, 2.5.0 > > Attachments: HADOOP-10702.patch > > > With HADOOP-10158, it is possible to load multiple principal names or all > HTTP principals in the key tab by specifying “*”. > Each principal name is logged when when the principal is loaded from key tab. > But there is a bug due to which principal name is logged each time as either > “*” or the full list of principals. > The log snippet is as below: > 2014-06-15 00:19:13,288 INFO > org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: > Login using keytab /etc/hadoop/hadoop.keytab, for principal * > 2014-06-15 00:19:13,292 INFO > org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: > Login using keytab /etc/hadoop/hadoop.keytab, for principal * > 2014-06-15 00:19:13,294 INFO > org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: > Login using keytab /etc/hadoop/hadoop.keytab, for principal * > 2014-06-15 00:19:13,295 INFO > org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: > Login using keytab /etc/hadoop/hadoop.keytab, for principal * -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10657) Have RetryInvocationHandler log failover attempt at INFO level
[ https://issues.apache.org/jira/browse/HADOOP-10657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033908#comment-14033908 ] Hudson commented on HADOOP-10657: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1804 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1804/]) HADOOP-10657. Have RetryInvocationHandler log failover attempt at INFO level. Contributed by Ming Ma. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1602941) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryInvocationHandler.java > Have RetryInvocationHandler log failover attempt at INFO level > -- > > Key: HADOOP-10657 > URL: https://issues.apache.org/jira/browse/HADOOP-10657 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ming Ma >Assignee: Ming Ma > Fix For: 2.5.0 > > Attachments: HADOOP-10657-2.patch, HADOOP-10657.patch > > > RetryInovcationHandler uses worthLogging flag to decide if it will do > logging. worthLogging will be false for first fails over given > invocationFailoverCount is zero. That addresses the log noise where the > second-listed NN is active. > For other failover scenarios, it will be useful to log the error message at > info level for analysis purpose. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10699) Fix build native library on mac osx
[ https://issues.apache.org/jira/browse/HADOOP-10699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033904#comment-14033904 ] Hudson commented on HADOOP-10699: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1804 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1804/]) HADOOP-10699. Fix build native library on mac osx. Contributed by Binglin Chang (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1603042) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c > Fix build native library on mac osx > --- > > Key: HADOOP-10699 > URL: https://issues.apache.org/jira/browse/HADOOP-10699 > Project: Hadoop Common > Issue Type: Bug >Reporter: Kirill A. Korinskiy >Assignee: Binglin Chang > Fix For: 3.0.0, 2.5.0 > > Attachments: HADOOP-10699-common.v3.patch, > HADOOP-9648-native-osx.1.0.4.patch, HADOOP-9648-native-osx.1.1.2.patch, > HADOOP-9648-native-osx.1.2.0.patch, > HADOOP-9648-native-osx.2.0.5-alpha-rc1.patch, HADOOP-9648.v2.patch > > > Some patches for fixing build a hadoop native library on os x 10.7/10.8. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10715) Make GraphiteSink#setWriter() private
Ted Yu created HADOOP-10715: --- Summary: Make GraphiteSink#setWriter() private Key: HADOOP-10715 URL: https://issues.apache.org/jira/browse/HADOOP-10715 Project: Hadoop Common Issue Type: Task Reporter: Ted Yu Priority: Minor During review of HADOOP-10660, Ravi brought up the notion of making GraphiteSink#setWriter() private. This JIRA is to address Ravi's comment. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10660) GraphiteSink should implement Closeable
[ https://issues.apache.org/jira/browse/HADOOP-10660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033856#comment-14033856 ] Ted Yu commented on HADOOP-10660: - I logged HADOOP-10715 for making GraphiteSink#setWriter() private. > GraphiteSink should implement Closeable > --- > > Key: HADOOP-10660 > URL: https://issues.apache.org/jira/browse/HADOOP-10660 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ted Yu >Assignee: Chen He > Attachments: HADOOP-10660-v2.patch, HADOOP-10660-v3.patch, > HADOOP-10660.patch > > > GraphiteSink wraps OutputStreamWriter around socket's output stream. > Currently the socket is never closed. > GraphiteSink should implement Closeable such that MetricsSystem can close the > socket when it is stopped. -- This message was sent by Atlassian JIRA (v6.2#6252)
