[jira] [Commented] (HADOOP-15518) Authentication filter calling handler after request already authenticated
[
https://issues.apache.org/jira/browse/HADOOP-15518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17160548#comment-17160548
]
Jitendra Nath Pandey commented on HADOOP-15518:
---
[~prabhujoseph]
Is this jira still relevant given bunch of changes have gone in with
HADOOP-16095 and its subtasks, particularly HADOOP-16314?
> Authentication filter calling handler after request already authenticated
> -
>
> Key: HADOOP-15518
> URL: https://issues.apache.org/jira/browse/HADOOP-15518
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.7.1
>Reporter: Kevin Minder
>Assignee: Kevin Minder
>Priority: Major
> Attachments: HADOOP-15518-001.patch, HADOOP-15518.002.patch
>
>
> The hadoop-auth AuthenticationFilter will invoke its handler even if a prior
> successful authentication has occurred in the current request. This
> primarily affects situations where multiple authentication mechanism has been
> configured. For example when core-site.xml's has
> hadoop.http.authentication.type=kerberos and yarn-site.xml has
> yarn.timeline-service.http-authentication.type=kerberos the result is an
> attempt to perform two Kerberos authentications for the same request. This
> in turn results in Kerberos triggering a replay attack detection. The
> javadocs for AuthenticationHandler
> ([https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationHandler.java)]
> indicate for the authenticate method that
> {quote}This method is invoked by the AuthenticationFilter only if the HTTP
> client request is not yet authenticated.
> {quote}
> This does not appear to be the case in practice.
> I've create a patch and tested on a limited number of functional use cases
> (e.g. the timeline-service issue noted above). If there is general agreement
> that the change is valid I'll add unit tests to the patch.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15518) Authentication filter calling handler after request already authenticated
[
https://issues.apache.org/jira/browse/HADOOP-15518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158702#comment-17158702
]
Jitendra Nath Pandey commented on HADOOP-15518:
---
[~kminder], [~eyang], is this patch still valid?
cc [~xyao]
> Authentication filter calling handler after request already authenticated
> -
>
> Key: HADOOP-15518
> URL: https://issues.apache.org/jira/browse/HADOOP-15518
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.7.1
>Reporter: Kevin Minder
>Assignee: Kevin Minder
>Priority: Major
> Attachments: HADOOP-15518-001.patch
>
>
> The hadoop-auth AuthenticationFilter will invoke its handler even if a prior
> successful authentication has occurred in the current request. This
> primarily affects situations where multiple authentication mechanism has been
> configured. For example when core-site.xml's has
> hadoop.http.authentication.type=kerberos and yarn-site.xml has
> yarn.timeline-service.http-authentication.type=kerberos the result is an
> attempt to perform two Kerberos authentications for the same request. This
> in turn results in Kerberos triggering a replay attack detection. The
> javadocs for AuthenticationHandler
> ([https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationHandler.java)]
> indicate for the authenticate method that
> {quote}This method is invoked by the AuthenticationFilter only if the HTTP
> client request is not yet authenticated.
> {quote}
> This does not appear to be the case in practice.
> I've create a patch and tested on a limited number of functional use cases
> (e.g. the timeline-service issue noted above). If there is general agreement
> that the change is valid I'll add unit tests to the patch.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16761) KMSClientProvider does not work with client using ticket logged in externally
[ https://issues.apache.org/jira/browse/HADOOP-16761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HADOOP-16761: -- Priority: Blocker (was: Major) > KMSClientProvider does not work with client using ticket logged in externally > -- > > Key: HADOOP-16761 > URL: https://issues.apache.org/jira/browse/HADOOP-16761 > Project: Hadoop Common > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Blocker > > This is a regression from HDFS-13682 that checks not only the kerberos > credential but also enforce the login is non-external. This breaks client > applications that need to access HDFS encrypted file using kerberos ticket > that logged in external in ticket cache. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16513) Upgrade Bootstrap, jquery and wildfly
[ https://issues.apache.org/jira/browse/HADOOP-16513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey reassigned HADOOP-16513: - Assignee: Vivek Ratnavel Subramanian > Upgrade Bootstrap, jquery and wildfly > - > > Key: HADOOP-16513 > URL: https://issues.apache.org/jira/browse/HADOOP-16513 > Project: Hadoop Common > Issue Type: Bug > Components: website >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > > The current versions of bootstrap, jquery and wildfly have multiple medium > severity CVEs reported till date and needs to be updated to the latest > versions with no reported CVEs. > > I suggest updating the following libraries: > ||Library||From version||To version|| > |Bootstrap|3.3.7|3.4.1| > |jQuery|3.3.1|3.4.1| > |Wildfly|11.0.0.Beta1|12.0.0| -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16198) Upgrade Jackson-databind version to 2.9.8
[ https://issues.apache.org/jira/browse/HADOOP-16198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HADOOP-16198: -- Description: Jackson-databind 2.9.8 has a few fixes which are important to include. (was: ++Jackson-databind 2.9.8 has a few fixes which are important to include.) > Upgrade Jackson-databind version to 2.9.8 > - > > Key: HADOOP-16198 > URL: https://issues.apache.org/jira/browse/HADOOP-16198 > Project: Hadoop Common > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > > Jackson-databind 2.9.8 has a few fixes which are important to include. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16198) Upgrade Jackson-databind version to 2.9.8
[ https://issues.apache.org/jira/browse/HADOOP-16198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HADOOP-16198: -- Description: ++Jackson-databind 2.9.8 has a few fixes which are important to include. > Upgrade Jackson-databind version to 2.9.8 > - > > Key: HADOOP-16198 > URL: https://issues.apache.org/jira/browse/HADOOP-16198 > Project: Hadoop Common > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > > ++Jackson-databind 2.9.8 has a few fixes which are important to include. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16198) Upgrade Jackson-databind version to 2.9.8
[ https://issues.apache.org/jira/browse/HADOOP-16198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HADOOP-16198: -- Description: (was: Jackson-databind is affected by below CVEs and are getting reported by Customers. CVE-2018-14719 CVE-2018-14720 CVE-2018-14721 CVE-2018-1000873 CVE-2018-7489 CVE-2018-19362 CVE-2017-15095 CVE-2018-19361 CVE-2017-7525 CVE-2018-19360 CVE-2017-17485 CVE-2018-5968 We need to upgrade this to version 2.9.8.) > Upgrade Jackson-databind version to 2.9.8 > - > > Key: HADOOP-16198 > URL: https://issues.apache.org/jira/browse/HADOOP-16198 > Project: Hadoop Common > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14556) S3A to support Delegation Tokens
[
https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HADOOP-14556:
--
Fix Version/s: 3.3.0
> S3A to support Delegation Tokens
>
>
> Key: HADOOP-14556
> URL: https://issues.apache.org/jira/browse/HADOOP-14556
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch,
> HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch,
> HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch,
> HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch,
> HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch,
> HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch,
> HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch,
> HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch,
> HADOOP-14556-024.patch, HADOOP-14556-025.patch, HADOOP-14556-026.patch,
> HADOOP-14556-027.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch
>
>
> S3A to support delegation tokens where
> * an authenticated client can request a token via
> {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id;
> these will be saved in the token and marshalled with jobs
> * A new authentication provider will look for a token for the current user
> and authenticate the user if found
> This will not support renewals; the lifespan of a token will be limited to
> the initial duration. Also, as you can't request an STS token from a
> temporary session, IAM instances won't be able to issue tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11867) FS API: Add a high-performance vectored Read to FSDataInputStream API
[
https://issues.apache.org/jira/browse/HADOOP-11867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16705366#comment-16705366
]
Jitendra Nath Pandey commented on HADOOP-11867:
---
{quote}I'd suggest that we add readAsync to PositionedReadable
{quote}
Yes, {{OzoneFSInputStream}} also extends FSInputStream, it would be lot better
to put it in PositionedReadable.
> FS API: Add a high-performance vectored Read to FSDataInputStream API
> -
>
> Key: HADOOP-11867
> URL: https://issues.apache.org/jira/browse/HADOOP-11867
> Project: Hadoop Common
> Issue Type: New Feature
> Components: hdfs-client
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Owen O'Malley
>Priority: Major
> Labels: performance
>
> The most significant way to read from a filesystem in an efficient way is to
> let the FileSystem implementation handle the seek behaviour underneath the
> API to be the most efficient as possible.
> A better approach to the seek problem is to provide a sequence of read
> locations as part of a single call, while letting the system schedule/plan
> the reads ahead of time.
> This is exceedingly useful for seek-heavy readers on HDFS, since this allows
> for potentially optimizing away the seek-gaps within the FSDataInputStream
> implementation.
> For seek+read systems with even more latency than locally-attached disks,
> something like a {{readFully(long[] offsets, ByteBuffer[] chunks)}} would
> take of the seeks internally while reading chunk.remaining() bytes into each
> chunk (which may be {{slice()}}ed off a bigger buffer).
> The base implementation can stub in this as a sequence of seeks + read() into
> ByteBuffers, without forcing each FS implementation to override this in any
> way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15957) WASB: Add asterisk wildcard support for PageBlobDirSet
[
https://issues.apache.org/jira/browse/HADOOP-15957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702701#comment-16702701
]
Jitendra Nath Pandey commented on HADOOP-15957:
---
cc [~steve_l]
> WASB: Add asterisk wildcard support for PageBlobDirSet
> --
>
> Key: HADOOP-15957
> URL: https://issues.apache.org/jira/browse/HADOOP-15957
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Da Zhou
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-15957-001.patch
>
>
> In WASB, property "*fs.azure.page.blob.dir*" only support literal directory
> name.
> We need to add support for wildcard '*' to represent for any directory name.
> For example, the following pattern should be supported:
> {code:java}
> /dir1/dir2
> /dir1/*/dir3
> /dir1/*/*/dir4
> /dir1/*/*/file
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15857) Remove ozonefs class name definition from core-default.xml
[ https://issues.apache.org/jira/browse/HADOOP-15857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651731#comment-16651731 ] Jitendra Nath Pandey commented on HADOOP-15857: --- +1 for the addendum patch. Thanks for addressing this [~elek]. > Remove ozonefs class name definition from core-default.xml > -- > > Key: HADOOP-15857 > URL: https://issues.apache.org/jira/browse/HADOOP-15857 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Blocker > Fix For: 3.2.0 > > Attachments: HADOOP-15857-branch-3.2.001.patch, > HADOOP-15857-branch-3.2.addendum.patch > > > Ozone file system is under renaming in HDDS-651 from o3:// to o3fs://. But > branch-3.2 still contains a reference with o3://. > The easiest way to fix it just remove the fs.o3.imp definition from > core-default.xml from branch-3.2 as since HDDS-654 the file system could be > registered with Service Provider Interface (META-INF/services...) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15857) Remove ozonefs class name definition from core-default.xml
[ https://issues.apache.org/jira/browse/HADOOP-15857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651711#comment-16651711 ] Jitendra Nath Pandey commented on HADOOP-15857: --- I think we should remove fs.AbstractFileSystem.o3.impl as well from branch-3.2. > Remove ozonefs class name definition from core-default.xml > -- > > Key: HADOOP-15857 > URL: https://issues.apache.org/jira/browse/HADOOP-15857 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Blocker > Fix For: 3.2.0 > > Attachments: HADOOP-15857-branch-3.2.001.patch > > > Ozone file system is under renaming in HDDS-651 from o3:// to o3fs://. But > branch-3.2 still contains a reference with o3://. > The easiest way to fix it just remove the fs.o3.imp definition from > core-default.xml from branch-3.2 as since HDDS-654 the file system could be > registered with Service Provider Interface (META-INF/services...) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15483) Upgrade jquery to version 3.3.1
[ https://issues.apache.org/jira/browse/HADOOP-15483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520755#comment-16520755 ] Jitendra Nath Pandey commented on HADOOP-15483: --- I am +1 to backport this to 3.1. > Upgrade jquery to version 3.3.1 > --- > > Key: HADOOP-15483 > URL: https://issues.apache.org/jira/browse/HADOOP-15483 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15483.001.patch, HADOOP-15483.002.patch, > HADOOP-15483.003.patch, HADOOP-15483.004.patch, HADOOP-15483.005.patch, > HADOOP-15483.006.patch, HADOOP-15483.007.patch, HADOOP-15483.008.patch > > > This Jira aims to upgrade jquery to version 3.3.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15483) Upgrade jquery to version 3.3.1
[ https://issues.apache.org/jira/browse/HADOOP-15483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HADOOP-15483: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Upgrade jquery to version 3.3.1 > --- > > Key: HADOOP-15483 > URL: https://issues.apache.org/jira/browse/HADOOP-15483 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15483.001.patch, HADOOP-15483.002.patch, > HADOOP-15483.003.patch, HADOOP-15483.004.patch, HADOOP-15483.005.patch, > HADOOP-15483.006.patch, HADOOP-15483.007.patch, HADOOP-15483.008.patch > > > This Jira aims to upgrade jquery to version 3.3.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15484) Upgrade moment.js to version 2.22.1
[ https://issues.apache.org/jira/browse/HADOOP-15484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HADOOP-15484: -- Resolution: Duplicate Status: Resolved (was: Patch Available) These changes are covered by HADOOP-15483. > Upgrade moment.js to version 2.22.1 > --- > > Key: HADOOP-15484 > URL: https://issues.apache.org/jira/browse/HADOOP-15484 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HADOOP-15484.001.patch > > > This Jira aims to upgrade moment.js to version 2.22.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15483) Upgrade jquery to version 3.3.1
[ https://issues.apache.org/jira/browse/HADOOP-15483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510728#comment-16510728 ] Jitendra Nath Pandey commented on HADOOP-15483: --- I have committed this to trunk. Thanks [~ljain], [~msingh] and [~sunilg]. > Upgrade jquery to version 3.3.1 > --- > > Key: HADOOP-15483 > URL: https://issues.apache.org/jira/browse/HADOOP-15483 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15483.001.patch, HADOOP-15483.002.patch, > HADOOP-15483.003.patch, HADOOP-15483.004.patch, HADOOP-15483.005.patch, > HADOOP-15483.006.patch, HADOOP-15483.007.patch, HADOOP-15483.008.patch > > > This Jira aims to upgrade jquery to version 3.3.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15483) Upgrade jquery to version 3.3.1
[ https://issues.apache.org/jira/browse/HADOOP-15483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HADOOP-15483: -- Fix Version/s: 3.2.0 > Upgrade jquery to version 3.3.1 > --- > > Key: HADOOP-15483 > URL: https://issues.apache.org/jira/browse/HADOOP-15483 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15483.001.patch, HADOOP-15483.002.patch, > HADOOP-15483.003.patch, HADOOP-15483.004.patch, HADOOP-15483.005.patch, > HADOOP-15483.006.patch, HADOOP-15483.007.patch, HADOOP-15483.008.patch > > > This Jira aims to upgrade jquery to version 3.3.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15483) Upgrade jquery to version 3.3.1
[ https://issues.apache.org/jira/browse/HADOOP-15483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508686#comment-16508686 ] Jitendra Nath Pandey commented on HADOOP-15483: --- [~sunilg], [~msingh], one question. This patch adds bootstrap-3.3.7, however removes only a few files from bootstrap-3.3.7. Why don't we remove bootstrap-3.0.2 altogether? I don't see it being used anywhere. > Upgrade jquery to version 3.3.1 > --- > > Key: HADOOP-15483 > URL: https://issues.apache.org/jira/browse/HADOOP-15483 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HADOOP-15483.001.patch, HADOOP-15483.002.patch, > HADOOP-15483.003.patch, HADOOP-15483.004.patch, HADOOP-15483.005.patch, > HADOOP-15483.006.patch, HADOOP-15483.007.patch > > > This Jira aims to upgrade jquery to version 3.3.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15483) Upgrade jquery to version 3.3.1
[ https://issues.apache.org/jira/browse/HADOOP-15483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508352#comment-16508352 ] Jitendra Nath Pandey commented on HADOOP-15483: --- Since both HDFS and Yarn UI are verified with this change, I am inclined to commit it. +1 > Upgrade jquery to version 3.3.1 > --- > > Key: HADOOP-15483 > URL: https://issues.apache.org/jira/browse/HADOOP-15483 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HADOOP-15483.001.patch, HADOOP-15483.002.patch, > HADOOP-15483.003.patch, HADOOP-15483.004.patch, HADOOP-15483.005.patch, > HADOOP-15483.006.patch, HADOOP-15483.007.patch > > > This Jira aims to upgrade jquery to version 3.3.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15483) Upgrade jquery to version 3.3.1
[ https://issues.apache.org/jira/browse/HADOOP-15483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508351#comment-16508351 ] Jitendra Nath Pandey commented on HADOOP-15483: --- It seems the patch in this Jira is a super set of HADOOP-15484. > Upgrade jquery to version 3.3.1 > --- > > Key: HADOOP-15483 > URL: https://issues.apache.org/jira/browse/HADOOP-15483 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HADOOP-15483.001.patch, HADOOP-15483.002.patch, > HADOOP-15483.003.patch, HADOOP-15483.004.patch, HADOOP-15483.005.patch, > HADOOP-15483.006.patch, HADOOP-15483.007.patch > > > This Jira aims to upgrade jquery to version 3.3.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15471) Hdfs recursive listing operation is very slow
[ https://issues.apache.org/jira/browse/HADOOP-15471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506689#comment-16506689 ] Jitendra Nath Pandey commented on HADOOP-15471: --- [~ajaysachdev], please review the javac/findbugs issues and test failures. Please fix them if they are related to this patch. > Hdfs recursive listing operation is very slow > - > > Key: HADOOP-15471 > URL: https://issues.apache.org/jira/browse/HADOOP-15471 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.1 > Environment: HCFS file system where HDP 2.6.1 is connected to ECS > (Object Store). >Reporter: Ajay Sachdev >Assignee: Ajay Sachdev >Priority: Major > Fix For: 2.7.1 > > Attachments: HDFS-13398.001.patch, HDFS-13398.002.patch, > HDFS-13398.003.patch, parallelfsPatch > > > The hdfs dfs -ls -R command is sequential in nature and is very slow for a > HCFS system. We have seen around 6 mins for 40K directory/files structure. > The proposal is to use multithreading approach to speed up recursive list, du > and count operations. > We have tried a ForkJoinPool implementation to improve performance for > recursive listing operation. > [https://github.com/jasoncwik/hadoop-release/tree/parallel-fs-cli] > commit id : > 82387c8cd76c2e2761bd7f651122f83d45ae8876 > Another implementation is to use Java Executor Service to improve performance > to run listing operation in multiple threads in parallel. This has > significantly reduced the time to 40 secs from 6 mins. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15482) Upgrade jackson-databind to version 2.9.5
[ https://issues.apache.org/jira/browse/HADOOP-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505734#comment-16505734 ] Jitendra Nath Pandey commented on HADOOP-15482: --- I have committed to trunk. Thanks [~ljain] for the contribution. > Upgrade jackson-databind to version 2.9.5 > - > > Key: HADOOP-15482 > URL: https://issues.apache.org/jira/browse/HADOOP-15482 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15482.001.patch, HADOOP-15482.002.patch > > > This Jira aims to upgrade jackson-databind to version 2.9.5 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15482) Upgrade jackson-databind to version 2.9.5
[ https://issues.apache.org/jira/browse/HADOOP-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HADOOP-15482: -- Resolution: Fixed Fix Version/s: 3.2.0 Status: Resolved (was: Patch Available) > Upgrade jackson-databind to version 2.9.5 > - > > Key: HADOOP-15482 > URL: https://issues.apache.org/jira/browse/HADOOP-15482 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15482.001.patch, HADOOP-15482.002.patch > > > This Jira aims to upgrade jackson-databind to version 2.9.5 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15484) Upgrade moment.js to version 2.22.1
[ https://issues.apache.org/jira/browse/HADOOP-15484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500790#comment-16500790 ] Jitendra Nath Pandey commented on HADOOP-15484: --- The patch looks good to me. +1 > Upgrade moment.js to version 2.22.1 > --- > > Key: HADOOP-15484 > URL: https://issues.apache.org/jira/browse/HADOOP-15484 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HADOOP-15484.001.patch > > > This Jira aims to upgrade moment.js to version 2.22.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15483) Upgrade jquery to version 3.3.1
[ https://issues.apache.org/jira/browse/HADOOP-15483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500791#comment-16500791 ] Jitendra Nath Pandey commented on HADOOP-15483: --- +1 for the patch. > Upgrade jquery to version 3.3.1 > --- > > Key: HADOOP-15483 > URL: https://issues.apache.org/jira/browse/HADOOP-15483 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HADOOP-15483.001.patch, HADOOP-15483.002.patch, > HADOOP-15483.003.patch, HADOOP-15483.004.patch, HADOOP-15483.005.patch, > HADOOP-15483.006.patch > > > This Jira aims to upgrade jquery to version 3.3.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15137) ClassNotFoundException: org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using hadoop-client-minicluster
[
https://issues.apache.org/jira/browse/HADOOP-15137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500603#comment-16500603
]
Jitendra Nath Pandey commented on HADOOP-15137:
---
I think this patch is ready to be committed +1.
Any further enhancements needed should be added as subtasks for the parent jira.
> ClassNotFoundException:
> org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using
> hadoop-client-minicluster
> --
>
> Key: HADOOP-15137
> URL: https://issues.apache.org/jira/browse/HADOOP-15137
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Jeff Zhang
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HADOOP-15137.01.patch, HADOOP-15137.02.patch,
> YARN-7673.00.patch
>
>
> I'd like to use hadoop-client-minicluster for hadoop downstream project, but
> I encounter the following exception when starting hadoop minicluster. And I
> check the hadoop-client-minicluster, it indeed does not have this class. Is
> this something that is missing when packaging the published jar ?
> {code}
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/yarn/server/api/DistributedSchedulingAMProtocol
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.createResourceManager(MiniYARNCluster.java:851)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.serviceInit(MiniYARNCluster.java:285)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15482) Upgrade jackson-databind to version 2.9.5
[ https://issues.apache.org/jira/browse/HADOOP-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500533#comment-16500533 ] Jitendra Nath Pandey commented on HADOOP-15482: --- I am +1 to commit this, unless there are objections, will wait for a couple of days. > Upgrade jackson-databind to version 2.9.5 > - > > Key: HADOOP-15482 > URL: https://issues.apache.org/jira/browse/HADOOP-15482 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HADOOP-15482.001.patch, HADOOP-15482.002.patch > > > This Jira aims to upgrade jackson-databind to version 2.9.5 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15482) Upgrade jackson-databind to version 2.9.5
[ https://issues.apache.org/jira/browse/HADOOP-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499709#comment-16499709 ] Jitendra Nath Pandey commented on HADOOP-15482: --- [~ste...@apache.org], so far in our internal testing, we haven't found any issue with spark/hive builds or tests due to jackson databind upgrade. It seems this time we have been lucky, with these version changes. A big jump in version number came from HADOOP-15299. This Jira changes only from 2.9.4 to 2.9.5. > Upgrade jackson-databind to version 2.9.5 > - > > Key: HADOOP-15482 > URL: https://issues.apache.org/jira/browse/HADOOP-15482 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HADOOP-15482.001.patch, HADOOP-15482.002.patch > > > This Jira aims to upgrade jackson-databind to version 2.9.5 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15484) Upgrade moment.js to version 2.22.1
[ https://issues.apache.org/jira/browse/HADOOP-15484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16498624#comment-16498624 ] Jitendra Nath Pandey commented on HADOOP-15484: --- cc [~wangda], [~vinodkv] > Upgrade moment.js to version 2.22.1 > --- > > Key: HADOOP-15484 > URL: https://issues.apache.org/jira/browse/HADOOP-15484 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HADOOP-15484.001.patch > > > This Jira aims to upgrade moment.js to version 2.22.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15483) Upgrade jquery to version 3.3.1
[ https://issues.apache.org/jira/browse/HADOOP-15483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16498623#comment-16498623 ] Jitendra Nath Pandey commented on HADOOP-15483: --- cc [~wangda] [~vinodkv] > Upgrade jquery to version 3.3.1 > --- > > Key: HADOOP-15483 > URL: https://issues.apache.org/jira/browse/HADOOP-15483 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HADOOP-15483.001.patch, HADOOP-15483.002.patch, > HADOOP-15483.003.patch > > > This Jira aims to upgrade jquery to version 3.3.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15482) Upgrade jackson-databind to version 2.9.5
[
https://issues.apache.org/jira/browse/HADOOP-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16482893#comment-16482893
]
Jitendra Nath Pandey commented on HADOOP-15482:
---
{quote}Can we modify the jackson2.version property instead of overwriting its
usage?
{quote}
Agreed, that makes sense.
> Upgrade jackson-databind to version 2.9.5
> -
>
> Key: HADOOP-15482
> URL: https://issues.apache.org/jira/browse/HADOOP-15482
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Lokesh Jain
>Assignee: Lokesh Jain
>Priority: Major
> Attachments: HADOOP-15482.001.patch
>
>
> This Jira aims to upgrade jackson-databind to version 2.9.5
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15482) Upgrade jackson-databind to version 2.9.5
[ https://issues.apache.org/jira/browse/HADOOP-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16482851#comment-16482851 ] Jitendra Nath Pandey commented on HADOOP-15482: --- [~mackrorysd], [~ste...@apache.org] , thoughts? > Upgrade jackson-databind to version 2.9.5 > - > > Key: HADOOP-15482 > URL: https://issues.apache.org/jira/browse/HADOOP-15482 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HADOOP-15482.001.patch > > > This Jira aims to upgrade jackson-databind to version 2.9.5 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15482) Upgrade jackson-databind to version 2.9.5
[ https://issues.apache.org/jira/browse/HADOOP-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16482844#comment-16482844 ] Jitendra Nath Pandey commented on HADOOP-15482: --- HADOOP-15299 updates the dependency to 2.9.4. Since 3.2 is not released yet, it is better to upgrade to 2.9.5 as well with no additional impact. > Upgrade jackson-databind to version 2.9.5 > - > > Key: HADOOP-15482 > URL: https://issues.apache.org/jira/browse/HADOOP-15482 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HADOOP-15482.001.patch > > > This Jira aims to upgrade jackson-databind to version 2.9.5 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15482) Upgrade jackson-databind to version 2.9.5
[ https://issues.apache.org/jira/browse/HADOOP-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HADOOP-15482: -- Summary: Upgrade jackson-databind to version 2.9.5 (was: Upgrade jackson-databind to version 2.8.11.1) > Upgrade jackson-databind to version 2.9.5 > - > > Key: HADOOP-15482 > URL: https://issues.apache.org/jira/browse/HADOOP-15482 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > > This Jira aims to upgrade jackson-databind to version 2.8.11.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15482) Upgrade jackson-databind to version 2.9.5
[ https://issues.apache.org/jira/browse/HADOOP-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HADOOP-15482: -- Description: This Jira aims to upgrade jackson-databind to version 2.9.5 (was: This Jira aims to upgrade jackson-databind to version 2.8.11.1.) > Upgrade jackson-databind to version 2.9.5 > - > > Key: HADOOP-15482 > URL: https://issues.apache.org/jira/browse/HADOOP-15482 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > > This Jira aims to upgrade jackson-databind to version 2.9.5 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14067) VersionInfo should load version-info.properties from its own classloader
[ https://issues.apache.org/jira/browse/HADOOP-14067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HADOOP-14067: -- Resolution: Fixed Fix Version/s: 3.2.0 Status: Resolved (was: Patch Available) I have committed this to trunk. Thanks [~thejas]! > VersionInfo should load version-info.properties from its own classloader > > > Key: HADOOP-14067 > URL: https://issues.apache.org/jira/browse/HADOOP-14067 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.8.3, 3.0.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-14067.01.patch, HADOOP-14067.01.patch, > HADOOP-14067.02.patch, HADOOP-14067.03.patch > > > org.apache.hadoop.util.VersionInfo loads the version-info.properties file via > the current thread classloader. > However, in case of applications that are using hadoop classes dynamically > (eg jdbc based tools such as SQuirreL SQL) the current thread might not be > the one that loaded the hadoop classes including VersionInfo, and it would > fail to fine the properties file. > The right place to look for the properties file is in the classloader of > VersionInfo class, as right version is the one that is associated with rest > of the loaded hadoop classes, and not necessarily the one in current thread > classloader. > Created a related jira - HADOOP-14066 to make methods to get version via > VersionInfo a public api. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14067) VersionInfo should load version-info.properties from its own classloader
[ https://issues.apache.org/jira/browse/HADOOP-14067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409756#comment-16409756 ] Jitendra Nath Pandey commented on HADOOP-14067: --- +1, Thanks for addressing style/javadoc issues. I will commit shortly. > VersionInfo should load version-info.properties from its own classloader > > > Key: HADOOP-14067 > URL: https://issues.apache.org/jira/browse/HADOOP-14067 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.8.3, 3.0.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Major > Attachments: HADOOP-14067.01.patch, HADOOP-14067.01.patch, > HADOOP-14067.02.patch, HADOOP-14067.03.patch > > > org.apache.hadoop.util.VersionInfo loads the version-info.properties file via > the current thread classloader. > However, in case of applications that are using hadoop classes dynamically > (eg jdbc based tools such as SQuirreL SQL) the current thread might not be > the one that loaded the hadoop classes including VersionInfo, and it would > fail to fine the properties file. > The right place to look for the properties file is in the classloader of > VersionInfo class, as right version is the one that is associated with rest > of the loaded hadoop classes, and not necessarily the one in current thread > classloader. > Created a related jira - HADOOP-14066 to make methods to get version via > VersionInfo a public api. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14067) VersionInfo should load version-info.properties from its own classloader
[ https://issues.apache.org/jira/browse/HADOOP-14067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405520#comment-16405520 ] Jitendra Nath Pandey edited comment on HADOOP-14067 at 3/19/18 11:21 PM: - +1 [~thejas], please review the javadoc, checkstyle issues in the patch. was (Author: jnp): +1 > VersionInfo should load version-info.properties from its own classloader > > > Key: HADOOP-14067 > URL: https://issues.apache.org/jira/browse/HADOOP-14067 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.8.3, 3.0.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Major > Attachments: HADOOP-14067.01.patch, HADOOP-14067.01.patch > > > org.apache.hadoop.util.VersionInfo loads the version-info.properties file via > the current thread classloader. > However, in case of applications that are using hadoop classes dynamically > (eg jdbc based tools such as SQuirreL SQL) the current thread might not be > the one that loaded the hadoop classes including VersionInfo, and it would > fail to fine the properties file. > The right place to look for the properties file is in the classloader of > VersionInfo class, as right version is the one that is associated with rest > of the loaded hadoop classes, and not necessarily the one in current thread > classloader. > Created a related jira - HADOOP-14066 to make methods to get version via > VersionInfo a public api. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14067) VersionInfo should load version-info.properties from its own classloader
[ https://issues.apache.org/jira/browse/HADOOP-14067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405520#comment-16405520 ] Jitendra Nath Pandey commented on HADOOP-14067: --- +1 > VersionInfo should load version-info.properties from its own classloader > > > Key: HADOOP-14067 > URL: https://issues.apache.org/jira/browse/HADOOP-14067 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.8.3, 3.0.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Major > Attachments: HADOOP-14067.01.patch > > > org.apache.hadoop.util.VersionInfo loads the version-info.properties file via > the current thread classloader. > However, in case of applications that are using hadoop classes dynamically > (eg jdbc based tools such as SQuirreL SQL) the current thread might not be > the one that loaded the hadoop classes including VersionInfo, and it would > fail to fine the properties file. > The right place to look for the properties file is in the classloader of > VersionInfo class, as right version is the one that is associated with rest > of the loaded hadoop classes, and not necessarily the one in current thread > classloader. > Created a related jira - HADOOP-14066 to make methods to get version via > VersionInfo a public api. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15255) Upper/Lower case conversion support for group names in LdapGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-15255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey reassigned HADOOP-15255:
-
Assignee: Nanda kumar
> Upper/Lower case conversion support for group names in LdapGroupsMapping
>
>
> Key: HADOOP-15255
> URL: https://issues.apache.org/jira/browse/HADOOP-15255
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HADOOP-15255.000.patch
>
>
> On most LDAP servers the user and group names are case-insensitive. When we
> use {{JniBasedUnixGroupsMappingWithFallback}} and have {{SSSD}} in place, it
> is possible to configure {{SSSD}} to force the group names to be returned in
> lowercase. If we use {{LdapGroupsMapping}}, we don't have any such option.
> This jira proposes to introduce a new {{hadoop.security.group.mapping}}
> implementation based on LdapGroupsMapping which supports force lower/upper
> case group names.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14903) Add json-smart explicitly to pom.xml
[
https://issues.apache.org/jira/browse/HADOOP-14903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373581#comment-16373581
]
Jitendra Nath Pandey commented on HADOOP-14903:
---
Can this be resolved as fixed?
> Add json-smart explicitly to pom.xml
>
>
> Key: HADOOP-14903
> URL: https://issues.apache.org/jira/browse/HADOOP-14903
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.0.0-beta1
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>Priority: Major
> Fix For: 3.0.0-beta1, 2.10.0, 2.9.1, 2.8.4
>
> Attachments: HADOOP-14903-003-branch-2.patch,
> HADOOP-14903-branch-2-003.patch,
> HADOOP-14903-branch-2-004-ForExecutingTests.patch,
> HADOOP-14903-branch-2-004.patch, HADOOP-14903.001.patch,
> HADOOP-14903.002.patch, HADOOP-14903.003.patch
>
>
> With the library update in HADOOP-14799, maven knows how to pull in
> net.minidev:json-smart for tests, but not for packaging. This needs to be
> added to the main project pom in order to avoid this warning:
> {noformat}
> [WARNING] The POM for net.minidev:json-smart:jar:2.3-SNAPSHOT is missing, no
> dependency information available
> {noformat}
> This is pulled in from a few places:
> {noformat}
> [INFO] | +- org.apache.hadoop:hadoop-auth:jar:3.1.0-SNAPSHOT:compile
> [INFO] | | +- com.nimbusds:nimbus-jose-jwt:jar:4.41.1:compile
> [INFO] | | | +- com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
> [INFO] | | | \- net.minidev:json-smart:jar:2.3:compile
> [INFO] | | \- org.apache.kerby:token-provider:jar:1.0.1:compile
> [INFO] | | \- com.nimbusds:nimbus-jose-jwt:jar:4.41.1:compile
> [INFO] | |+-
> com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
> [INFO] | |\- net.minidev:json-smart:jar:2.3:compile
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15171) Hadoop native ZLIB decompressor produces 0 bytes for some input
[
https://issues.apache.org/jira/browse/HADOOP-15171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey reassigned HADOOP-15171:
-
Assignee: Lokesh Jain
> Hadoop native ZLIB decompressor produces 0 bytes for some input
> ---
>
> Key: HADOOP-15171
> URL: https://issues.apache.org/jira/browse/HADOOP-15171
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Sergey Shelukhin
>Assignee: Lokesh Jain
>Priority: Blocker
> Fix For: 3.1.0, 3.0.1
>
>
> While reading some ORC file via direct buffers, Hive gets a 0-sized buffer
> for a particular compressed segment of the file. We narrowed it down to
> Hadoop native ZLIB codec; when the data is copied to heap-based buffer and
> the JDK Inflater is used, it produces correct output. Input is only 127 bytes
> so I can paste it here.
> All the other (many) blocks of the file are decompressed without problems by
> the same code.
> {noformat}
> 2018-01-13T02:47:40,815 TRACE [IO-Elevator-Thread-0
> (1515637158315_0079_1_00_00_0)] encoded.EncodedReaderImpl: Decompressing
> 127 bytes to dest buffer pos 524288, limit 786432
> 2018-01-13T02:47:40,816 WARN [IO-Elevator-Thread-0
> (1515637158315_0079_1_00_00_0)] encoded.EncodedReaderImpl: The codec has
> produced 0 bytes for 127 bytes at pos 0, data hash 1719565039: [e3 92 e1 62
> 66 60 60 10 12 e5 98 e0 27 c4 c7 f1 e8 12 8f 40 c3 7b 5e 89 09 7f 6e 74 73 04
> 30 70 c9 72 b1 30 14 4d 60 82 49 37 bd e7 15 58 d0 cd 2f 31 a1 a1 e3 35 4c fa
> 15 a3 02 4c 7a 51 37 bf c0 81 e5 02 12 13 5a b6 9f e2 04 ea 96 e3 62 65 b8 c3
> b4 01 ae fd d0 72 01 81 07 87 05 25 26 74 3c 5b c9 05 35 fd 0a b3 03 50 7b 83
> 11 c8 f2 c3 82 02 0f 96 0b 49 34 7c fa ff 9f 2d 80 01 00
> 2018-01-13T02:47:40,816 WARN [IO-Elevator-Thread-0
> (1515637158315_0079_1_00_00_0)] encoded.EncodedReaderImpl: Fell back to
> JDK decompressor with memcopy; got 155 bytes
> {noformat}
> Hadoop version is based on 3.1 snapshot.
> The size of libhadoop.so is 824403 bytes, and libgplcompression is 78273
> FWIW. Not sure how to extract versions from those.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13786) Add S3A committer for zero-rename commits to S3 endpoints
[ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341876#comment-16341876 ] Jitendra Nath Pandey commented on HADOOP-13786: --- [~ste...@apache.org], [~rdblue], it seems some of the files in Netflix S3Committer are not Apache Licensed. for example [https://github.com/rdblue/s3committer/blob/master/src/main/java/com/netflix/bdp/s3/Tasks.java|https://github.com/rdblue/s3committer/blob/master/src/main/java/com/netflix/bdp/s3/Tasks.java.] The home page of the project [https://github.com/rdblue/s3committer,] also doesn't explicitly state a license. This could be just an oversight, but given that the above file is under Netflix copyright, copying any code from that file into Apache may need explicit permission. > Add S3A committer for zero-rename commits to S3 endpoints > - > > Key: HADOOP-13786 > URL: https://issues.apache.org/jira/browse/HADOOP-13786 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.1.0 > > Attachments: HADOOP-13786-036.patch, HADOOP-13786-037.patch, > HADOOP-13786-038.patch, HADOOP-13786-039.patch, > HADOOP-13786-HADOOP-13345-001.patch, HADOOP-13786-HADOOP-13345-002.patch, > HADOOP-13786-HADOOP-13345-003.patch, HADOOP-13786-HADOOP-13345-004.patch, > HADOOP-13786-HADOOP-13345-005.patch, HADOOP-13786-HADOOP-13345-006.patch, > HADOOP-13786-HADOOP-13345-006.patch, HADOOP-13786-HADOOP-13345-007.patch, > HADOOP-13786-HADOOP-13345-009.patch, HADOOP-13786-HADOOP-13345-010.patch, > HADOOP-13786-HADOOP-13345-011.patch, HADOOP-13786-HADOOP-13345-012.patch, > HADOOP-13786-HADOOP-13345-013.patch, HADOOP-13786-HADOOP-13345-015.patch, > HADOOP-13786-HADOOP-13345-016.patch, HADOOP-13786-HADOOP-13345-017.patch, > HADOOP-13786-HADOOP-13345-018.patch, HADOOP-13786-HADOOP-13345-019.patch, > HADOOP-13786-HADOOP-13345-020.patch, HADOOP-13786-HADOOP-13345-021.patch, > HADOOP-13786-HADOOP-13345-022.patch, HADOOP-13786-HADOOP-13345-023.patch, > HADOOP-13786-HADOOP-13345-024.patch, HADOOP-13786-HADOOP-13345-025.patch, > HADOOP-13786-HADOOP-13345-026.patch, HADOOP-13786-HADOOP-13345-027.patch, > HADOOP-13786-HADOOP-13345-028.patch, HADOOP-13786-HADOOP-13345-028.patch, > HADOOP-13786-HADOOP-13345-029.patch, HADOOP-13786-HADOOP-13345-030.patch, > HADOOP-13786-HADOOP-13345-031.patch, HADOOP-13786-HADOOP-13345-032.patch, > HADOOP-13786-HADOOP-13345-033.patch, HADOOP-13786-HADOOP-13345-035.patch, > MAPREDUCE-6823-003.patch, cloud-intergration-test-failure.log, > objectstore.pdf, s3committer-master.zip > > > A goal of this code is "support O(1) commits to S3 repositories in the > presence of failures". Implement it, including whatever is needed to > demonstrate the correctness of the algorithm. (that is, assuming that s3guard > provides a consistent view of the presence/absence of blobs, show that we can > commit directly). > I consider ourselves free to expose the blobstore-ness of the s3 output > streams (ie. not visible until the close()), if we need to use that to allow > us to abort commit operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15143) NPE due to Invalid KerberosTicket in UGI
[
https://issues.apache.org/jira/browse/HADOOP-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307708#comment-16307708
]
Jitendra Nath Pandey commented on HADOOP-15143:
---
Committed to branch-2 as well. Thanks to [~msingh].
> NPE due to Invalid KerberosTicket in UGI
>
>
> Key: HADOOP-15143
> URL: https://issues.apache.org/jira/browse/HADOOP-15143
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.4
>Reporter: Jitendra Nath Pandey
>Assignee: Mukul Kumar Singh
> Fix For: 3.1.0, 2.9.1
>
> Attachments: HADOOP-15143-branch-2.001.patch, HADOOP-15143.001.patch,
> HADOOP-15143.002.patch
>
>
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.security.UserGroupInformation.fixKerberosTicketOrder(UserGroupInformation.java:1170)
>
> at
> org.apache.hadoop.security.UserGroupInformation.reloginFromKeytab(UserGroupInformation.java:1247)
>
> at
> org.apache.hadoop.security.UserGroupInformation.checkTGTAndReloginFromKeytab(UserGroupInformation.java:1157)
>
> {code}
> It could be related to jdk issue
> http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/rev/fd0e0898721c
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15143) NPE due to Invalid KerberosTicket in UGI
[
https://issues.apache.org/jira/browse/HADOOP-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HADOOP-15143:
--
Resolution: Fixed
Fix Version/s: 2.9.1
Status: Resolved (was: Patch Available)
> NPE due to Invalid KerberosTicket in UGI
>
>
> Key: HADOOP-15143
> URL: https://issues.apache.org/jira/browse/HADOOP-15143
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.4
>Reporter: Jitendra Nath Pandey
>Assignee: Mukul Kumar Singh
> Fix For: 3.1.0, 2.9.1
>
> Attachments: HADOOP-15143-branch-2.001.patch, HADOOP-15143.001.patch,
> HADOOP-15143.002.patch
>
>
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.security.UserGroupInformation.fixKerberosTicketOrder(UserGroupInformation.java:1170)
>
> at
> org.apache.hadoop.security.UserGroupInformation.reloginFromKeytab(UserGroupInformation.java:1247)
>
> at
> org.apache.hadoop.security.UserGroupInformation.checkTGTAndReloginFromKeytab(UserGroupInformation.java:1157)
>
> {code}
> It could be related to jdk issue
> http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/rev/fd0e0898721c
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15143) NPE due to Invalid KerberosTicket in UGI
[
https://issues.apache.org/jira/browse/HADOOP-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305127#comment-16305127
]
Jitendra Nath Pandey commented on HADOOP-15143:
---
I have committed this to trunk, however keeping the bug open to get it
committed to branch-2.
[~msingh], could you please provide a branch-2 patch?
> NPE due to Invalid KerberosTicket in UGI
>
>
> Key: HADOOP-15143
> URL: https://issues.apache.org/jira/browse/HADOOP-15143
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.4
>Reporter: Jitendra Nath Pandey
>Assignee: Mukul Kumar Singh
> Fix For: 3.1.0
>
> Attachments: HADOOP-15143.001.patch, HADOOP-15143.002.patch
>
>
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.security.UserGroupInformation.fixKerberosTicketOrder(UserGroupInformation.java:1170)
>
> at
> org.apache.hadoop.security.UserGroupInformation.reloginFromKeytab(UserGroupInformation.java:1247)
>
> at
> org.apache.hadoop.security.UserGroupInformation.checkTGTAndReloginFromKeytab(UserGroupInformation.java:1157)
>
> {code}
> It could be related to jdk issue
> http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/rev/fd0e0898721c
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15143) NPE due to Invalid KerberosTicket in UGI
[
https://issues.apache.org/jira/browse/HADOOP-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HADOOP-15143:
--
Fix Version/s: 3.1.0
> NPE due to Invalid KerberosTicket in UGI
>
>
> Key: HADOOP-15143
> URL: https://issues.apache.org/jira/browse/HADOOP-15143
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.4
>Reporter: Jitendra Nath Pandey
>Assignee: Mukul Kumar Singh
> Fix For: 3.1.0
>
> Attachments: HADOOP-15143.001.patch, HADOOP-15143.002.patch
>
>
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.security.UserGroupInformation.fixKerberosTicketOrder(UserGroupInformation.java:1170)
>
> at
> org.apache.hadoop.security.UserGroupInformation.reloginFromKeytab(UserGroupInformation.java:1247)
>
> at
> org.apache.hadoop.security.UserGroupInformation.checkTGTAndReloginFromKeytab(UserGroupInformation.java:1157)
>
> {code}
> It could be related to jdk issue
> http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/rev/fd0e0898721c
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15143) NPE due to Invalid KerberosTicket in UGI
[
https://issues.apache.org/jira/browse/HADOOP-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HADOOP-15143:
--
Target Version/s: 2.9.0
> NPE due to Invalid KerberosTicket in UGI
>
>
> Key: HADOOP-15143
> URL: https://issues.apache.org/jira/browse/HADOOP-15143
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.4
>Reporter: Jitendra Nath Pandey
>Assignee: Mukul Kumar Singh
> Fix For: 3.1.0
>
> Attachments: HADOOP-15143.001.patch, HADOOP-15143.002.patch
>
>
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.security.UserGroupInformation.fixKerberosTicketOrder(UserGroupInformation.java:1170)
>
> at
> org.apache.hadoop.security.UserGroupInformation.reloginFromKeytab(UserGroupInformation.java:1247)
>
> at
> org.apache.hadoop.security.UserGroupInformation.checkTGTAndReloginFromKeytab(UserGroupInformation.java:1157)
>
> {code}
> It could be related to jdk issue
> http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/rev/fd0e0898721c
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15143) NPE due to Invalid KerberosTicket in UGI
[
https://issues.apache.org/jira/browse/HADOOP-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305091#comment-16305091
]
Jitendra Nath Pandey commented on HADOOP-15143:
---
+1 for the latest patch.
> NPE due to Invalid KerberosTicket in UGI
>
>
> Key: HADOOP-15143
> URL: https://issues.apache.org/jira/browse/HADOOP-15143
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.4
>Reporter: Jitendra Nath Pandey
>Assignee: Mukul Kumar Singh
> Attachments: HADOOP-15143.001.patch, HADOOP-15143.002.patch
>
>
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.security.UserGroupInformation.fixKerberosTicketOrder(UserGroupInformation.java:1170)
>
> at
> org.apache.hadoop.security.UserGroupInformation.reloginFromKeytab(UserGroupInformation.java:1247)
>
> at
> org.apache.hadoop.security.UserGroupInformation.checkTGTAndReloginFromKeytab(UserGroupInformation.java:1157)
>
> {code}
> It could be related to jdk issue
> http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/rev/fd0e0898721c
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15143) NPE due to Invalid KerberosTicket in UGI
[
https://issues.apache.org/jira/browse/HADOOP-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16304724#comment-16304724
]
Jitendra Nath Pandey commented on HADOOP-15143:
---
[~msingh]
On a second look at the KerberosTicket code, the destroy method is public and
is not synchronized, and therefore it is possible that server is set to null
but destroyed flag is not set. We may get a bit conservative and check for
getServer() returning null as well?
> NPE due to Invalid KerberosTicket in UGI
>
>
> Key: HADOOP-15143
> URL: https://issues.apache.org/jira/browse/HADOOP-15143
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.4
>Reporter: Jitendra Nath Pandey
>Assignee: Mukul Kumar Singh
> Attachments: HADOOP-15143.001.patch
>
>
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.security.UserGroupInformation.fixKerberosTicketOrder(UserGroupInformation.java:1170)
>
> at
> org.apache.hadoop.security.UserGroupInformation.reloginFromKeytab(UserGroupInformation.java:1247)
>
> at
> org.apache.hadoop.security.UserGroupInformation.checkTGTAndReloginFromKeytab(UserGroupInformation.java:1157)
>
> {code}
> It could be related to jdk issue
> http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/rev/fd0e0898721c
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15143) NPE due to Invalid KerberosTicket in UGI
[
https://issues.apache.org/jira/browse/HADOOP-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16304715#comment-16304715
]
Jitendra Nath Pandey commented on HADOOP-15143:
---
+1 for the patch. I will commit it today.
> NPE due to Invalid KerberosTicket in UGI
>
>
> Key: HADOOP-15143
> URL: https://issues.apache.org/jira/browse/HADOOP-15143
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.4
>Reporter: Jitendra Nath Pandey
>Assignee: Mukul Kumar Singh
> Attachments: HADOOP-15143.001.patch
>
>
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.security.UserGroupInformation.fixKerberosTicketOrder(UserGroupInformation.java:1170)
>
> at
> org.apache.hadoop.security.UserGroupInformation.reloginFromKeytab(UserGroupInformation.java:1247)
>
> at
> org.apache.hadoop.security.UserGroupInformation.checkTGTAndReloginFromKeytab(UserGroupInformation.java:1157)
>
> {code}
> It could be related to jdk issue
> http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/rev/fd0e0898721c
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15143) NPE due to Invalid KerberosTicket in UGI
[
https://issues.apache.org/jira/browse/HADOOP-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HADOOP-15143:
--
Description:
{code}
java.lang.NullPointerException
at
org.apache.hadoop.security.UserGroupInformation.fixKerberosTicketOrder(UserGroupInformation.java:1170)
at
org.apache.hadoop.security.UserGroupInformation.reloginFromKeytab(UserGroupInformation.java:1247)
at
org.apache.hadoop.security.UserGroupInformation.checkTGTAndReloginFromKeytab(UserGroupInformation.java:1157)
{code}
It could be related to jdk issue
http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/rev/fd0e0898721c
was:
{code}
java.lang.NullPointerException
at
org.apache.hadoop.security.UserGroupInformation.fixKerberosTicketOrder(UserGroupInformation.java:1170)
at
org.apache.hadoop.security.UserGroupInformation.reloginFromKeytab(UserGroupInformation.java:1247)
at
org.apache.hadoop.security.UserGroupInformation.checkTGTAndReloginFromKeytab(UserGroupInformation.java:1157)
{code}
> NPE due to Invalid KerberosTicket in UGI
>
>
> Key: HADOOP-15143
> URL: https://issues.apache.org/jira/browse/HADOOP-15143
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.4
>Reporter: Jitendra Nath Pandey
>Assignee: Mukul Kumar Singh
>
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.security.UserGroupInformation.fixKerberosTicketOrder(UserGroupInformation.java:1170)
>
> at
> org.apache.hadoop.security.UserGroupInformation.reloginFromKeytab(UserGroupInformation.java:1247)
>
> at
> org.apache.hadoop.security.UserGroupInformation.checkTGTAndReloginFromKeytab(UserGroupInformation.java:1157)
>
> {code}
> It could be related to jdk issue
> http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/rev/fd0e0898721c
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15143) NPE due to Invalid KerberosTicket in UGI
Jitendra Nath Pandey created HADOOP-15143:
-
Summary: NPE due to Invalid KerberosTicket in UGI
Key: HADOOP-15143
URL: https://issues.apache.org/jira/browse/HADOOP-15143
Project: Hadoop Common
Issue Type: Bug
Reporter: Jitendra Nath Pandey
Assignee: Mukul Kumar Singh
{code}
java.lang.NullPointerException
at
org.apache.hadoop.security.UserGroupInformation.fixKerberosTicketOrder(UserGroupInformation.java:1170)
at
org.apache.hadoop.security.UserGroupInformation.reloginFromKeytab(UserGroupInformation.java:1247)
at
org.apache.hadoop.security.UserGroupInformation.checkTGTAndReloginFromKeytab(UserGroupInformation.java:1157)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15143) NPE due to Invalid KerberosTicket in UGI
[
https://issues.apache.org/jira/browse/HADOOP-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HADOOP-15143:
--
Affects Version/s: 2.7.4
> NPE due to Invalid KerberosTicket in UGI
>
>
> Key: HADOOP-15143
> URL: https://issues.apache.org/jira/browse/HADOOP-15143
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.4
>Reporter: Jitendra Nath Pandey
>Assignee: Mukul Kumar Singh
>
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.security.UserGroupInformation.fixKerberosTicketOrder(UserGroupInformation.java:1170)
>
> at
> org.apache.hadoop.security.UserGroupInformation.reloginFromKeytab(UserGroupInformation.java:1247)
>
> at
> org.apache.hadoop.security.UserGroupInformation.checkTGTAndReloginFromKeytab(UserGroupInformation.java:1157)
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15092) Proxy failures during NamenodeWebHdfsMethods are not logged
[
https://issues.apache.org/jira/browse/HADOOP-15092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey reassigned HADOOP-15092:
-
Assignee: Prabhu Joseph
> Proxy failures during NamenodeWebHdfsMethods are not logged
>
>
> Key: HADOOP-15092
> URL: https://issues.apache.org/jira/browse/HADOOP-15092
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.7.3
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>
> When GetDelegationToken http request fails with proxy issue, there is no logs
> from NameNode to indicate it's a proxy issue. Below is the only log.
> IOExceptions are logged but AuthorizationException are not logged. It will be
> helpful to log proxy failures from JspHelper getUGI().
> {code}
> 2017-12-05 13:05:02,045 INFO namenode.GetDelegationTokenServlet
> (GetDelegationTokenServlet.java:doGet(56)) - Request for token received with
> no authentication from 172.26.93.73
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14699) Impersonation errors with UGI after second principal relogin
[
https://issues.apache.org/jira/browse/HADOOP-14699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238448#comment-16238448
]
Jitendra Nath Pandey commented on HADOOP-14699:
---
[~jtstorck], How is the second UGI created? The impersonation will happen only
if a real user is set into the subject of this UGI. There shouldn't be a real
user unless this second UGI was created as a proxy user.
> Impersonation errors with UGI after second principal relogin
>
>
> Key: HADOOP-14699
> URL: https://issues.apache.org/jira/browse/HADOOP-14699
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.6.2, 2.7.3, 2.8.1
>Reporter: Jeff Storck
>Priority: Major
>
> Multiple principals that are logged in using UGI instances that are
> instantiated from a UGI class loaded by the same classloader will encounter
> problems when the second principal attempts to relogin and perform an action
> using a UGI.doAs(). An impersonation will occur and the operation attempted
> by the second principal after relogging in will fail. There should not be an
> implicit attempt to impersonate the second principal through the first
> principal that logged in.
> I have created a GitHub project that exhibits the impersonation error with
> brief instructions on how to set up for the test and run it:
> https://github.com/jtstorck/kerberos-examples/tree/master/hadoop/ugi-test
> {noformat}18:44:55.687 [pool-2-thread-2] WARN
> h.u.u.ugirunnable.ugite...@example.com - Unexpected exception while
> performing task for [ugite...@example.com (auth:KERBEROS)]
> org.apache.hadoop.ipc.RemoteException: User: ugite...@example.com is not
> allowed to impersonate ugite...@example.com
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1481)
> at org.apache.hadoop.ipc.Client.call(Client.java:1427)
> at org.apache.hadoop.ipc.Client.call(Client.java:1337)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:787)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335)
> at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1700)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1436)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1433)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1448)
> at
> hadoop.ugitest.UgiTestMain$UgiRunnable.lambda$run$2(UgiTestMain.java:194)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> at hadoop.ugitest.UgiTestMain$UgiRunnable.run(UgiTestMain.java:194)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745){noformat}
[jira] [Updated] (HADOOP-14910) Upgrade netty-all jar to latest 4.0.x.Final
[ https://issues.apache.org/jira/browse/HADOOP-14910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HADOOP-14910: -- Resolution: Fixed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Available) > Upgrade netty-all jar to latest 4.0.x.Final > --- > > Key: HADOOP-14910 > URL: https://issues.apache.org/jira/browse/HADOOP-14910 > Project: Hadoop Common > Issue Type: Bug >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Critical > Fix For: 3.1.0 > > Attachments: HADOOP-14910-01.patch, HADOOP-14910-02.patch, > HADOOP-14910-03.patch > > > Upgrade netty-all jar to 4.0.37.Final version to fix latest vulnerabilities > reported. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14910) Upgrade netty-all jar to latest 4.0.x.Final
[ https://issues.apache.org/jira/browse/HADOOP-14910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199114#comment-16199114 ] Jitendra Nath Pandey commented on HADOOP-14910: --- I have committed this to trunk. Thanks [~vinayrpet]. > Upgrade netty-all jar to latest 4.0.x.Final > --- > > Key: HADOOP-14910 > URL: https://issues.apache.org/jira/browse/HADOOP-14910 > Project: Hadoop Common > Issue Type: Bug >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Critical > Fix For: 3.1.0 > > Attachments: HADOOP-14910-01.patch, HADOOP-14910-02.patch, > HADOOP-14910-03.patch > > > Upgrade netty-all jar to 4.0.37.Final version to fix latest vulnerabilities > reported. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14910) Upgrade netty-all jar to latest 4.0.x.Final
[ https://issues.apache.org/jira/browse/HADOOP-14910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HADOOP-14910: -- Attachment: HADOOP-14910-03.patch Updated patch that removes the dummy change. > Upgrade netty-all jar to latest 4.0.x.Final > --- > > Key: HADOOP-14910 > URL: https://issues.apache.org/jira/browse/HADOOP-14910 > Project: Hadoop Common > Issue Type: Bug >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Critical > Attachments: HADOOP-14910-01.patch, HADOOP-14910-02.patch, > HADOOP-14910-03.patch > > > Upgrade netty-all jar to 4.0.37.Final version to fix latest vulnerabilities > reported. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14910) Upgrade netty-all jar to latest 4.0.x.Final
[ https://issues.apache.org/jira/browse/HADOOP-14910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193762#comment-16193762 ] Jitendra Nath Pandey commented on HADOOP-14910: --- I will commit this tomorrow, if there are no objections. > Upgrade netty-all jar to latest 4.0.x.Final > --- > > Key: HADOOP-14910 > URL: https://issues.apache.org/jira/browse/HADOOP-14910 > Project: Hadoop Common > Issue Type: Bug >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Critical > Attachments: HADOOP-14910-01.patch, HADOOP-14910-02.patch > > > Upgrade netty-all jar to 4.0.37.Final version to fix latest vulnerabilities > reported. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14910) Upgrade netty-all jar to latest 4.0.x.Final
[ https://issues.apache.org/jira/browse/HADOOP-14910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192045#comment-16192045 ] Jitendra Nath Pandey commented on HADOOP-14910: --- I am fine with cleanup up decodeComponent in a subsequent jira. +1 for the patch. > Upgrade netty-all jar to latest 4.0.x.Final > --- > > Key: HADOOP-14910 > URL: https://issues.apache.org/jira/browse/HADOOP-14910 > Project: Hadoop Common > Issue Type: Bug >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Critical > Attachments: HADOOP-14910-01.patch, HADOOP-14910-02.patch > > > Upgrade netty-all jar to 4.0.37.Final version to fix latest vulnerabilities > reported. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14910) Upgrade netty-all jar to latest 4.0.x.Final
[ https://issues.apache.org/jira/browse/HADOOP-14910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188993#comment-16188993 ] Jitendra Nath Pandey commented on HADOOP-14910: --- Thanks for the patch [~vinayrpet]. bq. Basically reverted changes of HDFS-6662 and HDFS-7816 which seems to be fixed automatically in netty. Is this change related to netty-all version upgrade? > Upgrade netty-all jar to latest 4.0.x.Final > --- > > Key: HADOOP-14910 > URL: https://issues.apache.org/jira/browse/HADOOP-14910 > Project: Hadoop Common > Issue Type: Bug >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Critical > Attachments: HADOOP-14910-01.patch, HADOOP-14910-02.patch > > > Upgrade netty-all jar to 4.0.37.Final version to fix latest vulnerabilities > reported. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13500) Concurrency issues when using Configuration iterator
[ https://issues.apache.org/jira/browse/HADOOP-13500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183499#comment-16183499 ] Jitendra Nath Pandey commented on HADOOP-13500: --- [~lmccay] had a patch for this, IIRC. > Concurrency issues when using Configuration iterator > > > Key: HADOOP-13500 > URL: https://issues.apache.org/jira/browse/HADOOP-13500 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Reporter: Jason Lowe >Assignee: Ajay Kumar > > It is possible to encounter a ConcurrentModificationException while trying to > iterate a Configuration object. The iterator method tries to walk the > underlying Property object without proper synchronization, so another thread > simultaneously calling the set method can trigger it. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[ https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170858#comment-16170858 ] Jitendra Nath Pandey commented on HADOOP-14520: --- Fix version 2.0.6-alpha didn't look right, changed it to 2.9. [~steve_l] > WASB: Block compaction for Azure Block Blobs > > > Key: HADOOP-14520 > URL: https://issues.apache.org/jira/browse/HADOOP-14520 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 2.7.4 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HADOOP-14520-006.patch, HADOOP-14520-008.patch, > HADOOP-14520-009.patch, HADOOP-14520-05.patch, HADOOP_14520_07.patch, > HADOOP_14520_08.patch, HADOOP_14520_09.patch, HADOOP_14520_10.patch, > hadoop-14520-branch-2-010.patch, HADOOP-14520-patch-07-08.diff, > HADOOP-14520-patch-07-09.diff > > > Block Compaction for WASB allows uploading new blocks for every hflush/hsync > call. When the number of blocks is above 32000, next hflush/hsync triggers > the block compaction process. Block compaction replaces a sequence of blocks > with one block. From all the sequences with total length less than 4M, > compaction chooses the longest one. It is a greedy algorithm that preserve > all potential candidates for the next round. Block Compaction for WASB > increases data durability and allows using block blobs instead of page blobs. > By default, block compaction is disabled. Similar to the configuration for > page blobs, the client needs to specify HDFS folders where block compaction > over block blobs is enabled. > Results for HADOOP_14520_07.patch > tested endpoint: fs.azure.account.key.hdfs4.blob.core.windows.net > Tests run: 777, Failures: 0, Errors: 0, Skipped: 155 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14520) WASB: Block compaction for Azure Block Blobs
[ https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HADOOP-14520: -- Fix Version/s: (was: 2.0.6-alpha) 2.9.0 > WASB: Block compaction for Azure Block Blobs > > > Key: HADOOP-14520 > URL: https://issues.apache.org/jira/browse/HADOOP-14520 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 2.7.4 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HADOOP-14520-006.patch, HADOOP-14520-008.patch, > HADOOP-14520-009.patch, HADOOP-14520-05.patch, HADOOP_14520_07.patch, > HADOOP_14520_08.patch, HADOOP_14520_09.patch, HADOOP_14520_10.patch, > hadoop-14520-branch-2-010.patch, HADOOP-14520-patch-07-08.diff, > HADOOP-14520-patch-07-09.diff > > > Block Compaction for WASB allows uploading new blocks for every hflush/hsync > call. When the number of blocks is above 32000, next hflush/hsync triggers > the block compaction process. Block compaction replaces a sequence of blocks > with one block. From all the sequences with total length less than 4M, > compaction chooses the longest one. It is a greedy algorithm that preserve > all potential candidates for the next round. Block Compaction for WASB > increases data durability and allows using block blobs instead of page blobs. > By default, block compaction is disabled. Similar to the configuration for > page blobs, the client needs to specify HDFS folders where block compaction > over block blobs is enabled. > Results for HADOOP_14520_07.patch > tested endpoint: fs.azure.account.key.hdfs4.blob.core.windows.net > Tests run: 777, Failures: 0, Errors: 0, Skipped: 155 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14674) Correct javadoc for getRandomizedTempPath
[
https://issues.apache.org/jira/browse/HADOOP-14674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HADOOP-14674:
--
Fix Version/s: 3.0.0-beta1
> Correct javadoc for getRandomizedTempPath
> -
>
> Key: HADOOP-14674
> URL: https://issues.apache.org/jira/browse/HADOOP-14674
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14674.001.patch, HADOOP-14674.002.patch
>
>
> getRandomizedTempPath has incorrect javadoc where the javadoc specifies a
> parameter to the function however the function doesnt expects one.
> {code}
> /**
>* Get a temp path. This may or may not be relative; it depends on what the
>* {@link #SYSPROP_TEST_DATA_DIR} is set to. If unset, it returns a path
>* under the relative path {@link #DEFAULT_TEST_DATA_PATH}
>* @param subpath sub path, with no leading "/" character
>* @return a string to use in paths
>*/
> public static String getRandomizedTempPath() {
> return getTempPath(RandomStringUtils.randomAlphanumeric(10));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14674) Correct javadoc for getRandomizedTempPath
[
https://issues.apache.org/jira/browse/HADOOP-14674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HADOOP-14674:
--
Status: Patch Available (was: Reopened)
> Correct javadoc for getRandomizedTempPath
> -
>
> Key: HADOOP-14674
> URL: https://issues.apache.org/jira/browse/HADOOP-14674
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HADOOP-14674.001.patch
>
>
> getRandomizedTempPath has incorrect javadoc where the javadoc specifies a
> parameter to the function however the function doesnt expects one.
> {code}
> /**
>* Get a temp path. This may or may not be relative; it depends on what the
>* {@link #SYSPROP_TEST_DATA_DIR} is set to. If unset, it returns a path
>* under the relative path {@link #DEFAULT_TEST_DATA_PATH}
>* @param subpath sub path, with no leading "/" character
>* @return a string to use in paths
>*/
> public static String getRandomizedTempPath() {
> return getTempPath(RandomStringUtils.randomAlphanumeric(10));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14674) Correct javadoc for getRandomizedTempPath
[
https://issues.apache.org/jira/browse/HADOOP-14674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150031#comment-16150031
]
Jitendra Nath Pandey commented on HADOOP-14674:
---
Thanks for catching this [~andrew.wang]. I am not entirely sure what happened,
most likely I forgot to git push. Sorry about that.
Before committing, I will wait for another jenkins run, since patch is now more
than a month old.
> Correct javadoc for getRandomizedTempPath
> -
>
> Key: HADOOP-14674
> URL: https://issues.apache.org/jira/browse/HADOOP-14674
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HADOOP-14674.001.patch
>
>
> getRandomizedTempPath has incorrect javadoc where the javadoc specifies a
> parameter to the function however the function doesnt expects one.
> {code}
> /**
>* Get a temp path. This may or may not be relative; it depends on what the
>* {@link #SYSPROP_TEST_DATA_DIR} is set to. If unset, it returns a path
>* under the relative path {@link #DEFAULT_TEST_DATA_PATH}
>* @param subpath sub path, with no leading "/" character
>* @return a string to use in paths
>*/
> public static String getRandomizedTempPath() {
> return getTempPath(RandomStringUtils.randomAlphanumeric(10));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14680) Azure: IndexOutOfBoundsException in BlockBlobInputStream
[
https://issues.apache.org/jira/browse/HADOOP-14680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HADOOP-14680:
--
Fix Version/s: 3.0.0-beta1
> Azure: IndexOutOfBoundsException in BlockBlobInputStream
>
>
> Key: HADOOP-14680
> URL: https://issues.apache.org/jira/browse/HADOOP-14680
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Rajesh Balamohan
>Assignee: Thomas Marquardt
>Priority: Minor
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14680-001.patch, HADOOP-14680-branch-2.01.patch
>
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/BlockBlobInputStream.java#L361
> On certain conditions, BlockBlobInputStream can throw
> IndexOutOfBoundsException. Following is an example
> {{length:297898, offset:4194304, buf.len:4492202, writePos:4194304}} :
> In this case, {{MemoryOutputStream::capacity()}} would end up returning
> negative value and can cause {{IndexOutOfBoundsException}}
> It should be {{return buffer.length - offset;}} to determine current capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14680) Azure: IndexOutOfBoundsException in BlockBlobInputStream
[
https://issues.apache.org/jira/browse/HADOOP-14680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100950#comment-16100950
]
Jitendra Nath Pandey edited comment on HADOOP-14680 at 7/25/17 11:40 PM:
-
I have committed this to trunk. Attached branch-2 patch for jenkins.
was (Author: jnp):
Attaching branch-2 patch for jenkins.
> Azure: IndexOutOfBoundsException in BlockBlobInputStream
>
>
> Key: HADOOP-14680
> URL: https://issues.apache.org/jira/browse/HADOOP-14680
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Rajesh Balamohan
>Assignee: Thomas Marquardt
>Priority: Minor
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14680-001.patch, HADOOP-14680-branch-2.01.patch
>
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/BlockBlobInputStream.java#L361
> On certain conditions, BlockBlobInputStream can throw
> IndexOutOfBoundsException. Following is an example
> {{length:297898, offset:4194304, buf.len:4492202, writePos:4194304}} :
> In this case, {{MemoryOutputStream::capacity()}} would end up returning
> negative value and can cause {{IndexOutOfBoundsException}}
> It should be {{return buffer.length - offset;}} to determine current capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14680) Azure: IndexOutOfBoundsException in BlockBlobInputStream
[
https://issues.apache.org/jira/browse/HADOOP-14680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HADOOP-14680:
--
Attachment: HADOOP-14680-branch-2.01.patch
Attaching branch-2 patch for jenkins.
> Azure: IndexOutOfBoundsException in BlockBlobInputStream
>
>
> Key: HADOOP-14680
> URL: https://issues.apache.org/jira/browse/HADOOP-14680
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Rajesh Balamohan
>Assignee: Thomas Marquardt
>Priority: Minor
> Attachments: HADOOP-14680-001.patch, HADOOP-14680-branch-2.01.patch
>
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/BlockBlobInputStream.java#L361
> On certain conditions, BlockBlobInputStream can throw
> IndexOutOfBoundsException. Following is an example
> {{length:297898, offset:4194304, buf.len:4492202, writePos:4194304}} :
> In this case, {{MemoryOutputStream::capacity()}} would end up returning
> negative value and can cause {{IndexOutOfBoundsException}}
> It should be {{return buffer.length - offset;}} to determine current capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14680) Azure: IndexOutOfBoundsException in BlockBlobInputStream
[
https://issues.apache.org/jira/browse/HADOOP-14680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100944#comment-16100944
]
Jitendra Nath Pandey commented on HADOOP-14680:
---
+1. I will commit the patch shortly.
> Azure: IndexOutOfBoundsException in BlockBlobInputStream
>
>
> Key: HADOOP-14680
> URL: https://issues.apache.org/jira/browse/HADOOP-14680
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Rajesh Balamohan
>Assignee: Thomas Marquardt
>Priority: Minor
> Attachments: HADOOP-14680-001.patch
>
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/BlockBlobInputStream.java#L361
> On certain conditions, BlockBlobInputStream can throw
> IndexOutOfBoundsException. Following is an example
> {{length:297898, offset:4194304, buf.len:4492202, writePos:4194304}} :
> In this case, {{MemoryOutputStream::capacity()}} would end up returning
> negative value and can cause {{IndexOutOfBoundsException}}
> It should be {{return buffer.length - offset;}} to determine current capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14518) Customize User-Agent header sent in HTTP/HTTPS requests by WASB.
[ https://issues.apache.org/jira/browse/HADOOP-14518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100505#comment-16100505 ] Jitendra Nath Pandey commented on HADOOP-14518: --- I have committed this to trunk and branch-2. Thanks for the contribution, [~Georgi]. > Customize User-Agent header sent in HTTP/HTTPS requests by WASB. > > > Key: HADOOP-14518 > URL: https://issues.apache.org/jira/browse/HADOOP-14518 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.0.0-beta1 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov >Priority: Minor > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HADOOP-14518-01.patch, HADOOP-14518-01-test.txt, > HADOOP-14518-02.patch, HADOOP-14518-03.patch, HADOOP-14518-04.patch, > HADOOP-14518-05.patch, HADOOP-14518-06.patch, HADOOP-14518-branch-2.01.patch > > > WASB passes a User-Agent header to the Azure back-end. Right now, it uses the > default value set by the Azure Client SDK, so Hadoop traffic doesn't appear > any different from general Blob traffic. If we customize the User-Agent > header, then it will enable better troubleshooting and analysis by Azure > service. > The following configuration > > fs.azure.user.agent.prefix > MSFT > > set the user agent to > User-Agent: WASB/3.0.0-alpha4-SNAPSHOT (MSFT) Azure-Storage/4.2.0 > (JavaJRE 1.8.0_131; WindowsServer2012R2 6.3) > Test Results : > Tests run: 703, Failures: 0, Errors: 0, Skipped: 119 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14518) Customize User-Agent header sent in HTTP/HTTPS requests by WASB.
[ https://issues.apache.org/jira/browse/HADOOP-14518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HADOOP-14518: -- Resolution: Fixed Fix Version/s: 3.0.0-beta1 2.9.0 Target Version/s: 2.8.1, 3.0.0-beta1 (was: 3.0.0-beta1, 2.8.1) Status: Resolved (was: Patch Available) > Customize User-Agent header sent in HTTP/HTTPS requests by WASB. > > > Key: HADOOP-14518 > URL: https://issues.apache.org/jira/browse/HADOOP-14518 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.0.0-beta1 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov >Priority: Minor > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HADOOP-14518-01.patch, HADOOP-14518-01-test.txt, > HADOOP-14518-02.patch, HADOOP-14518-03.patch, HADOOP-14518-04.patch, > HADOOP-14518-05.patch, HADOOP-14518-06.patch, HADOOP-14518-branch-2.01.patch > > > WASB passes a User-Agent header to the Azure back-end. Right now, it uses the > default value set by the Azure Client SDK, so Hadoop traffic doesn't appear > any different from general Blob traffic. If we customize the User-Agent > header, then it will enable better troubleshooting and analysis by Azure > service. > The following configuration > > fs.azure.user.agent.prefix > MSFT > > set the user agent to > User-Agent: WASB/3.0.0-alpha4-SNAPSHOT (MSFT) Azure-Storage/4.2.0 > (JavaJRE 1.8.0_131; WindowsServer2012R2 6.3) > Test Results : > Tests run: 703, Failures: 0, Errors: 0, Skipped: 119 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14518) Customize User-Agent header sent in HTTP/HTTPS requests by WASB.
[ https://issues.apache.org/jira/browse/HADOOP-14518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HADOOP-14518: -- Attachment: HADOOP-14518-branch-2.01.patch Attaching branch-2 patch for pre-commit run. > Customize User-Agent header sent in HTTP/HTTPS requests by WASB. > > > Key: HADOOP-14518 > URL: https://issues.apache.org/jira/browse/HADOOP-14518 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.0.0-beta1 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov >Priority: Minor > Attachments: HADOOP-14518-01.patch, HADOOP-14518-01-test.txt, > HADOOP-14518-02.patch, HADOOP-14518-03.patch, HADOOP-14518-04.patch, > HADOOP-14518-05.patch, HADOOP-14518-06.patch, HADOOP-14518-branch-2.01.patch > > > WASB passes a User-Agent header to the Azure back-end. Right now, it uses the > default value set by the Azure Client SDK, so Hadoop traffic doesn't appear > any different from general Blob traffic. If we customize the User-Agent > header, then it will enable better troubleshooting and analysis by Azure > service. > The following configuration > > fs.azure.user.agent.prefix > MSFT > > set the user agent to > User-Agent: WASB/3.0.0-alpha4-SNAPSHOT (MSFT) Azure-Storage/4.2.0 > (JavaJRE 1.8.0_131; WindowsServer2012R2 6.3) > Test Results : > Tests run: 703, Failures: 0, Errors: 0, Skipped: 119 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14518) Customize User-Agent header sent in HTTP/HTTPS requests by WASB.
[ https://issues.apache.org/jira/browse/HADOOP-14518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099127#comment-16099127 ] Jitendra Nath Pandey commented on HADOOP-14518: --- +1 for the latest patch. The test failures are unrelated to the patch. > Customize User-Agent header sent in HTTP/HTTPS requests by WASB. > > > Key: HADOOP-14518 > URL: https://issues.apache.org/jira/browse/HADOOP-14518 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.0.0-beta1 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov >Priority: Minor > Attachments: HADOOP-14518-01.patch, HADOOP-14518-01-test.txt, > HADOOP-14518-02.patch, HADOOP-14518-03.patch, HADOOP-14518-04.patch, > HADOOP-14518-05.patch, HADOOP-14518-06.patch > > > WASB passes a User-Agent header to the Azure back-end. Right now, it uses the > default value set by the Azure Client SDK, so Hadoop traffic doesn't appear > any different from general Blob traffic. If we customize the User-Agent > header, then it will enable better troubleshooting and analysis by Azure > service. > The following configuration > > fs.azure.user.agent.prefix > MSFT > > set the user agent to > User-Agent: WASB/3.0.0-alpha4-SNAPSHOT (MSFT) Azure-Storage/4.2.0 > (JavaJRE 1.8.0_131; WindowsServer2012R2 6.3) > Test Results : > Tests run: 703, Failures: 0, Errors: 0, Skipped: 119 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14674) Correct javadoc for getRandomizedTempPath
[
https://issues.apache.org/jira/browse/HADOOP-14674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HADOOP-14674:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
I have committed this to trunk. Thanks [~msingh].
> Correct javadoc for getRandomizedTempPath
> -
>
> Key: HADOOP-14674
> URL: https://issues.apache.org/jira/browse/HADOOP-14674
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HADOOP-14674.001.patch
>
>
> getRandomizedTempPath has incorrect javadoc where the javadoc specifies a
> parameter to the function however the function doesnt expects one.
> {code}
> /**
>* Get a temp path. This may or may not be relative; it depends on what the
>* {@link #SYSPROP_TEST_DATA_DIR} is set to. If unset, it returns a path
>* under the relative path {@link #DEFAULT_TEST_DATA_PATH}
>* @param subpath sub path, with no leading "/" character
>* @return a string to use in paths
>*/
> public static String getRandomizedTempPath() {
> return getTempPath(RandomStringUtils.randomAlphanumeric(10));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14674) Correct javadoc for getRandomizedTempPath
[
https://issues.apache.org/jira/browse/HADOOP-14674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HADOOP-14674:
--
Fix Version/s: 3.0.0-beta1
> Correct javadoc for getRandomizedTempPath
> -
>
> Key: HADOOP-14674
> URL: https://issues.apache.org/jira/browse/HADOOP-14674
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14674.001.patch
>
>
> getRandomizedTempPath has incorrect javadoc where the javadoc specifies a
> parameter to the function however the function doesnt expects one.
> {code}
> /**
>* Get a temp path. This may or may not be relative; it depends on what the
>* {@link #SYSPROP_TEST_DATA_DIR} is set to. If unset, it returns a path
>* under the relative path {@link #DEFAULT_TEST_DATA_PATH}
>* @param subpath sub path, with no leading "/" character
>* @return a string to use in paths
>*/
> public static String getRandomizedTempPath() {
> return getTempPath(RandomStringUtils.randomAlphanumeric(10));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14674) Correct javadoc for getRandomizedTempPath
[
https://issues.apache.org/jira/browse/HADOOP-14674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16096607#comment-16096607
]
Jitendra Nath Pandey commented on HADOOP-14674:
---
+1
> Correct javadoc for getRandomizedTempPath
> -
>
> Key: HADOOP-14674
> URL: https://issues.apache.org/jira/browse/HADOOP-14674
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HADOOP-14674.001.patch
>
>
> getRandomizedTempPath has incorrect javadoc where the javadoc specifies a
> parameter to the function however the function doesnt expects one.
> {code}
> /**
>* Get a temp path. This may or may not be relative; it depends on what the
>* {@link #SYSPROP_TEST_DATA_DIR} is set to. If unset, it returns a path
>* under the relative path {@link #DEFAULT_TEST_DATA_PATH}
>* @param subpath sub path, with no leading "/" character
>* @return a string to use in paths
>*/
> public static String getRandomizedTempPath() {
> return getTempPath(RandomStringUtils.randomAlphanumeric(10));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14518) Customize User-Agent header sent in HTTP/HTTPS requests by WASB.
[ https://issues.apache.org/jira/browse/HADOOP-14518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095415#comment-16095415 ] Jitendra Nath Pandey commented on HADOOP-14518: --- The patch looks ok to me, however it needs to be rebased to latest trunk. Also, the latest patch has many unintended diff lines that make the patch look large. [~Georgi], please upload a rebased patch. > Customize User-Agent header sent in HTTP/HTTPS requests by WASB. > > > Key: HADOOP-14518 > URL: https://issues.apache.org/jira/browse/HADOOP-14518 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov >Priority: Minor > Attachments: HADOOP-14518-01.patch, HADOOP-14518-01-test.txt, > HADOOP-14518-02.patch, HADOOP-14518-03.patch, HADOOP-14518-04.patch, > HADOOP-14518-05.patch > > > WASB passes a User-Agent header to the Azure back-end. Right now, it uses the > default value set by the Azure Client SDK, so Hadoop traffic doesn't appear > any different from general Blob traffic. If we customize the User-Agent > header, then it will enable better troubleshooting and analysis by Azure > service. > The following configuration > > fs.azure.user.agent.prefix > MSFT > > set the user agent to > User-Agent: WASB/3.0.0-alpha4-SNAPSHOT (MSFT) Azure-Storage/4.2.0 > (JavaJRE 1.8.0_131; WindowsServer2012R2 6.3) > Test Results : > Tests run: 703, Failures: 0, Errors: 0, Skipped: 119 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14642) wasb: add support for caching Authorization and SASKeys
[ https://issues.apache.org/jira/browse/HADOOP-14642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HADOOP-14642: -- Resolution: Fixed Fix Version/s: 2.9.0 Status: Resolved (was: Patch Available) I have committed this to branch-2. Thanks [~sisan...@microsoft.com] for the contribution. > wasb: add support for caching Authorization and SASKeys > --- > > Key: HADOOP-14642 > URL: https://issues.apache.org/jira/browse/HADOOP-14642 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HADOOP-14642.001.patch, HADOOP-14642.002.patch, > HADOOP-14642.003.patch, HADOOP-14642.004.patch, HADOOP-14642.005.patch, > HADOOP-14642-branch-2.1.patch > > > TPCDS runs showed performance issues with Secure Clusters. > One of the main issues was identified to be repeated calls to DTS for the > same SASKey and Authorization request. > The performance issue was mitigated to a large extent by introducing caching > for saskeys and authorization requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14642) wasb: add support for caching Authorization and SASKeys
[ https://issues.apache.org/jira/browse/HADOOP-14642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HADOOP-14642: -- Labels: azure fs secure wasb (was: 2.9.0 azure fs secure wasb) > wasb: add support for caching Authorization and SASKeys > --- > > Key: HADOOP-14642 > URL: https://issues.apache.org/jira/browse/HADOOP-14642 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-14642.001.patch, HADOOP-14642.002.patch, > HADOOP-14642.003.patch, HADOOP-14642.004.patch, HADOOP-14642.005.patch, > HADOOP-14642-branch-2.1.patch > > > TPCDS runs showed performance issues with Secure Clusters. > One of the main issues was identified to be repeated calls to DTS for the > same SASKey and Authorization request. > The performance issue was mitigated to a large extent by introducing caching > for saskeys and authorization requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14642) wasb: add support for caching Authorization and SASKeys
[ https://issues.apache.org/jira/browse/HADOOP-14642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HADOOP-14642: -- Target Version/s: 2.9.0 > wasb: add support for caching Authorization and SASKeys > --- > > Key: HADOOP-14642 > URL: https://issues.apache.org/jira/browse/HADOOP-14642 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-14642.001.patch, HADOOP-14642.002.patch, > HADOOP-14642.003.patch, HADOOP-14642.004.patch, HADOOP-14642.005.patch, > HADOOP-14642-branch-2.1.patch > > > TPCDS runs showed performance issues with Secure Clusters. > One of the main issues was identified to be repeated calls to DTS for the > same SASKey and Authorization request. > The performance issue was mitigated to a large extent by introducing caching > for saskeys and authorization requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14642) wasb: add support for caching Authorization and SASKeys
[ https://issues.apache.org/jira/browse/HADOOP-14642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HADOOP-14642: -- Fix Version/s: 3.0.0-beta1 > wasb: add support for caching Authorization and SASKeys > --- > > Key: HADOOP-14642 > URL: https://issues.apache.org/jira/browse/HADOOP-14642 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: 2.9.0, azure, fs, secure, wasb > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-14642.001.patch, HADOOP-14642.002.patch, > HADOOP-14642.003.patch, HADOOP-14642.004.patch, HADOOP-14642.005.patch, > HADOOP-14642-branch-2.1.patch > > > TPCDS runs showed performance issues with Secure Clusters. > One of the main issues was identified to be repeated calls to DTS for the > same SASKey and Authorization request. > The performance issue was mitigated to a large extent by introducing caching > for saskeys and authorization requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14642) wasb: add support for caching Authorization and SASKeys
[ https://issues.apache.org/jira/browse/HADOOP-14642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HADOOP-14642: -- Attachment: HADOOP-14642-branch-2.1.patch I have committed this to trunk. Patch for branch-2 attached for test run. > wasb: add support for caching Authorization and SASKeys > --- > > Key: HADOOP-14642 > URL: https://issues.apache.org/jira/browse/HADOOP-14642 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: 2.9.0, azure, fs, secure, wasb > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-14642.001.patch, HADOOP-14642.002.patch, > HADOOP-14642.003.patch, HADOOP-14642.004.patch, HADOOP-14642.005.patch, > HADOOP-14642-branch-2.1.patch > > > TPCDS runs showed performance issues with Secure Clusters. > One of the main issues was identified to be repeated calls to DTS for the > same SASKey and Authorization request. > The performance issue was mitigated to a large extent by introducing caching > for saskeys and authorization requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14642) wasb: add support for caching Authorization and SASKeys
[ https://issues.apache.org/jira/browse/HADOOP-14642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HADOOP-14642: -- Labels: 2.9.0 azure fs secure wasb (was: azure fs secure wasb) > wasb: add support for caching Authorization and SASKeys > --- > > Key: HADOOP-14642 > URL: https://issues.apache.org/jira/browse/HADOOP-14642 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: 2.9.0, azure, fs, secure, wasb > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-14642.001.patch, HADOOP-14642.002.patch, > HADOOP-14642.003.patch, HADOOP-14642.004.patch, HADOOP-14642.005.patch, > HADOOP-14642-branch-2.1.patch > > > TPCDS runs showed performance issues with Secure Clusters. > One of the main issues was identified to be repeated calls to DTS for the > same SASKey and Authorization request. > The performance issue was mitigated to a large extent by introducing caching > for saskeys and authorization requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14642) wasb: add support for caching Authorization and SASKeys
[ https://issues.apache.org/jira/browse/HADOOP-14642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092677#comment-16092677 ] Jitendra Nath Pandey commented on HADOOP-14642: --- +1. The latest patch looks good to me. > wasb: add support for caching Authorization and SASKeys > --- > > Key: HADOOP-14642 > URL: https://issues.apache.org/jira/browse/HADOOP-14642 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Jitendra Nath Pandey > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14642.001.patch, HADOOP-14642.002.patch, > HADOOP-14642.003.patch, HADOOP-14642.004.patch, HADOOP-14642.005.patch > > > TPCDS runs showed performance issues with Secure Clusters. > One of the main issues was identified to be repeated calls to DTS for the > same SASKey and Authorization request. > The performance issue was mitigated to a large extent by introducing caching > for saskeys and authorization requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14642) wasb: add support for caching Authorization and SASKeys
[ https://issues.apache.org/jira/browse/HADOOP-14642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey reassigned HADOOP-14642: - Assignee: Sivaguru Sankaridurg (was: Jitendra Nath Pandey) > wasb: add support for caching Authorization and SASKeys > --- > > Key: HADOOP-14642 > URL: https://issues.apache.org/jira/browse/HADOOP-14642 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14642.001.patch, HADOOP-14642.002.patch, > HADOOP-14642.003.patch, HADOOP-14642.004.patch, HADOOP-14642.005.patch > > > TPCDS runs showed performance issues with Secure Clusters. > One of the main issues was identified to be repeated calls to DTS for the > same SASKey and Authorization request. > The performance issue was mitigated to a large extent by introducing caching > for saskeys and authorization requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14640) Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation
[
https://issues.apache.org/jira/browse/HADOOP-14640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HADOOP-14640:
--
Resolution: Fixed
Fix Version/s: 2.9.0
Target Version/s: 3.0.0-alpha1, 2.9.0 (was: 2.9.0, 3.0.0-alpha1)
Status: Resolved (was: Patch Available)
Committed to branch-2 as well.
Thanks [~snayak] for the contribution.
> Azure: Support affinity for service running on localhost and reuse SPNEGO
> hadoop.auth cookie for authorization, SASKey and delegation token generation
> --
>
> Key: HADOOP-14640
> URL: https://issues.apache.org/jira/browse/HADOOP-14640
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Labels: security
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: HADOOP-14640.1.patch, HADOOP-14640.2.patch,
> HADOOP-14640.3.patch, HADOOP-14640.4.patch, HADOOP-14640-branch-2.01.patch
>
>
> Currently, {{WasbRemoteCallHelper}} can be configured to talk to comma
> separated list of URLs for authorization, SASKey generation and delegation
> token generation.
> To improve the performance, if service runs on the local machine, give it
> first preference over the other configured list of URLs.
> Currently, {{WasbRemoteCallHelper}} generates {{hadoop.auth}} cookie for
> every request by talking to the remote service, before making actual rest
> requests.
> The proposal is to reuse the {{hadoop.auth}} cookie for subsequent requests
> from same {{WasbRemoteCallHelper}} object until its expiry time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14642) wasb: add support for caching Authorization and SASKeys
[ https://issues.apache.org/jira/browse/HADOOP-14642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090917#comment-16090917 ] Jitendra Nath Pandey commented on HADOOP-14642: --- [~sisan...@microsoft.com], please rebase the patch, so that we can get a clean run. > wasb: add support for caching Authorization and SASKeys > --- > > Key: HADOOP-14642 > URL: https://issues.apache.org/jira/browse/HADOOP-14642 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14642.001.patch, HADOOP-14642.002.patch > > > TPCDS runs showed performance issues with Secure Clusters. > One of the main issues was identified to be repeated calls to DTS for the > same SASKey and Authorization request. > The performance issue was mitigated to a large extent by introducing caching > for saskeys and authorization requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14642) wasb: add support for caching Authorization and SASKeys
[ https://issues.apache.org/jira/browse/HADOOP-14642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HADOOP-14642: -- Status: Patch Available (was: Open) > wasb: add support for caching Authorization and SASKeys > --- > > Key: HADOOP-14642 > URL: https://issues.apache.org/jira/browse/HADOOP-14642 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14642.001.patch, HADOOP-14642.002.patch > > > TPCDS runs showed performance issues with Secure Clusters. > One of the main issues was identified to be repeated calls to DTS for the > same SASKey and Authorization request. > The performance issue was mitigated to a large extent by introducing caching > for saskeys and authorization requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14640) Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation
[
https://issues.apache.org/jira/browse/HADOOP-14640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HADOOP-14640:
--
Attachment: HADOOP-14640-branch-2.01.patch
The patch applies cleanly to branch-2. Attaching.
> Azure: Support affinity for service running on localhost and reuse SPNEGO
> hadoop.auth cookie for authorization, SASKey and delegation token generation
> --
>
> Key: HADOOP-14640
> URL: https://issues.apache.org/jira/browse/HADOOP-14640
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Labels: security
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14640.1.patch, HADOOP-14640.2.patch,
> HADOOP-14640.3.patch, HADOOP-14640.4.patch, HADOOP-14640-branch-2.01.patch
>
>
> Currently, {{WasbRemoteCallHelper}} can be configured to talk to comma
> separated list of URLs for authorization, SASKey generation and delegation
> token generation.
> To improve the performance, if service runs on the local machine, give it
> first preference over the other configured list of URLs.
> Currently, {{WasbRemoteCallHelper}} generates {{hadoop.auth}} cookie for
> every request by talking to the remote service, before making actual rest
> requests.
> The proposal is to reuse the {{hadoop.auth}} cookie for subsequent requests
> from same {{WasbRemoteCallHelper}} object until its expiry time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14640) Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation
[
https://issues.apache.org/jira/browse/HADOOP-14640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HADOOP-14640:
--
Target Version/s: 3.0.0-alpha1, 2.9.0 (was: 3.0.0-alpha1)
> Azure: Support affinity for service running on localhost and reuse SPNEGO
> hadoop.auth cookie for authorization, SASKey and delegation token generation
> --
>
> Key: HADOOP-14640
> URL: https://issues.apache.org/jira/browse/HADOOP-14640
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Labels: security
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14640.1.patch, HADOOP-14640.2.patch,
> HADOOP-14640.3.patch, HADOOP-14640.4.patch
>
>
> Currently, {{WasbRemoteCallHelper}} can be configured to talk to comma
> separated list of URLs for authorization, SASKey generation and delegation
> token generation.
> To improve the performance, if service runs on the local machine, give it
> first preference over the other configured list of URLs.
> Currently, {{WasbRemoteCallHelper}} generates {{hadoop.auth}} cookie for
> every request by talking to the remote service, before making actual rest
> requests.
> The proposal is to reuse the {{hadoop.auth}} cookie for subsequent requests
> from same {{WasbRemoteCallHelper}} object until its expiry time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14640) Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation
[
https://issues.apache.org/jira/browse/HADOOP-14640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HADOOP-14640:
--
Fix Version/s: 3.0.0-beta1
> Azure: Support affinity for service running on localhost and reuse SPNEGO
> hadoop.auth cookie for authorization, SASKey and delegation token generation
> --
>
> Key: HADOOP-14640
> URL: https://issues.apache.org/jira/browse/HADOOP-14640
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Labels: security
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14640.1.patch, HADOOP-14640.2.patch,
> HADOOP-14640.3.patch, HADOOP-14640.4.patch
>
>
> Currently, {{WasbRemoteCallHelper}} can be configured to talk to comma
> separated list of URLs for authorization, SASKey generation and delegation
> token generation.
> To improve the performance, if service runs on the local machine, give it
> first preference over the other configured list of URLs.
> Currently, {{WasbRemoteCallHelper}} generates {{hadoop.auth}} cookie for
> every request by talking to the remote service, before making actual rest
> requests.
> The proposal is to reuse the {{hadoop.auth}} cookie for subsequent requests
> from same {{WasbRemoteCallHelper}} object until its expiry time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14640) Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation
[
https://issues.apache.org/jira/browse/HADOOP-14640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089546#comment-16089546
]
Jitendra Nath Pandey commented on HADOOP-14640:
---
I have committed this to trunk. Keeping the jira open for commit to branch-2.
> Azure: Support affinity for service running on localhost and reuse SPNEGO
> hadoop.auth cookie for authorization, SASKey and delegation token generation
> --
>
> Key: HADOOP-14640
> URL: https://issues.apache.org/jira/browse/HADOOP-14640
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Labels: security
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14640.1.patch, HADOOP-14640.2.patch,
> HADOOP-14640.3.patch, HADOOP-14640.4.patch
>
>
> Currently, {{WasbRemoteCallHelper}} can be configured to talk to comma
> separated list of URLs for authorization, SASKey generation and delegation
> token generation.
> To improve the performance, if service runs on the local machine, give it
> first preference over the other configured list of URLs.
> Currently, {{WasbRemoteCallHelper}} generates {{hadoop.auth}} cookie for
> every request by talking to the remote service, before making actual rest
> requests.
> The proposal is to reuse the {{hadoop.auth}} cookie for subsequent requests
> from same {{WasbRemoteCallHelper}} object until its expiry time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14640) Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation
[
https://issues.apache.org/jira/browse/HADOOP-14640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089529#comment-16089529
]
Jitendra Nath Pandey commented on HADOOP-14640:
---
+1 The latest patch looks good to me.
Thanks for addressing the comments [~snayak].
> Azure: Support affinity for service running on localhost and reuse SPNEGO
> hadoop.auth cookie for authorization, SASKey and delegation token generation
> --
>
> Key: HADOOP-14640
> URL: https://issues.apache.org/jira/browse/HADOOP-14640
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Labels: security
> Attachments: HADOOP-14640.1.patch, HADOOP-14640.2.patch,
> HADOOP-14640.3.patch, HADOOP-14640.4.patch
>
>
> Currently, {{WasbRemoteCallHelper}} can be configured to talk to comma
> separated list of URLs for authorization, SASKey generation and delegation
> token generation.
> To improve the performance, if service runs on the local machine, give it
> first preference over the other configured list of URLs.
> Currently, {{WasbRemoteCallHelper}} generates {{hadoop.auth}} cookie for
> every request by talking to the remote service, before making actual rest
> requests.
> The proposal is to reuse the {{hadoop.auth}} cookie for subsequent requests
> from same {{WasbRemoteCallHelper}} object until its expiry time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14640) Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation
[
https://issues.apache.org/jira/browse/HADOOP-14640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088271#comment-16088271
]
Jitendra Nath Pandey commented on HADOOP-14640:
---
- The other findbug issue for delegationToken is also worth looking at. Why
does makeRemoteRequest tries to lookup a delegationToken, but getHttpRequest
doesn't ?
- I see a change to format of License Header. In some azure files the
license link is wrapped inside {{..}}. Lets be consistent with rest of
hadoop. For other files which are already committed, we can address it in a
follow up jira.
> Azure: Support affinity for service running on localhost and reuse SPNEGO
> hadoop.auth cookie for authorization, SASKey and delegation token generation
> --
>
> Key: HADOOP-14640
> URL: https://issues.apache.org/jira/browse/HADOOP-14640
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Labels: security
> Attachments: HADOOP-14640.1.patch, HADOOP-14640.2.patch,
> HADOOP-14640.3.patch
>
>
> Currently, {{WasbRemoteCallHelper}} can be configured to talk to comma
> separated list of URLs for authorization, SASKey generation and delegation
> token generation.
> To improve the performance, if service runs on the local machine, give it
> first preference over the other configured list of URLs.
> Currently, {{WasbRemoteCallHelper}} generates {{hadoop.auth}} cookie for
> every request by talking to the remote service, before making actual rest
> requests.
> The proposal is to reuse the {{hadoop.auth}} cookie for subsequent requests
> from same {{WasbRemoteCallHelper}} object until its expiry time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14640) Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation
[
https://issues.apache.org/jira/browse/HADOOP-14640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088243#comment-16088243
]
Jitendra Nath Pandey edited comment on HADOOP-14640 at 7/14/17 10:52 PM:
-
This seems like a good catch by findbugs. It will cause NPE if statusLine is
null.
{code}
if (statusLine == null
|| statusLine.getStatusCode() != HttpStatus.SC_OK) {
requiresNewAuth =
(statusLine.getStatusCode() == HttpStatus.SC_UNAUTHORIZED);
{code}
was (Author: jnp):
This is a good catch by findbugs:
{code}
if (statusLine == null
|| statusLine.getStatusCode() != HttpStatus.SC_OK) {
requiresNewAuth =
(statusLine.getStatusCode() == HttpStatus.SC_UNAUTHORIZED);
{code}
stat
> Azure: Support affinity for service running on localhost and reuse SPNEGO
> hadoop.auth cookie for authorization, SASKey and delegation token generation
> --
>
> Key: HADOOP-14640
> URL: https://issues.apache.org/jira/browse/HADOOP-14640
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Labels: security
> Attachments: HADOOP-14640.1.patch, HADOOP-14640.2.patch,
> HADOOP-14640.3.patch
>
>
> Currently, {{WasbRemoteCallHelper}} can be configured to talk to comma
> separated list of URLs for authorization, SASKey generation and delegation
> token generation.
> To improve the performance, if service runs on the local machine, give it
> first preference over the other configured list of URLs.
> Currently, {{WasbRemoteCallHelper}} generates {{hadoop.auth}} cookie for
> every request by talking to the remote service, before making actual rest
> requests.
> The proposal is to reuse the {{hadoop.auth}} cookie for subsequent requests
> from same {{WasbRemoteCallHelper}} object until its expiry time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14640) Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation
[
https://issues.apache.org/jira/browse/HADOOP-14640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088243#comment-16088243
]
Jitendra Nath Pandey commented on HADOOP-14640:
---
This is a good catch by findbugs:
{code}
if (statusLine == null
|| statusLine.getStatusCode() != HttpStatus.SC_OK) {
requiresNewAuth =
(statusLine.getStatusCode() == HttpStatus.SC_UNAUTHORIZED);
{code}
stat
> Azure: Support affinity for service running on localhost and reuse SPNEGO
> hadoop.auth cookie for authorization, SASKey and delegation token generation
> --
>
> Key: HADOOP-14640
> URL: https://issues.apache.org/jira/browse/HADOOP-14640
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Labels: security
> Attachments: HADOOP-14640.1.patch, HADOOP-14640.2.patch,
> HADOOP-14640.3.patch
>
>
> Currently, {{WasbRemoteCallHelper}} can be configured to talk to comma
> separated list of URLs for authorization, SASKey generation and delegation
> token generation.
> To improve the performance, if service runs on the local machine, give it
> first preference over the other configured list of URLs.
> Currently, {{WasbRemoteCallHelper}} generates {{hadoop.auth}} cookie for
> every request by talking to the remote service, before making actual rest
> requests.
> The proposal is to reuse the {{hadoop.auth}} cookie for subsequent requests
> from same {{WasbRemoteCallHelper}} object until its expiry time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14640) Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation
[
https://issues.apache.org/jira/browse/HADOOP-14640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HADOOP-14640:
--
Status: Patch Available (was: Open)
> Azure: Support affinity for service running on localhost and reuse SPNEGO
> hadoop.auth cookie for authorization, SASKey and delegation token generation
> --
>
> Key: HADOOP-14640
> URL: https://issues.apache.org/jira/browse/HADOOP-14640
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Santhosh G Nayak
>Assignee: Santhosh G Nayak
> Labels: security
> Attachments: HADOOP-14640.1.patch, HADOOP-14640.2.patch,
> HADOOP-14640.3.patch
>
>
> Currently, {{WasbRemoteCallHelper}} can be configured to talk to comma
> separated list of URLs for authorization, SASKey generation and delegation
> token generation.
> To improve the performance, if service runs on the local machine, give it
> first preference over the other configured list of URLs.
> Currently, {{WasbRemoteCallHelper}} generates {{hadoop.auth}} cookie for
> every request by talking to the remote service, before making actual rest
> requests.
> The proposal is to reuse the {{hadoop.auth}} cookie for subsequent requests
> from same {{WasbRemoteCallHelper}} object until its expiry time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
