[jira] [Commented] (HADOOP-13558) UserGroupInformation created from a Subject incorrectly tries to renew the Kerberos ticket
[
https://issues.apache.org/jira/browse/HADOOP-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15460131#comment-15460131
]
Xiao Chen commented on HADOOP-13558:
Did a full hadoop unit test, and a smoke test run in a kerberized cluster. No
regression found.
I plan to commit this on Tuesday (given Monday is a U.S. holiday). Feel free to
comment if any objections.
> UserGroupInformation created from a Subject incorrectly tries to renew the
> Kerberos ticket
> --
>
> Key: HADOOP-13558
> URL: https://issues.apache.org/jira/browse/HADOOP-13558
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.7.2, 2.6.4, 3.0.0-alpha2
>Reporter: Alejandro Abdelnur
>Assignee: Xiao Chen
> Attachments: HADOOP-13558.01.patch, HADOOP-13558.02.patch
>
>
> The UGI {{checkTGTAndReloginFromKeytab()}} method checks certain conditions
> and if they are met it invokes the {{reloginFromKeytab()}}. The
> {{reloginFromKeytab()}} method then fails with an {{IOException}}
> "loginUserFromKeyTab must be done first" because there is no keytab
> associated with the UGI.
> The {{checkTGTAndReloginFromKeytab()}} method checks if there is a keytab
> ({{isKeytab}} UGI instance variable) associated with the UGI, if there is one
> it triggers a call to {{reloginFromKeytab()}}. The problem is that the
> {{keytabFile}} UGI instance variable is NULL, and that triggers the mentioned
> {{IOException}}.
> The root of the problem seems to be when creating a UGI via the
> {{UGI.loginUserFromSubject(Subject)}} method, this method uses the
> {{UserGroupInformation(Subject)}} constructor, and this constructor does the
> following to determine if there is a keytab or not.
> {code}
> this.isKeytab = KerberosUtil.hasKerberosKeyTab(subject);
> {code}
> If the {{Subject}} given had a keytab, then the UGI instance will have the
> {{isKeytab}} set to TRUE.
> It sets the UGI instance as it would have a keytab because the Subject has a
> keytab. This has 2 problems:
> First, it does not set the keytab file (and this, having the {{isKeytab}} set
> to TRUE and the {{keytabFile}} set to NULL) is what triggers the
> {{IOException}} in the method {{reloginFromKeytab()}}.
> Second (and even if the first problem is fixed, this still is a problem), it
> assumes that because the subject has a keytab it is up to UGI to do the
> relogin using the keytab. This is incorrect if the UGI was created using the
> {{UGI.loginUserFromSubject(Subject)}} method. In such case, the owner of the
> Subject is not the UGI, but the caller, so the caller is responsible for
> renewing the Kerberos tickets and the UGI should not try to do so.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13577) Download page should link the ASF mirrora for KEYS, sigs, hashes
[ https://issues.apache.org/jira/browse/HADOOP-13577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459911#comment-15459911 ] Sebb commented on HADOOP-13577: --- Note that KEYS, sigs and hashes are deliberately not synched to 3rd party mirrors. > Download page should link the ASF mirrora for KEYS, sigs, hashes > > > Key: HADOOP-13577 > URL: https://issues.apache.org/jira/browse/HADOOP-13577 > Project: Hadoop Common > Issue Type: Bug > Environment: http://hadoop.apache.org/releases.html >Reporter: Sebb > > The download page currently points to > https://dist.apache.org/repos/dist/release/hadoop/... > for KEYS, sigs and hashes. > However the dist SVN tree is not designed for this; such files must be > downloaded from the ASF mirrors, i.e > https://www.apache.org/dist/hadoop/... > Please can you adjust the links accordingly? > The links should use HTTPS. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13577) Download page should link the ASF mirrora for KEYS, sigs, hashes
[ https://issues.apache.org/jira/browse/HADOOP-13577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459908#comment-15459908 ] Sebb commented on HADOOP-13577: --- I meant that the files need to be served from the ASF mirror *sources*, i.e. https://www.apache.org/dist/hadoop/... That is an ASF host, just as is: https://dist.apache.org/repos/dist/release/hadoop/... > Download page should link the ASF mirrora for KEYS, sigs, hashes > > > Key: HADOOP-13577 > URL: https://issues.apache.org/jira/browse/HADOOP-13577 > Project: Hadoop Common > Issue Type: Bug > Environment: http://hadoop.apache.org/releases.html >Reporter: Sebb > > The download page currently points to > https://dist.apache.org/repos/dist/release/hadoop/... > for KEYS, sigs and hashes. > However the dist SVN tree is not designed for this; such files must be > downloaded from the ASF mirrors, i.e > https://www.apache.org/dist/hadoop/... > Please can you adjust the links accordingly? > The links should use HTTPS. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13577) Download page should link the ASF mirrora for KEYS, sigs, hashes
[ https://issues.apache.org/jira/browse/HADOOP-13577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459892#comment-15459892 ] Allen Wittenauer commented on HADOOP-13577: --- The whole point of having KEYS, sigs, etc come from a different server is to make sure that the mirror site didn't get compromised. This is why httpd, maven, etc, are all doing the same thing--we all point to dist.apache.org to download these files. These files are lightweight compared to the rest of the content and should be fine to be downloaded from apache.org. > Download page should link the ASF mirrora for KEYS, sigs, hashes > > > Key: HADOOP-13577 > URL: https://issues.apache.org/jira/browse/HADOOP-13577 > Project: Hadoop Common > Issue Type: Bug > Environment: http://hadoop.apache.org/releases.html >Reporter: Sebb > > The download page currently points to > https://dist.apache.org/repos/dist/release/hadoop/... > for KEYS, sigs and hashes. > However the dist SVN tree is not designed for this; such files must be > downloaded from the ASF mirrors, i.e > https://www.apache.org/dist/hadoop/... > Please can you adjust the links accordingly? > The links should use HTTPS. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13578) Add Codec for ZStandard Compression
churro morales created HADOOP-13578: --- Summary: Add Codec for ZStandard Compression Key: HADOOP-13578 URL: https://issues.apache.org/jira/browse/HADOOP-13578 Project: Hadoop Common Issue Type: New Feature Reporter: churro morales Fix For: 3.0.0-alpha1 ZStandard: https://github.com/facebook/zstd has been used in production for 6 months by facebook now. v1.0 was recently released. Create a codec for this library. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13577) Download page should link the ASF mirrora for KEYS, sigs, hashes
Sebb created HADOOP-13577: - Summary: Download page should link the ASF mirrora for KEYS, sigs, hashes Key: HADOOP-13577 URL: https://issues.apache.org/jira/browse/HADOOP-13577 Project: Hadoop Common Issue Type: Bug Environment: http://hadoop.apache.org/releases.html Reporter: Sebb The download page currently points to https://dist.apache.org/repos/dist/release/hadoop/... for KEYS, sigs and hashes. However the dist SVN tree is not designed for this; such files must be downloaded from the ASF mirrors, i.e https://www.apache.org/dist/hadoop/... Please can you adjust the links accordingly? The links should use HTTPS. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10128) Please delete old releases from mirroring system
[ https://issues.apache.org/jira/browse/HADOOP-10128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459802#comment-15459802 ] Sebb commented on HADOOP-10128: --- Please could someone take care of removing old releases? i.e. everything apart from 2.7.3, 2.6.4, 2.5.2 The following need to be removed please: https://dist.apache.org/repos/dist/release/hadoop/common/hadoop-1.2.1/ https://dist.apache.org/repos/dist/release/hadoop/common/hadoop-2.6.0/ https://dist.apache.org/repos/dist/release/hadoop/common/hadoop-2.6.1/ https://dist.apache.org/repos/dist/release/hadoop/common/hadoop-2.6.2/ https://dist.apache.org/repos/dist/release/hadoop/common/hadoop-2.6.3/ https://dist.apache.org/repos/dist/release/hadoop/common/hadoop-2.7.0/ https://dist.apache.org/repos/dist/release/hadoop/common/hadoop-2.7.1/ https://dist.apache.org/repos/dist/release/hadoop/common/hadoop-2.7.2/ Please also update any release manager documentation to include tidying old releases. It's unfair on the 3rd party mirrors if they have to continue carrying superseded releases. > Please delete old releases from mirroring system > > > Key: HADOOP-10128 > URL: https://issues.apache.org/jira/browse/HADOOP-10128 > Project: Hadoop Common > Issue Type: Bug > Environment: http://www.apache.org/dist/hadoop/common/ > http://www.apache.org/dist/hadoop/core/ >Reporter: Sebb > > To reduce the load on the ASF mirrors, projects are required to delete old > releases. > Please can you remove all non-current releases? > i.e. anything except > 0.23.9 > 1.2.1 > 2.2.0 > Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13375) o.a.h.security.TestGroupsCaching.testBackgroundRefreshCounters seems flaky
[
https://issues.apache.org/jira/browse/HADOOP-13375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459674#comment-15459674
]
Andrew Wang commented on HADOOP-13375:
--
Gentle reminder to please include the appropriate 3.0.0 fix version when
committing to trunk, thanks!
> o.a.h.security.TestGroupsCaching.testBackgroundRefreshCounters seems flaky
> --
>
> Key: HADOOP-13375
> URL: https://issues.apache.org/jira/browse/HADOOP-13375
> Project: Hadoop Common
> Issue Type: Bug
> Components: security, test
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Weiwei Yang
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13375.001.patch, HADOOP-13375.002.patch,
> HADOOP-13375.003.patch, HADOOP-13375.004.patch, HADOOP-13375.005.patch,
> HADOOP-13375.006.patch, HADOOP-13375.007.patch
>
>
> h5. Error Message
> bq. expected:<1> but was:<0>
> h5. Stacktrace
> {quote}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.security.TestGroupsCaching.testBackgroundRefreshCounters(TestGroupsCaching.java:638)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13375) o.a.h.security.TestGroupsCaching.testBackgroundRefreshCounters seems flaky
[
https://issues.apache.org/jira/browse/HADOOP-13375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HADOOP-13375:
-
Fix Version/s: 3.0.0-alpha2
> o.a.h.security.TestGroupsCaching.testBackgroundRefreshCounters seems flaky
> --
>
> Key: HADOOP-13375
> URL: https://issues.apache.org/jira/browse/HADOOP-13375
> Project: Hadoop Common
> Issue Type: Bug
> Components: security, test
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Weiwei Yang
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13375.001.patch, HADOOP-13375.002.patch,
> HADOOP-13375.003.patch, HADOOP-13375.004.patch, HADOOP-13375.005.patch,
> HADOOP-13375.006.patch, HADOOP-13375.007.patch
>
>
> h5. Error Message
> bq. expected:<1> but was:<0>
> h5. Stacktrace
> {quote}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.security.TestGroupsCaching.testBackgroundRefreshCounters(TestGroupsCaching.java:638)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13575) Website shows incorrect 25 January, 2016 as the release date for 2.7.3
[ https://issues.apache.org/jira/browse/HADOOP-13575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459437#comment-15459437 ] Xiao Chen commented on HADOOP-13575: Thanks for creating this, Jacek. [~vinodkv], could you please check and update? Thanks. > Website shows incorrect 25 January, 2016 as the release date for 2.7.3 > -- > > Key: HADOOP-13575 > URL: https://issues.apache.org/jira/browse/HADOOP-13575 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.3 >Reporter: Jacek Laskowski > > http://hadoop.apache.org/releases.html shows 2.7.3 released on 25 January, > 2016 which should rather be 25 August, 2016. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13541) explicitly declare the Joda time version S3A depends on
[ https://issues.apache.org/jira/browse/HADOOP-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459272#comment-15459272 ] Andrew Wang commented on HADOOP-13541: -- I'll +1 on faith, I assume you ran mvn dependency plugin to verify. Any thoughts about how we can prevent/verify this kind of thing for the future? I don't think it's picked up by our API checking tool. > explicitly declare the Joda time version S3A depends on > --- > > Key: HADOOP-13541 > URL: https://issues.apache.org/jira/browse/HADOOP-13541 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3 >Affects Versions: 2.8.0, 2.7.3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-13541-branch-2.8-001.patch > > > Different builds of Hadoop are pulling in wildly different versions of Joda > time, depending on what other transitive dependencies are involved. Example: > 2.7.3 is somehow picking up Joda time 2.9.4; branch-2.8 is actually behind on > 2.8.1. That's going to cause confusion when people upgrade from 2.7.x to 2.8 > and find a dependency has got older > I propose explicitly declaring a dependency on joda-time in s3a, then set the > version to 2.9.4; upgrades are things we can manage -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13537) Support external calls in the RPC call queue
[
https://issues.apache.org/jira/browse/HADOOP-13537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459169#comment-15459169
]
Hadoop QA commented on HADOOP-13537:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HADOOP-13537 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-13537 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12825129/HADOOP-13537.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10444/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Support external calls in the RPC call queue
>
>
> Key: HADOOP-13537
> URL: https://issues.apache.org/jira/browse/HADOOP-13537
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HADOOP-13537.patch
>
>
> Leveraging HADOOP-13465 will allow non-rpc calls to be added to the call
> queue. This is intended to support routing webhdfs calls through the call
> queue to provide a unified and protocol-independent QoS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13537) Support external calls in the RPC call queue
[ https://issues.apache.org/jira/browse/HADOOP-13537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HADOOP-13537: Status: Patch Available (was: Open) > Support external calls in the RPC call queue > > > Key: HADOOP-13537 > URL: https://issues.apache.org/jira/browse/HADOOP-13537 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HADOOP-13537.patch > > > Leveraging HADOOP-13465 will allow non-rpc calls to be added to the call > queue. This is intended to support routing webhdfs calls through the call > queue to provide a unified and protocol-independent QoS. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13537) Support external calls in the RPC call queue
[ https://issues.apache.org/jira/browse/HADOOP-13537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459138#comment-15459138 ] Kihwal Lee commented on HADOOP-13537: - Submitted... > Support external calls in the RPC call queue > > > Key: HADOOP-13537 > URL: https://issues.apache.org/jira/browse/HADOOP-13537 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HADOOP-13537.patch > > > Leveraging HADOOP-13465 will allow non-rpc calls to be added to the call > queue. This is intended to support routing webhdfs calls through the call > queue to provide a unified and protocol-independent QoS. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13558) UserGroupInformation created from a Subject incorrectly tries to renew the Kerberos ticket
[
https://issues.apache.org/jira/browse/HADOOP-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459136#comment-15459136
]
Xiao Chen commented on HADOOP-13558:
FYI - I'm doing some final testing as Steve suggested in his earlier comments.
Will update here later today.
Thanks for reviewing!
> UserGroupInformation created from a Subject incorrectly tries to renew the
> Kerberos ticket
> --
>
> Key: HADOOP-13558
> URL: https://issues.apache.org/jira/browse/HADOOP-13558
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.7.2, 2.6.4, 3.0.0-alpha2
>Reporter: Alejandro Abdelnur
>Assignee: Xiao Chen
> Attachments: HADOOP-13558.01.patch, HADOOP-13558.02.patch
>
>
> The UGI {{checkTGTAndReloginFromKeytab()}} method checks certain conditions
> and if they are met it invokes the {{reloginFromKeytab()}}. The
> {{reloginFromKeytab()}} method then fails with an {{IOException}}
> "loginUserFromKeyTab must be done first" because there is no keytab
> associated with the UGI.
> The {{checkTGTAndReloginFromKeytab()}} method checks if there is a keytab
> ({{isKeytab}} UGI instance variable) associated with the UGI, if there is one
> it triggers a call to {{reloginFromKeytab()}}. The problem is that the
> {{keytabFile}} UGI instance variable is NULL, and that triggers the mentioned
> {{IOException}}.
> The root of the problem seems to be when creating a UGI via the
> {{UGI.loginUserFromSubject(Subject)}} method, this method uses the
> {{UserGroupInformation(Subject)}} constructor, and this constructor does the
> following to determine if there is a keytab or not.
> {code}
> this.isKeytab = KerberosUtil.hasKerberosKeyTab(subject);
> {code}
> If the {{Subject}} given had a keytab, then the UGI instance will have the
> {{isKeytab}} set to TRUE.
> It sets the UGI instance as it would have a keytab because the Subject has a
> keytab. This has 2 problems:
> First, it does not set the keytab file (and this, having the {{isKeytab}} set
> to TRUE and the {{keytabFile}} set to NULL) is what triggers the
> {{IOException}} in the method {{reloginFromKeytab()}}.
> Second (and even if the first problem is fixed, this still is a problem), it
> assumes that because the subject has a keytab it is up to UGI to do the
> relogin using the keytab. This is incorrect if the UGI was created using the
> {{UGI.loginUserFromSubject(Subject)}} method. In such case, the owner of the
> Subject is not the UGI, but the caller, so the caller is responsible for
> renewing the Kerberos tickets and the UGI should not try to do so.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13282) S3 blob etags to be made visible in status/getFileChecksum() calls
[
https://issues.apache.org/jira/browse/HADOOP-13282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13282:
Affects Version/s: 2.9.0
> S3 blob etags to be made visible in status/getFileChecksum() calls
> --
>
> Key: HADOOP-13282
> URL: https://issues.apache.org/jira/browse/HADOOP-13282
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Priority: Minor
>
> If the etags of blobs were exported via {{getFileChecksum() }}, it'd be
> possible to probe for a blob being in sync with a local file. Distcp could
> use this to decide whether to skip a file or not.
> Now, there's a problem there: distcp needs source and dest filesystems to
> implement the same algorithm. It'd only work out the box if you were copying
> between S3 instances. There are also quirks with encryption and multipart:
> [s3
> docs|http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html].
> At the very least, it's something which could be used when indexing the FS,
> to check for changes later.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HADOOP-13549) Eliminate intermediate buffer for server-side PB encoding
[ https://issues.apache.org/jira/browse/HADOOP-13549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HADOOP-13549: Comment: was deleted (was: committed to 2.8.) > Eliminate intermediate buffer for server-side PB encoding > - > > Key: HADOOP-13549 > URL: https://issues.apache.org/jira/browse/HADOOP-13549 > Project: Hadoop Common > Issue Type: Sub-task > Components: ipc >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HADOOP-13549.patch > > > HADOOP-13426 improved encoding and added framed buffers. Upon further > review, the intermediate response buffer is completely unnecessary for PB > responses since the size can be pre-computed unlike Writables. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13549) Eliminate intermediate buffer for server-side PB encoding
[ https://issues.apache.org/jira/browse/HADOOP-13549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15458969#comment-15458969 ] Kihwal Lee commented on HADOOP-13549: - committed to 2.8. > Eliminate intermediate buffer for server-side PB encoding > - > > Key: HADOOP-13549 > URL: https://issues.apache.org/jira/browse/HADOOP-13549 > Project: Hadoop Common > Issue Type: Sub-task > Components: ipc >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HADOOP-13549.patch > > > HADOOP-13426 improved encoding and added framed buffers. Upon further > review, the intermediate response buffer is completely unnecessary for PB > responses since the size can be pre-computed unlike Writables. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13447) Refactor S3AFileSystem to support introduction of separate metadata repository and tests.
[
https://issues.apache.org/jira/browse/HADOOP-13447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15458965#comment-15458965
]
Hadoop QA commented on HADOOP-13447:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 12s{color} | {color:orange} hadoop-tools/hadoop-aws: The patch generated 3
new + 9 unchanged - 11 fixed = 12 total (was 20) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} hadoop-tools_hadoop-aws generated 0 new + 0
unchanged - 2 fixed = 0 total (was 2) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13447 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824912/HADOOP-13447.005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml findbugs checkstyle |
| uname | Linux 7ba4fdfc96a6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 23abb09 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10443/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10443/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10443/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Refactor S3AFileSystem to support introduction of separate metadata
> repository and tests.
> -
>
>
[jira] [Commented] (HADOOP-13547) Optimize IPC client protobuf decoding
[ https://issues.apache.org/jira/browse/HADOOP-13547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15458951#comment-15458951 ] Hudson commented on HADOOP-13547: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10390 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10390/]) HADOOP-13547. Optimize IPC client protobuf decoding. Contributed by (kihwal: rev 23abb09c1f979d8c18ece81e32630a35ed569399) * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcWritable.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ResponseBuffer.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SaslRpcClient.java > Optimize IPC client protobuf decoding > - > > Key: HADOOP-13547 > URL: https://issues.apache.org/jira/browse/HADOOP-13547 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-13547.1.patch, HADOOP-13547.branch-2.1.patch, > HADOOP-13547.branch-2.patch, HADOOP-13547.patch > > > Counterpart to HADOOP-13438. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13547) Optimize IPC client protobuf decoding
[ https://issues.apache.org/jira/browse/HADOOP-13547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HADOOP-13547: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2 2.8.0 Status: Resolved (was: Patch Available) Committed to trunk through branch-2.8. > Optimize IPC client protobuf decoding > - > > Key: HADOOP-13547 > URL: https://issues.apache.org/jira/browse/HADOOP-13547 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-13547.1.patch, HADOOP-13547.branch-2.1.patch, > HADOOP-13547.branch-2.patch, HADOOP-13547.patch > > > Counterpart to HADOOP-13438. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13547) Optimize IPC client protobuf decoding
[ https://issues.apache.org/jira/browse/HADOOP-13547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15458912#comment-15458912 ] Kihwal Lee commented on HADOOP-13547: - +1 the patch looks good. > Optimize IPC client protobuf decoding > - > > Key: HADOOP-13547 > URL: https://issues.apache.org/jira/browse/HADOOP-13547 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HADOOP-13547.1.patch, HADOOP-13547.branch-2.1.patch, > HADOOP-13547.branch-2.patch, HADOOP-13547.patch > > > Counterpart to HADOOP-13438. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13447) Refactor S3AFileSystem to support introduction of separate metadata repository and tests.
[
https://issues.apache.org/jira/browse/HADOOP-13447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-13447:
---
Status: Patch Available (was: Open)
> Refactor S3AFileSystem to support introduction of separate metadata
> repository and tests.
> -
>
> Key: HADOOP-13447
> URL: https://issues.apache.org/jira/browse/HADOOP-13447
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HADOOP-13447-HADOOP-13446.001.patch,
> HADOOP-13447-HADOOP-13446.002.patch, HADOOP-13447.003.patch,
> HADOOP-13447.004.patch, HADOOP-13447.005.patch
>
>
> The scope of this issue is to refactor the existing {{S3AFileSystem}} into
> multiple coordinating classes. The goal of this refactoring is to separate
> the {{FileSystem}} API binding from the AWS SDK integration, make code
> maintenance easier while we're making changes for S3Guard, and make it easier
> to mock some implementation details so that tests can simulate eventual
> consistency behavior in a deterministic way.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13438) Optimize IPC server protobuf decoding
[ https://issues.apache.org/jira/browse/HADOOP-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HADOOP-13438: Fix Version/s: (was: 2.9.0) 2.8.0 > Optimize IPC server protobuf decoding > - > > Key: HADOOP-13438 > URL: https://issues.apache.org/jira/browse/HADOOP-13438 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HADOOP-13438.patch, HADOOP-13438.patch.1 > > > The current use of the protobuf API uses an expensive code path. The builder > uses the parser to instantiate a message, then copies the message into the > builder. The parser is creating multi-layered internally buffering streams > that cause excessive byte[] allocations. > Using the parser directly with a coded input stream backed by the byte[] from > the wire will take a fast-path straight to the pb message's ctor. > Substantially less garbage is generated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13576) Old Hadoop user mailing lists still exist
Sebb created HADOOP-13576: - Summary: Old Hadoop user mailing lists still exist Key: HADOOP-13576 URL: https://issues.apache.org/jira/browse/HADOOP-13576 Project: Hadoop Common Issue Type: Bug Reporter: Sebb Priority: Minor The mailing lists for (common,hdfs,mapreduce)-user@ are still active; i.e. users can subscribe and post messages. If that was not the intention [1] I suggest raising an INFRA JIRA to get the old lists closed down and/or aliased to user@. See also INFRA-12554. [1] http://mail-archives.apache.org/mod_mbox/hadoop-user/201208.mbox/%3C0A0CA23A-313F-4BA7-ADC5-31A1DB76B4FA%40hortonworks.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13574) Unnecessary file existence check causes problems with S3
[
https://issues.apache.org/jira/browse/HADOOP-13574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15458221#comment-15458221
]
Steve Loughran commented on HADOOP-13574:
-
I see the point of this, and I can also see that this would be an optimisation
as you'd skip the overhead of 1-3 HTTP GET requests,
But...we've effectively frozen all dev of s3n as it is stable enough that
things work, and it gives us a fallback while s3a stabilises & takes on
performance improvements.
Looking at the S3A code, the same-ish problem exists. S3A adds the fix for
HADOOP-13188, not only doing the GET but failing if the destination path is a
directory. That is, even if overwrite==false, you want to make sure that you
aren't unintentionally creating a file over a dir, so unintentionally losing
all access to the data underneath (it'd break the listing code, see). If I were
to go near S3n, I'd probably focus on that bug, rather than dealing with the
intermittent caching of the 404 that S3 can do.
Your particular problem is really due to the fact that the output committer is
doing a rename(), which from a performance perspective is the wrong thing to
ask an object store what to do. In HADOOP-13345, S3Guard, we're planning to
deal with this —anything you can do to help there would be welcome.
> Unnecessary file existence check causes problems with S3
>
>
> Key: HADOOP-13574
> URL: https://issues.apache.org/jira/browse/HADOOP-13574
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Reporter: Alex Zolotko
>Priority: Minor
> Labels: s3
>
> We recently got the following exception on production:
> {noformat}
> java.io.FileNotFoundException: Key
> 'xxx/_temporary/0/_temporary/attempt_201609010631__m_001128_1128/part-01128'
> does not exist in S3
> at
> org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleServiceException(Jets3tNativeFileSystemStore.java:234)
> at
> org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.copy(Jets3tNativeFileSystemStore.java:201)
> at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at org.apache.hadoop.fs.s3native.$Proxy13.copy(Unknown Source)
> at
> org.apache.hadoop.fs.s3native.NativeS3FileSystem.rename(NativeS3FileSystem.java:659)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitTask(FileOutputCommitter.java:435)
> at
> org.apache.hadoop.mapred.FileOutputCommitter.commitTask(FileOutputCommitter.java:172)
> at
> org.apache.hadoop.mapred.OutputCommitter.commitTask(OutputCommitter.java:291)
> at
> org.apache.spark.mapred.SparkHadoopMapRedUtil$.performCommit$1(SparkHadoopMapRedUtil.scala:98)
> at
> org.apache.spark.mapred.SparkHadoopMapRedUtil$.commitTask(SparkHadoopMapRedUtil.scala:124)
> at
> org.apache.spark.SparkHadoopWriter.commit(SparkHadoopWriter.scala:107)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1204)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1183)
> {noformat}
> FileOutputCommitter.commitTask() does check that the file exists before
> trying to rename it, but due to S3's relaxed consistency guarantees the
> following fs.rename(taskAttemptPath, committedTaskPath) still fails.
> Here's an excerpt from the Amazon S3 documentation
> ([https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html]):
> {quote}
> Amazon S3 Data Consistency Model
> Amazon S3 provides read-after-write consistency for PUTS of new objects in
> your S3 bucket in all regions with one caveat. The caveat is that if you make
> a HEAD or GET request to the key name (to find if the object exists) before
> creating the object, Amazon S3 provides eventual consistency for
> read-after-write.
> {quote}
> The problematic S3 object existence check, that causes S3 to fallback to
> eventual consistency, is in NativeS3FileSystem.create():
> {code}
> if (exists(f) && !overwrite) {
> throw new IOException("File already exists:"+f);
> }
> {code}
> If the "overwrite" parameter is set to "true" (as in our case), calling
> exists(f) is unnecessary and only "upsets" S3.
> The proposed fix is to switch the order of the predicates:
> {code}
> if (!overwrite && exists(f))
[jira] [Commented] (HADOOP-13218) Migrate other Hadoop side tests to prepare for removing WritableRPCEngine
[
https://issues.apache.org/jira/browse/HADOOP-13218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15458105#comment-15458105
]
Hadoop QA commented on HADOOP-13218:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 12 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
56s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 38s{color} | {color:orange} root: The patch generated 2 new + 835 unchanged
- 46 fixed = 837 total (was 881) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
25s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 48s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m
48s{color} | {color:green} hadoop-mapreduce-client-hs in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}145m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDecommissionWithStriped |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13218 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12826779/HADOOP-13218-v02.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux ddb6cb3c699c 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 0690f09 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10442/artifact/patchprocess/diff-checkstyle-root.txt
|
| unit |
[jira] [Created] (HADOOP-13575) Website shows incorrect 25 January, 2016 as the release date for 2.7.3
Jacek Laskowski created HADOOP-13575: Summary: Website shows incorrect 25 January, 2016 as the release date for 2.7.3 Key: HADOOP-13575 URL: https://issues.apache.org/jira/browse/HADOOP-13575 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.7.3 Reporter: Jacek Laskowski http://hadoop.apache.org/releases.html shows 2.7.3 released on 25 January, 2016 which should rather be 25 August, 2016. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13574) Unnecessary file existence check causes problems with S3
[
https://issues.apache.org/jira/browse/HADOOP-13574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Zolotko updated HADOOP-13574:
--
Description:
We recently got the following exception on production:
{noformat}
java.io.FileNotFoundException: Key
'xxx/_temporary/0/_temporary/attempt_201609010631__m_001128_1128/part-01128'
does not exist in S3
at
org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleServiceException(Jets3tNativeFileSystemStore.java:234)
at
org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.copy(Jets3tNativeFileSystemStore.java:201)
at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at org.apache.hadoop.fs.s3native.$Proxy13.copy(Unknown Source)
at
org.apache.hadoop.fs.s3native.NativeS3FileSystem.rename(NativeS3FileSystem.java:659)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitTask(FileOutputCommitter.java:435)
at
org.apache.hadoop.mapred.FileOutputCommitter.commitTask(FileOutputCommitter.java:172)
at
org.apache.hadoop.mapred.OutputCommitter.commitTask(OutputCommitter.java:291)
at
org.apache.spark.mapred.SparkHadoopMapRedUtil$.performCommit$1(SparkHadoopMapRedUtil.scala:98)
at
org.apache.spark.mapred.SparkHadoopMapRedUtil$.commitTask(SparkHadoopMapRedUtil.scala:124)
at
org.apache.spark.SparkHadoopWriter.commit(SparkHadoopWriter.scala:107)
at
org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1204)
at
org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1183)
{noformat}
FileOutputCommitter.commitTask() does check that the file exists before trying
to rename it, but due to S3's relaxed consistency guarantees the following
fs.rename(taskAttemptPath, committedTaskPath) still fails.
Here's an excerpt from the Amazon S3 documentation
([https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html]):
{quote}
Amazon S3 Data Consistency Model
Amazon S3 provides read-after-write consistency for PUTS of new objects in your
S3 bucket in all regions with one caveat. The caveat is that if you make a HEAD
or GET request to the key name (to find if the object exists) before creating
the object, Amazon S3 provides eventual consistency for read-after-write.
{quote}
The problematic S3 object existence check, that causes S3 to fallback to
eventual consistency, is in NativeS3FileSystem.create():
{code}
if (exists(f) && !overwrite) {
throw new IOException("File already exists:"+f);
}
{code}
If the "overwrite" parameter is set to "true" (as in our case), calling
exists(f) is unnecessary and only "upsets" S3.
The proposed fix is to switch the order of the predicates:
{code}
if (!overwrite && exists(f)) {
throw new IOException("File already exists:"+f);
}
{code}
was:
We recently got the following exception on production:
{noformat}
java.io.FileNotFoundException: Key
'xxx/_temporary/0/_temporary/attempt_201609010631__m_001128_1128/part-01128'
does not exist in S3
at
org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleServiceException(Jets3tNativeFileSystemStore.java:234)
at
org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.copy(Jets3tNativeFileSystemStore.java:201)
at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at org.apache.hadoop.fs.s3native.$Proxy13.copy(Unknown Source)
at
org.apache.hadoop.fs.s3native.NativeS3FileSystem.rename(NativeS3FileSystem.java:659)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitTask(FileOutputCommitter.java:435)
at
org.apache.hadoop.mapred.FileOutputCommitter.commitTask(FileOutputCommitter.java:172)
at
org.apache.hadoop.mapred.OutputCommitter.commitTask(OutputCommitter.java:291)
at
org.apache.spark.mapred.SparkHadoopMapRedUtil$.performCommit$1(SparkHadoopMapRedUtil.scala:98)
at
org.apache.spark.mapred.SparkHadoopMapRedUtil$.commitTask(SparkHadoopMapRedUtil.scala:124)
at
[jira] [Created] (HADOOP-13574) Unnecessary file existence check causes problems with S3
Alex Zolotko created HADOOP-13574:
-
Summary: Unnecessary file existence check causes problems with S3
Key: HADOOP-13574
URL: https://issues.apache.org/jira/browse/HADOOP-13574
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Reporter: Alex Zolotko
Priority: Minor
We recently got the following exception on production:
{noformat}
java.io.FileNotFoundException: Key
'xxx/_temporary/0/_temporary/attempt_201609010631__m_001128_1128/part-01128'
does not exist in S3
at
org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleServiceException(Jets3tNativeFileSystemStore.java:234)
at
org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.copy(Jets3tNativeFileSystemStore.java:201)
at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at org.apache.hadoop.fs.s3native.$Proxy13.copy(Unknown Source)
at
org.apache.hadoop.fs.s3native.NativeS3FileSystem.rename(NativeS3FileSystem.java:659)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitTask(FileOutputCommitter.java:435)
at
org.apache.hadoop.mapred.FileOutputCommitter.commitTask(FileOutputCommitter.java:172)
at
org.apache.hadoop.mapred.OutputCommitter.commitTask(OutputCommitter.java:291)
at
org.apache.spark.mapred.SparkHadoopMapRedUtil$.performCommit$1(SparkHadoopMapRedUtil.scala:98)
at
org.apache.spark.mapred.SparkHadoopMapRedUtil$.commitTask(SparkHadoopMapRedUtil.scala:124)
at
org.apache.spark.SparkHadoopWriter.commit(SparkHadoopWriter.scala:107)
at
org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1204)
at
org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1183)
{noformat}
FileOutputCommitter.commitTask() does check that the file exists before trying
to rename it, but due to S3's relaxed consistency guarantees the following
fs.rename(taskAttemptPath, committedTaskPath) still fails.
Here's an excerpt from the Amazon S3 documentation
([https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html]):
{quote}
Amazon S3 Data Consistency Model
Amazon S3 provides read-after-write consistency for PUTS of new objects in your
S3 bucket in all regions with one caveat. The caveat is that if you make a HEAD
or GET request to the key name (to find if the object exists) before creating
the object, Amazon S3 provides eventual consistency for read-after-write.
{quote}
The problematic S3 object existence check, that causes S3 to fallback to
eventual consistency, is in NativeS3FileSystem.create():
{code}
if (exists(f) && !overwrite) {
throw new IOException("File already exists:"+f);
}
{code}
If the overwrite parameter is set to "true" (as in our case), calling exists(f)
is unnecessary and only "upsets" S3.
The proposed fix is switch the order of the predicates:
{code}
if (!overwrite && exists(f)) {
throw new IOException("File already exists:"+f);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13218) Migrate other Hadoop side tests to prepare for removing WritableRPCEngine
[ https://issues.apache.org/jira/browse/HADOOP-13218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zhou updated HADOOP-13218: -- Attachment: HADOOP-13218-v02.patch Update the patch and fix issues reported by Jenkins, thanks! > Migrate other Hadoop side tests to prepare for removing WritableRPCEngine > - > > Key: HADOOP-13218 > URL: https://issues.apache.org/jira/browse/HADOOP-13218 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Wei Zhou > Attachments: HADOOP-13218-v01.patch, HADOOP-13218-v02.patch > > > Patch for HADOOP-12579 contains lots of work to migrate the left Hadoop side > tests basing on the new RPC engine and nice cleanups. HADOOP-12579 will be > reverted to allow some time for YARN/Mapreduce side related changes, open > this to recommit most of the test related work in HADOOP-12579 for easier > tracking and maintain, as other sub-tasks did. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
