[jira] [Commented] (HADOOP-14875) Create end user documentation from the compatibility guidelines
[ https://issues.apache.org/jira/browse/HADOOP-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368105#comment-16368105 ] Hudson commented on HADOOP-14875: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13675 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13675/]) HADOOP-14875. Create end user documentation from the compatibility (wangda: rev 31db977cb33ac7dde04d6001b9ad69012df4e3a0) * (edit) hadoop-project/src/site/site.xml * (add) hadoop-common-project/hadoop-common/src/site/markdown/AdminCompatibilityGuide.md * (edit) hadoop-common-project/hadoop-common/src/site/markdown/Compatibility.md > Create end user documentation from the compatibility guidelines > --- > > Key: HADOOP-14875 > URL: https://issues.apache.org/jira/browse/HADOOP-14875 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 3.0.0-beta1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Fix For: 3.1.0 > > Attachments: AdminCompatibilityGuide.pdf, Compatibility.htm, > HADOOP-14875.001.patch, HADOOP-14875.002.patch, HADOOP-14875.003.patch, > HADOOP-14875.004.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14875) Create end user documentation from the compatibility guidelines
[ https://issues.apache.org/jira/browse/HADOOP-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated HADOOP-14875: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Available) Committed to trunk/branch-3.1, thanks [~templedf] > Create end user documentation from the compatibility guidelines > --- > > Key: HADOOP-14875 > URL: https://issues.apache.org/jira/browse/HADOOP-14875 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 3.0.0-beta1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Fix For: 3.1.0 > > Attachments: AdminCompatibilityGuide.pdf, Compatibility.htm, > HADOOP-14875.001.patch, HADOOP-14875.002.patch, HADOOP-14875.003.patch, > HADOOP-14875.004.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14875) Create end user documentation from the compatibility guidelines
[ https://issues.apache.org/jira/browse/HADOOP-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368090#comment-16368090 ] Wangda Tan commented on HADOOP-14875: - Thanks [~templedf], will commit shortly. > Create end user documentation from the compatibility guidelines > --- > > Key: HADOOP-14875 > URL: https://issues.apache.org/jira/browse/HADOOP-14875 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 3.0.0-beta1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: AdminCompatibilityGuide.pdf, Compatibility.htm, > HADOOP-14875.001.patch, HADOOP-14875.002.patch, HADOOP-14875.003.patch, > HADOOP-14875.004.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15007) Stabilize and document Configuration element
[
https://issues.apache.org/jira/browse/HADOOP-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368078#comment-16368078
]
genericqa commented on HADOOP-15007:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
26s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 42s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 352 unchanged - 0 fixed = 353 total (was 352) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 3s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
47s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 84m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15007 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910987/HADOOP-15007.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux 8ab26ac85328 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c008d10 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14160/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14160/testReport/ |
| Max. process+thread count | 1718 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
[jira] [Commented] (HADOOP-15007) Stabilize and document Configuration element
[
https://issues.apache.org/jira/browse/HADOOP-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368053#comment-16368053
]
Ajay Kumar commented on HADOOP-15007:
-
[~anu] patch v2 to remove HADOOP_SYSTEM_TAGS_DEFAULT.
{quote}Not sure why we callthe removeUndeclaredTags, if we don't do that we get
custom tag support right? Am I missing something here?{quote}
Intention is to remove all undeclared,mis-spelled tags. I am open to removing
it if you think otherwise.
> Stabilize and document Configuration element
> --
>
> Key: HADOOP-15007
> URL: https://issues.apache.org/jira/browse/HADOOP-15007
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Ajay Kumar
>Priority: Blocker
> Attachments: HADOOP-15007.000.patch, HADOOP-15007.001.patch,
> HADOOP-15007.002.patch
>
>
> HDFS-12350 (moved to HADOOP-15005). Adds the ability to tag properties with a
> value.
> We need to make sure that this feature is backwards compatible & usable in
> production. That's docs, testing, marshalling etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15007) Stabilize and document Configuration element
[ https://issues.apache.org/jira/browse/HADOOP-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HADOOP-15007: Attachment: HADOOP-15007.002.patch > Stabilize and document Configuration element > -- > > Key: HADOOP-15007 > URL: https://issues.apache.org/jira/browse/HADOOP-15007 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Ajay Kumar >Priority: Blocker > Attachments: HADOOP-15007.000.patch, HADOOP-15007.001.patch, > HADOOP-15007.002.patch > > > HDFS-12350 (moved to HADOOP-15005). Adds the ability to tag properties with a > value. > We need to make sure that this feature is backwards compatible & usable in > production. That's docs, testing, marshalling etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14875) Create end user documentation from the compatibility guidelines
[
https://issues.apache.org/jira/browse/HADOOP-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367981#comment-16367981
]
genericqa commented on HADOOP-14875:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
29m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 31s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14875 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910976/HADOOP-14875.004.patch
|
| Optional Tests | asflicense mvnsite xml |
| uname | Linux 06bc5baf412e 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 0898ff4 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 301 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-common-project/hadoop-common U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14159/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Create end user documentation from the compatibility guidelines
> ---
>
> Key: HADOOP-14875
> URL: https://issues.apache.org/jira/browse/HADOOP-14875
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 3.0.0-beta1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: AdminCompatibilityGuide.pdf, Compatibility.htm,
> HADOOP-14875.001.patch, HADOOP-14875.002.patch, HADOOP-14875.003.patch,
> HADOOP-14875.004.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[
https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367954#comment-16367954
]
Aaron Fabbri commented on HADOOP-13761:
---
{quote}Mark up the seek/read calls with @RetryPolicy, so it's easier to see
what happens transitively
{quote}
Humm.. these codepaths are sort of a new case. They retry sometimes, depending
on s3guard state. Should we add some new annotations
S3GuardRetry[Mixed,Raw,Translated]? Or just call them Retry with a clarifying
comment–since they "are covered WRT retries that we actually want". I'm
thinking the latter.
{quote}only have the s3guardInvoker field set to S3GuardExistsRetryPolicy if
S3Guard is enabled, otherwise just set s3guardInvoker=invoker.
and maybe just call it readInvoker
then you can go context.readInvoker on those calls without much logic in the
input stream.
{quote}
I thought about this. I'm concerned about misuse of the new retry policy
breaking fail-fast behavior where it is needed, so I opted to make each
codepath very explicit where the change in behavior happens and why. Let me
take a look at it again.
{quote}the invoked operations themselves can be stuck inline
{quote}
This statement is true. ;)
Thanks for taking the time to think through this stuff–I wanted extra eyes on
it. I was a bit on the fence about the read failpoints, not knowing if they
really occur in the wild: getObject() seems to actually open an HTTP stream and
read usually consumes that, so not sure you can really get FNFE there.
Wondering if we could get throttling there too–but it depends on unknown S3
service behavior (i.e. does throttling only affect new read sessions and
metadata requests, or can it affect an open HTTP/TCP stream. I suppose we
could have a reconnect that hits another shard and gets FNFE?).
> S3Guard: implement retries for DDB failures and throttling; translate
> exceptions
>
>
> Key: HADOOP-13761
> URL: https://issues.apache.org/jira/browse/HADOOP-13761
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Aaron Fabbri
>Assignee: Aaron Fabbri
>Priority: Blocker
> Attachments: HADOOP-13761-004-to-005.patch,
> HADOOP-13761-005-to-006-approx.diff.txt, HADOOP-13761-005.patch,
> HADOOP-13761-006.patch, HADOOP-13761.001.patch, HADOOP-13761.002.patch,
> HADOOP-13761.003.patch, HADOOP-13761.004.patch
>
>
> Following the S3AFileSystem integration patch in HADOOP-13651, we need to add
> retry logic.
> In HADOOP-13651, I added TODO comments in most of the places retry loops are
> needed, including:
> - open(path). If MetadataStore reflects recent create/move of file path, but
> we fail to read it from S3, retry.
> - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the
> file exists, retry.
> - rename(src,dest). If source path is not visible in S3 yet, retry.
> - listFiles(). Skip for now. Not currently implemented in S3Guard. I will
> create a separate JIRA for this as it will likely require interface changes
> (i.e. prefix or subtree scan).
> We may miss some cases initially and we should do failure injection testing
> to make sure we're covered. Failure injection tests can be a separate JIRA
> to make this easier to review.
> We also need basic configuration parameters around retry policy. There
> should be a way to specify maximum retry duration, as some applications would
> prefer to receive an error eventually, than waiting indefinitely. We should
> also be keeping statistics when inconsistency is detected and we enter a
> retry loop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14875) Create end user documentation from the compatibility guidelines
[ https://issues.apache.org/jira/browse/HADOOP-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367944#comment-16367944 ] Daniel Templeton commented on HADOOP-14875: --- I attached a patch to address your first three concerns. The fourth one is correct as is. > Create end user documentation from the compatibility guidelines > --- > > Key: HADOOP-14875 > URL: https://issues.apache.org/jira/browse/HADOOP-14875 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 3.0.0-beta1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: AdminCompatibilityGuide.pdf, Compatibility.htm, > HADOOP-14875.001.patch, HADOOP-14875.002.patch, HADOOP-14875.003.patch, > HADOOP-14875.004.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14875) Create end user documentation from the compatibility guidelines
[ https://issues.apache.org/jira/browse/HADOOP-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated HADOOP-14875: -- Attachment: HADOOP-14875.004.patch > Create end user documentation from the compatibility guidelines > --- > > Key: HADOOP-14875 > URL: https://issues.apache.org/jira/browse/HADOOP-14875 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 3.0.0-beta1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: AdminCompatibilityGuide.pdf, Compatibility.htm, > HADOOP-14875.001.patch, HADOOP-14875.002.patch, HADOOP-14875.003.patch, > HADOOP-14875.004.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15206) BZip2 drops and duplicates records when input split size is small
[
https://issues.apache.org/jira/browse/HADOOP-15206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated HADOOP-15206:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.2
2.7.6
2.8.4
2.9.1
2.10.0
3.1.0
Status: Resolved (was: Patch Available)
Thanks, [~tanakahda]! I committed this to trunk, branch-3.1, branch-3.0,
branch-2, branch-2.9, branch-2.8, and branch-2.7.
> BZip2 drops and duplicates records when input split size is small
> -
>
> Key: HADOOP-15206
> URL: https://issues.apache.org/jira/browse/HADOOP-15206
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.3, 3.0.0
>Reporter: Aki Tanaka
>Assignee: Aki Tanaka
>Priority: Major
> Fix For: 3.1.0, 2.10.0, 2.9.1, 2.8.4, 2.7.6, 3.0.2
>
> Attachments: HADOOP-15206-test.patch, HADOOP-15206.001.patch,
> HADOOP-15206.002.patch, HADOOP-15206.003.patch, HADOOP-15206.004.patch,
> HADOOP-15206.005.patch, HADOOP-15206.006.patch, HADOOP-15206.007.patch,
> HADOOP-15206.008.patch
>
>
> BZip2 can drop and duplicate record when input split file is small. I
> confirmed that this issue happens when the input split size is between 1byte
> and 4bytes.
> I am seeing the following 2 problem behaviors.
>
> 1. Drop record:
> BZip2 skips the first record in the input file when the input split size is
> small
>
> Set the split size to 3 and tested to load 100 records (0, 1, 2..99)
> {code:java}
> 2018-02-01 10:52:33,502 INFO [Thread-17] mapred.TestTextInputFormat
> (TestTextInputFormat.java:verifyPartitions(317)) -
> splits[1]=file:/work/count-mismatch2/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/TestTextInputFormat/test.bz2:3+3
> count=99{code}
> > The input format read only 99 records but not 100 records
>
> 2. Duplicate Record:
> 2 input splits has same BZip2 records when the input split size is small
>
> Set the split size to 1 and tested to load 100 records (0, 1, 2..99)
>
> {code:java}
> 2018-02-01 11:18:49,309 INFO [Thread-17] mapred.TestTextInputFormat
> (TestTextInputFormat.java:verifyPartitions(318)) - splits[3]=file
> /work/count-mismatch2/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/TestTextInputFormat/test.bz2:3+1
> count=99
> 2018-02-01 11:18:49,310 WARN [Thread-17] mapred.TestTextInputFormat
> (TestTextInputFormat.java:verifyPartitions(308)) - conflict with 1 in split 4
> at position 8
> {code}
>
> I experienced this error when I execute Spark (SparkSQL) job under the
> following conditions:
> * The file size of the input files are small (around 1KB)
> * Hadoop cluster has many slave nodes (able to launch many executor tasks)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15206) BZip2 drops and duplicates records when input split size is small
[
https://issues.apache.org/jira/browse/HADOOP-15206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367883#comment-16367883
]
Hudson commented on HADOOP-15206:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13673 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13673/])
HADOOP-15206. BZip2 drops and duplicates records when input split size (jlowe:
rev 0898ff42e9e5c53f2fce7ccdeb4e1cd7d0f123b3)
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestTextInputFormat.java
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/BZip2Codec.java
> BZip2 drops and duplicates records when input split size is small
> -
>
> Key: HADOOP-15206
> URL: https://issues.apache.org/jira/browse/HADOOP-15206
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.3, 3.0.0
>Reporter: Aki Tanaka
>Assignee: Aki Tanaka
>Priority: Major
> Attachments: HADOOP-15206-test.patch, HADOOP-15206.001.patch,
> HADOOP-15206.002.patch, HADOOP-15206.003.patch, HADOOP-15206.004.patch,
> HADOOP-15206.005.patch, HADOOP-15206.006.patch, HADOOP-15206.007.patch,
> HADOOP-15206.008.patch
>
>
> BZip2 can drop and duplicate record when input split file is small. I
> confirmed that this issue happens when the input split size is between 1byte
> and 4bytes.
> I am seeing the following 2 problem behaviors.
>
> 1. Drop record:
> BZip2 skips the first record in the input file when the input split size is
> small
>
> Set the split size to 3 and tested to load 100 records (0, 1, 2..99)
> {code:java}
> 2018-02-01 10:52:33,502 INFO [Thread-17] mapred.TestTextInputFormat
> (TestTextInputFormat.java:verifyPartitions(317)) -
> splits[1]=file:/work/count-mismatch2/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/TestTextInputFormat/test.bz2:3+3
> count=99{code}
> > The input format read only 99 records but not 100 records
>
> 2. Duplicate Record:
> 2 input splits has same BZip2 records when the input split size is small
>
> Set the split size to 1 and tested to load 100 records (0, 1, 2..99)
>
> {code:java}
> 2018-02-01 11:18:49,309 INFO [Thread-17] mapred.TestTextInputFormat
> (TestTextInputFormat.java:verifyPartitions(318)) - splits[3]=file
> /work/count-mismatch2/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/TestTextInputFormat/test.bz2:3+1
> count=99
> 2018-02-01 11:18:49,310 WARN [Thread-17] mapred.TestTextInputFormat
> (TestTextInputFormat.java:verifyPartitions(308)) - conflict with 1 in split 4
> at position 8
> {code}
>
> I experienced this error when I execute Spark (SparkSQL) job under the
> following conditions:
> * The file size of the input files are small (around 1KB)
> * Hadoop cluster has many slave nodes (able to launch many executor tasks)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15206) BZip2 drops and duplicates records when input split size is small
[
https://issues.apache.org/jira/browse/HADOOP-15206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367851#comment-16367851
]
Jason Lowe commented on HADOOP-15206:
-
The TestMRJobs failure is unrelated and tracked by MAPREDUCE-7053.
Committing this.
> BZip2 drops and duplicates records when input split size is small
> -
>
> Key: HADOOP-15206
> URL: https://issues.apache.org/jira/browse/HADOOP-15206
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.3, 3.0.0
>Reporter: Aki Tanaka
>Assignee: Aki Tanaka
>Priority: Major
> Attachments: HADOOP-15206-test.patch, HADOOP-15206.001.patch,
> HADOOP-15206.002.patch, HADOOP-15206.003.patch, HADOOP-15206.004.patch,
> HADOOP-15206.005.patch, HADOOP-15206.006.patch, HADOOP-15206.007.patch,
> HADOOP-15206.008.patch
>
>
> BZip2 can drop and duplicate record when input split file is small. I
> confirmed that this issue happens when the input split size is between 1byte
> and 4bytes.
> I am seeing the following 2 problem behaviors.
>
> 1. Drop record:
> BZip2 skips the first record in the input file when the input split size is
> small
>
> Set the split size to 3 and tested to load 100 records (0, 1, 2..99)
> {code:java}
> 2018-02-01 10:52:33,502 INFO [Thread-17] mapred.TestTextInputFormat
> (TestTextInputFormat.java:verifyPartitions(317)) -
> splits[1]=file:/work/count-mismatch2/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/TestTextInputFormat/test.bz2:3+3
> count=99{code}
> > The input format read only 99 records but not 100 records
>
> 2. Duplicate Record:
> 2 input splits has same BZip2 records when the input split size is small
>
> Set the split size to 1 and tested to load 100 records (0, 1, 2..99)
>
> {code:java}
> 2018-02-01 11:18:49,309 INFO [Thread-17] mapred.TestTextInputFormat
> (TestTextInputFormat.java:verifyPartitions(318)) - splits[3]=file
> /work/count-mismatch2/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/TestTextInputFormat/test.bz2:3+1
> count=99
> 2018-02-01 11:18:49,310 WARN [Thread-17] mapred.TestTextInputFormat
> (TestTextInputFormat.java:verifyPartitions(308)) - conflict with 1 in split 4
> at position 8
> {code}
>
> I experienced this error when I execute Spark (SparkSQL) job under the
> following conditions:
> * The file size of the input files are small (around 1KB)
> * Hadoop cluster has many slave nodes (able to launch many executor tasks)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9747) Reduce unnecessary UGI synchronization
[
https://issues.apache.org/jira/browse/HADOOP-9747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367836#comment-16367836
]
genericqa commented on HADOOP-9747:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 4s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
26s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 39s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 32 new + 189 unchanged - 27 fixed = 221 total (was 216) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 9s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
44s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-9747 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910955/HADOOP-9747-trunk-04.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux ce8a8dd47a28 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 4c2119f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14158/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14158/testReport/ |
| Max. process+thread count | 1647 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
[jira] [Commented] (HADOOP-15230) org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
[
https://issues.apache.org/jira/browse/HADOOP-15230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367671#comment-16367671
]
genericqa commented on HADOOP-15230:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 30s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} hadoop-common-project/hadoop-common: The patch
generated 0 new + 2 unchanged - 254 fixed = 2 total (was 256) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 6s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
16s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 87m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15230 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910948/HADOOP-15230.007.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 9af0b7bd73a4 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 82f029f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14157/testReport/ |
| Max. process+thread count | 1356 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14157/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> org.apache.hadoop.metrics2.GraphiteSink is not implemented
[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367654#comment-16367654 ] Steve Loughran commented on HADOOP-13761: - bq. was this part of your diff intentional or accident? kind of unrelated, but as the IDE was telling me off I fixed it > S3Guard: implement retries for DDB failures and throttling; translate > exceptions > > > Key: HADOOP-13761 > URL: https://issues.apache.org/jira/browse/HADOOP-13761 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Blocker > Attachments: HADOOP-13761-004-to-005.patch, > HADOOP-13761-005-to-006-approx.diff.txt, HADOOP-13761-005.patch, > HADOOP-13761-006.patch, HADOOP-13761.001.patch, HADOOP-13761.002.patch, > HADOOP-13761.003.patch, HADOOP-13761.004.patch > > > Following the S3AFileSystem integration patch in HADOOP-13651, we need to add > retry logic. > In HADOOP-13651, I added TODO comments in most of the places retry loops are > needed, including: > - open(path). If MetadataStore reflects recent create/move of file path, but > we fail to read it from S3, retry. > - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the > file exists, retry. > - rename(src,dest). If source path is not visible in S3 yet, retry. > - listFiles(). Skip for now. Not currently implemented in S3Guard. I will > create a separate JIRA for this as it will likely require interface changes > (i.e. prefix or subtree scan). > We may miss some cases initially and we should do failure injection testing > to make sure we're covered. Failure injection tests can be a separate JIRA > to make this easier to review. > We also need basic configuration parameters around retry policy. There > should be a way to specify maximum retry duration, as some applications would > prefer to receive an error eventually, than waiting indefinitely. We should > also be keeping statistics when inconsistency is detected and we enter a > retry loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[
https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367653#comment-16367653
]
Steve Loughran commented on HADOOP-13761:
-
Spent a lot of time staring at that read code, good to follow those failure
modes. I like you did with the fault injection.
What this adds is not just the s3guard FNFE handling, but the whole resilience
to read failures. Which is nice.
Some thoughts
* Mark up the seek/read calls with @RetryPolicy, so it's easier to see what
happens transitively
* only have the s3guardInvoker field set to S3GuardExistsRetryPolicy if S3Guard
is enabled, otherwise just set {{s3guardInvoker=invoker}}.
* and maybe just call it {{readInvoker}}
* then you can go {{context.readInvoker}} on those calls without much logic in
the input stream.
* the invoked operations themselves can be stuck inline
BTW, one thing I don't remember us having any test for is the code sequence
{code}
Path p = path();
writeTextFile(fs, p, "hello")
InputStream s = fs.open(path)
fs.delete(p)
intercept(FileNotFoundException.class, () -> s.read())
{code}
This would let us check that the deletion of a file does still get picked up in
a bounded-time when s3guard is turned on.
+ same for readfully().
A really clever variant of that test would be a test which did the delete after
the first read and before a seek we knew would trigger a new GET (such as a
backwards one). Again, we'd expect a failure with/without s3guard.
Finally, you know, we don't actually implement skip() ourselves in S3A, just do
a read + discard all the way to the end. Filed HADOOP-15245 to do that later.
Luckily, none of the columnar formats appear to use skip(), or, if anything
uses skip, it's doing it over short distances. Whatever gets done there should
think about resilience testing too.
> S3Guard: implement retries for DDB failures and throttling; translate
> exceptions
>
>
> Key: HADOOP-13761
> URL: https://issues.apache.org/jira/browse/HADOOP-13761
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Aaron Fabbri
>Assignee: Aaron Fabbri
>Priority: Blocker
> Attachments: HADOOP-13761-004-to-005.patch,
> HADOOP-13761-005-to-006-approx.diff.txt, HADOOP-13761-005.patch,
> HADOOP-13761-006.patch, HADOOP-13761.001.patch, HADOOP-13761.002.patch,
> HADOOP-13761.003.patch, HADOOP-13761.004.patch
>
>
> Following the S3AFileSystem integration patch in HADOOP-13651, we need to add
> retry logic.
> In HADOOP-13651, I added TODO comments in most of the places retry loops are
> needed, including:
> - open(path). If MetadataStore reflects recent create/move of file path, but
> we fail to read it from S3, retry.
> - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the
> file exists, retry.
> - rename(src,dest). If source path is not visible in S3 yet, retry.
> - listFiles(). Skip for now. Not currently implemented in S3Guard. I will
> create a separate JIRA for this as it will likely require interface changes
> (i.e. prefix or subtree scan).
> We may miss some cases initially and we should do failure injection testing
> to make sure we're covered. Failure injection tests can be a separate JIRA
> to make this easier to review.
> We also need basic configuration parameters around retry policy. There
> should be a way to specify maximum retry duration, as some applications would
> prefer to receive an error eventually, than waiting indefinitely. We should
> also be keeping statistics when inconsistency is detected and we enter a
> retry loop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9747) Reduce unnecessary UGI synchronization
[ https://issues.apache.org/jira/browse/HADOOP-9747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367627#comment-16367627 ] Daryn Sharp commented on HADOOP-9747: - A synchronize somehow crept back into loginUserFromKeytab. Removed it. I put back the previous invalid options for IBM_JAVA because it doesn't hurt anything and is arguably the least risk. Added missing @Test annotations. Updated javadoc for getLoginUser to be a bit more informative about what it does (actually has always done). Don't understand the concern over a race between getLoginUser and other login methods (which does exist and is handled) is helped by synchronization. Sure synchronize guards against 2 racing getLoginUser calls but not concurrent calls to the other login methods. Whereas the CAS logic ensures that if any login call beats us to setting the default login user, that winner will be returned instead. I unwound the code a bit (logic is identical) with better comments to better convey what's happening. > Reduce unnecessary UGI synchronization > -- > > Key: HADOOP-9747 > URL: https://issues.apache.org/jira/browse/HADOOP-9747 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0-alpha1 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Attachments: HADOOP-9747-trunk-03.patch, HADOOP-9747-trunk-04.patch, > HADOOP-9747-trunk.01.patch, HADOOP-9747-trunk.02.patch, > HADOOP-9747.2.branch-2.patch, HADOOP-9747.2.trunk.patch, > HADOOP-9747.branch-2.patch, HADOOP-9747.trunk.patch > > > Jstacks of heavily loaded NNs show up to dozens of threads blocking in the > UGI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9747) Reduce unnecessary UGI synchronization
[ https://issues.apache.org/jira/browse/HADOOP-9747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HADOOP-9747: Attachment: HADOOP-9747-trunk-04.patch > Reduce unnecessary UGI synchronization > -- > > Key: HADOOP-9747 > URL: https://issues.apache.org/jira/browse/HADOOP-9747 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0-alpha1 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Attachments: HADOOP-9747-trunk-03.patch, HADOOP-9747-trunk-04.patch, > HADOOP-9747-trunk.01.patch, HADOOP-9747-trunk.02.patch, > HADOOP-9747.2.branch-2.patch, HADOOP-9747.2.trunk.patch, > HADOOP-9747.branch-2.patch, HADOOP-9747.trunk.patch > > > Jstacks of heavily loaded NNs show up to dozens of threads blocking in the > UGI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15245) S3AInputStream.skip() to use lazy seek
Steve Loughran created HADOOP-15245: --- Summary: S3AInputStream.skip() to use lazy seek Key: HADOOP-15245 URL: https://issues.apache.org/jira/browse/HADOOP-15245 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.1.0 Reporter: Steve Loughran the default skip() does a read and discard of all bytes, no matter how far ahead the skip is. This is very inefficient if the skip() is being done on S3A random IO, though exactly what to do when in sequential mode. Proposed: * add an optimized version of S3AInputStream.skip() which does a lazy seek, which itself will decided when to skip() vs issue a new GET. * add some more instrumentation to measure how often this gets used -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13278) S3AFileSystem mkdirs does not need to validate parent path components
[
https://issues.apache.org/jira/browse/HADOOP-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367595#comment-16367595
]
Steve Loughran commented on HADOOP-13278:
-
HADOOP-15176 handles the IAM write problem, where the caller doesn't have the
permission to create/delete parent directory markers. That addresses the
situation where a client has restricted write access.
I don't think we can handle restricted read access, as it it at odds with the
normal world view of a filesystem and so much could break.
> S3AFileSystem mkdirs does not need to validate parent path components
> -
>
> Key: HADOOP-13278
> URL: https://issues.apache.org/jira/browse/HADOOP-13278
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3, tools
>Reporter: Adrian Petrescu
>Priority: Minor
>
> According to S3 semantics, there is no conflict if a bucket contains a key
> named {{a/b}} and also a directory named {{a/b/c}}. "Directories" in S3 are,
> after all, nothing but prefixes.
> However, the {{mkdirs}} call in {{S3AFileSystem}} does go out of its way to
> traverse every parent path component for the directory it's trying to create,
> making sure there's no file with that name. This is suboptimal for three main
> reasons:
> * Wasted API calls, since the client is getting metadata for each path
> component
> * This can cause *major* problems with buckets whose permissions are being
> managed by IAM, where access may not be granted to the root bucket, but only
> to some prefix. When you call {{mkdirs}}, even on a prefix that you have
> access to, the traversal up the path will cause you to eventually hit the
> root bucket, which will fail with a 403 - even though the directory creation
> call would have succeeded.
> * Some people might actually have a file that matches some other file's
> prefix... I can't see why they would want to do that, but it's not against
> S3's rules.
> I've opened a pull request with a simple patch that just removes this portion
> of the check. I have tested it with my team's instance of Spark + Luigi, and
> can confirm it works, and resolves the aforementioned permissions issue for a
> bucket on which we only had prefix access.
> This is my first ticket/pull request against Hadoop, so let me know if I'm
> not following some convention properly :)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14606) S3AInputStream: Handle http stream skip(n) skipping < n bytes in a forward seek
[
https://issues.apache.org/jira/browse/HADOOP-14606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-14606.
-
Resolution: Invalid
Fix Version/s: 3.1.0
Reviewing the code here, the problem doesn't exist. When we seek forwards we
look for the return value, update our position, then, if it it doesn't match
the expected number, close the stream and do a new GET.
> S3AInputStream: Handle http stream skip(n) skipping < n bytes in a forward
> seek
> ---
>
> Key: HADOOP-14606
> URL: https://issues.apache.org/jira/browse/HADOOP-14606
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Priority: Critical
> Fix For: 3.1.0
>
>
> There's some hints in the InputStream docs that {{skip(n)}} may skip bytes. Codepaths only seem to do this if read() returns -1, meaning end of
> stream is reached.
> If that happens when doing a forward seek via skip, then we have got our
> numbers wrong and are in trouble. Look for a negative response, log @ ERROR
> and revert to a close/reopen seek to an absolute position.
> *I have no evidence of this acutally occurring*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14507) extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key
[
https://issues.apache.org/jira/browse/HADOOP-14507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367587#comment-16367587
]
Hudson commented on HADOOP-14507:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13671 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13671/])
HADOOP-14507. Extend per-bucket secret key config with explicit (stevel: rev
7ac88244c54ce483729af3d2736d9f4731e230ca)
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ClientFactory.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/AssumedRoleCredentialProvider.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/SimpleAWSCredentialsProvider.java
* (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestAssumeRole.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/TemporaryAWSCredentialsProvider.java
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestSSEConfiguration.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
> extend per-bucket secret key config with explicit getPassword() on
> fs.s3a.$bucket.secret.key
>
>
> Key: HADOOP-14507
> URL: https://issues.apache.org/jira/browse/HADOOP-14507
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Fix For: 3.1.0
>
> Attachments: HADOOP-14507-001.patch, HADOOP-14507-002.patch,
> HADOOP-14507-003.patch, HADOOP-14507-004.patch, HADOOP-14507-005.patch,
> HADOOP-14507-006.patch, HADOOP-14507-006.patch, HADOOP-14507-007.patch,
> HADOOP-14507-008.patch
>
>
> Per-bucket jceks support turns out to be complex as you have to manage
> multiple jecks files & configure the client to ask for the right one. This is
> because we're calling {{Configuration.getPassword{"fs,s3a.secret.key"}}.
> If before that, we do a check for the explict id, key, session key in the
> properties {{fs.s3a.$bucket.secret}} ( & c), we could have a single JCEKs
> file with all the secrets for different bucket. You would only need to
> explicitly point the base config to the secrets file, and the right
> credentials would be picked up, if set
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15223) Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() when available
[
https://issues.apache.org/jira/browse/HADOOP-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367572#comment-16367572
]
genericqa commented on HADOOP-15223:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 7m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
19s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
20s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 7s{color} | {color:orange} root: The patch generated 1 new + 350 unchanged
- 0 fixed = 351 total (was 350) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 7m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 33s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 13m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 0s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
33s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
50s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
43s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
11s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
10s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 18m 41s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 62m
48s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
2s{color} |
[jira] [Updated] (HADOOP-14507) extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key
[
https://issues.apache.org/jira/browse/HADOOP-14507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14507:
Resolution: Fixed
Fix Version/s: 3.1.0
Status: Resolved (was: Patch Available)
> extend per-bucket secret key config with explicit getPassword() on
> fs.s3a.$bucket.secret.key
>
>
> Key: HADOOP-14507
> URL: https://issues.apache.org/jira/browse/HADOOP-14507
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Fix For: 3.1.0
>
> Attachments: HADOOP-14507-001.patch, HADOOP-14507-002.patch,
> HADOOP-14507-003.patch, HADOOP-14507-004.patch, HADOOP-14507-005.patch,
> HADOOP-14507-006.patch, HADOOP-14507-006.patch, HADOOP-14507-007.patch,
> HADOOP-14507-008.patch
>
>
> Per-bucket jceks support turns out to be complex as you have to manage
> multiple jecks files & configure the client to ask for the right one. This is
> because we're calling {{Configuration.getPassword{"fs,s3a.secret.key"}}.
> If before that, we do a check for the explict id, key, session key in the
> properties {{fs.s3a.$bucket.secret}} ( & c), we could have a single JCEKs
> file with all the secrets for different bucket. You would only need to
> explicitly point the base config to the secrets file, and the right
> credentials would be picked up, if set
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15230) org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
[
https://issues.apache.org/jira/browse/HADOOP-15230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Howard Yoo updated HADOOP-15230:
Attachment: (was: HADOOP-15230.006.patch)
> org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
>
>
> Key: HADOOP-15230
> URL: https://issues.apache.org/jira/browse/HADOOP-15230
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Reporter: Howard Yoo
>Priority: Major
> Labels: GraphiteSink,, metrics
> Attachments: HADOOP-15230.007.patch
>
>
> org.apache.hadoop.metrics2.GraphiteSink's implementation has certain problems
> that would make it to generate metrics incorrectly.
> The problem lies with line 77 ~ 84 of the GraphiteSink java:
> {code:java}
> for (MetricsTag tag : record.tags()) {
> if (tag.value() != null) {
> metricsPathPrefix.append(".");
> metricsPathPrefix.append(tag.name());
> metricsPathPrefix.append("=");
> metricsPathPrefix.append(tag.value());
> }
> }
> {code}
> It produces point tags having name=value pair in the metrics. However, notice
> how the tags are added with '.' as its delimiters. Rather than using the '.'
> character, it should follow the following convention mentioned in the latest
> graphite doc of using ';' character.
> [http://graphite.readthedocs.io/en/latest/tags.html]
> Also, the value is not properly being escaped, meaning that if the value has
> a '.' character in it, it will easily confuse Graphite to accept it as a
> delimiter, rather than the value. A really good prime example is when the
> value is a hostname or ip address,
> {code:java}
> metrics.example.Hostname=this.is.a.hostname.and.this.is.Metrics 10.0{code}
> In this example, the since the value of the hostname contains '.', it is
> extremely hard for the receiving end to determine which part is hostname and
> which part is the rest of the metrics name. A good strategy is to convert any
> '.' character in the value to be converted to other characters, such as '_'.
> However, the best way would be to follow the latest metrics convention of
> using ';'
> {code:java}
> metrics.example.and.this.is.Metrics;Hostname=this.is.a.hostname 10.0{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15230) org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
[
https://issues.apache.org/jira/browse/HADOOP-15230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Howard Yoo updated HADOOP-15230:
Attachment: HADOOP-15230.007.patch
> org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
>
>
> Key: HADOOP-15230
> URL: https://issues.apache.org/jira/browse/HADOOP-15230
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Reporter: Howard Yoo
>Priority: Major
> Labels: GraphiteSink,, metrics
> Attachments: HADOOP-15230.007.patch
>
>
> org.apache.hadoop.metrics2.GraphiteSink's implementation has certain problems
> that would make it to generate metrics incorrectly.
> The problem lies with line 77 ~ 84 of the GraphiteSink java:
> {code:java}
> for (MetricsTag tag : record.tags()) {
> if (tag.value() != null) {
> metricsPathPrefix.append(".");
> metricsPathPrefix.append(tag.name());
> metricsPathPrefix.append("=");
> metricsPathPrefix.append(tag.value());
> }
> }
> {code}
> It produces point tags having name=value pair in the metrics. However, notice
> how the tags are added with '.' as its delimiters. Rather than using the '.'
> character, it should follow the following convention mentioned in the latest
> graphite doc of using ';' character.
> [http://graphite.readthedocs.io/en/latest/tags.html]
> Also, the value is not properly being escaped, meaning that if the value has
> a '.' character in it, it will easily confuse Graphite to accept it as a
> delimiter, rather than the value. A really good prime example is when the
> value is a hostname or ip address,
> {code:java}
> metrics.example.Hostname=this.is.a.hostname.and.this.is.Metrics 10.0{code}
> In this example, the since the value of the hostname contains '.', it is
> extremely hard for the receiving end to determine which part is hostname and
> which part is the rest of the metrics name. A good strategy is to convert any
> '.' character in the value to be converted to other characters, such as '_'.
> However, the best way would be to follow the latest metrics convention of
> using ';'
> {code:java}
> metrics.example.and.this.is.Metrics;Hostname=this.is.a.hostname 10.0{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15244) s3guard uploads command to add a way to complete outstanding uploads
Steve Loughran created HADOOP-15244: --- Summary: s3guard uploads command to add a way to complete outstanding uploads Key: HADOOP-15244 URL: https://issues.apache.org/jira/browse/HADOOP-15244 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.1.0 Reporter: Steve Loughran The AWS API lets you not only list & cancel outstanding upload (as s3guard uploads) does, but actually list the parts. We may be able to actually complete an outstanding upload through the CLI. What would that do? It'd let you restore all but the last block of any logs being written to s3 where the app/VM failed before the upload was completed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14507) extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key
[
https://issues.apache.org/jira/browse/HADOOP-14507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367377#comment-16367377
]
genericqa commented on HADOOP-14507:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} hadoop-tools/hadoop-aws: The patch generated 0 new +
8 unchanged - 2 fixed = 8 total (was 10) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 9s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
31s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14507 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910927/HADOOP-14507-008.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux df4cc38cc7a5 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a1e05e0 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14155/artifact/out/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14155/testReport/ |
| Max. process+thread count | 338 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14155/console |
| Powered by | Apache Yetus
[jira] [Updated] (HADOOP-14943) Add common getFileBlockLocations() emulation for object stores, including S3A
[
https://issues.apache.org/jira/browse/HADOOP-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14943:
Parent Issue: HADOOP-15220 (was: HADOOP-14831)
> Add common getFileBlockLocations() emulation for object stores, including S3A
> -
>
> Key: HADOOP-14943
> URL: https://issues.apache.org/jira/browse/HADOOP-14943
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-14943-001.patch, HADOOP-14943-002.patch,
> HADOOP-14943-002.patch, HADOOP-14943-003.patch, HADOOP-14943-004.patch
>
>
> It looks suspiciously like S3A isn't providing the partitioning data needed
> in {{listLocatedStatus}} and {{getFileBlockLocations()}} needed to break up a
> file by the blocksize. This will stop tools using the MRv1 APIS doing the
> partitioning properly if the input format isn't doing it own split logic.
> FileInputFormat in MRv2 is a bit more configurable about input split
> calculation & will split up large files. but otherwise, the partitioning is
> being done more by the default values of the executing engine, rather than
> any config data from the filesystem about what its "block size" is,
> NativeAzureFS does a better job; maybe that could be factored out to
> hadoop-common and reused?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10571) Use Log.*(Object, Throwable) overload to log exceptions
[
https://issues.apache.org/jira/browse/HADOOP-10571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367318#comment-16367318
]
genericqa commented on HADOOP-10571:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
7s{color} | {color:red} Docker failed to build yetus/hadoop:tp-17753. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-10571 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910929/HADOOP-10571-branch-3.0.001.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14156/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Use Log.*(Object, Throwable) overload to log exceptions
> ---
>
> Key: HADOOP-10571
> URL: https://issues.apache.org/jira/browse/HADOOP-10571
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Arpit Agarwal
>Assignee: Andras Bokor
>Priority: Major
> Attachments: HADOOP-10571-branch-3.0.001.patch,
> HADOOP-10571.01.patch, HADOOP-10571.01.patch, HADOOP-10571.02.patch,
> HADOOP-10571.03.patch, HADOOP-10571.04.patch, HADOOP-10571.05.patch,
> HADOOP-10571.06.patch, HADOOP-10571.07.patch
>
>
> When logging an exception, we often convert the exception to string or call
> {{.getMessage}}. Instead we can use the log method overloads which take
> {{Throwable}} as a parameter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14943) Add common getFileBlockLocations() emulation for object stores, including S3A
[
https://issues.apache.org/jira/browse/HADOOP-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14943:
Target Version/s: (was: 3.1.0)
> Add common getFileBlockLocations() emulation for object stores, including S3A
> -
>
> Key: HADOOP-14943
> URL: https://issues.apache.org/jira/browse/HADOOP-14943
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-14943-001.patch, HADOOP-14943-002.patch,
> HADOOP-14943-002.patch, HADOOP-14943-003.patch, HADOOP-14943-004.patch
>
>
> It looks suspiciously like S3A isn't providing the partitioning data needed
> in {{listLocatedStatus}} and {{getFileBlockLocations()}} needed to break up a
> file by the blocksize. This will stop tools using the MRv1 APIS doing the
> partitioning properly if the input format isn't doing it own split logic.
> FileInputFormat in MRv2 is a bit more configurable about input split
> calculation & will split up large files. but otherwise, the partitioning is
> being done more by the default values of the executing engine, rather than
> any config data from the filesystem about what its "block size" is,
> NativeAzureFS does a better job; maybe that could be factored out to
> hadoop-common and reused?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-10571) Use Log.*(Object, Throwable) overload to log exceptions
[
https://issues.apache.org/jira/browse/HADOOP-10571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Bokor updated HADOOP-10571:
--
Attachment: (was: HADOOP-10571-branch-3.0.01.patch)
> Use Log.*(Object, Throwable) overload to log exceptions
> ---
>
> Key: HADOOP-10571
> URL: https://issues.apache.org/jira/browse/HADOOP-10571
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Arpit Agarwal
>Assignee: Andras Bokor
>Priority: Major
> Attachments: HADOOP-10571-branch-3.0.001.patch,
> HADOOP-10571.01.patch, HADOOP-10571.01.patch, HADOOP-10571.02.patch,
> HADOOP-10571.03.patch, HADOOP-10571.04.patch, HADOOP-10571.05.patch,
> HADOOP-10571.06.patch, HADOOP-10571.07.patch
>
>
> When logging an exception, we often convert the exception to string or call
> {{.getMessage}}. Instead we can use the log method overloads which take
> {{Throwable}} as a parameter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14903) Add json-smart explicitly to pom.xml
[
https://issues.apache.org/jira/browse/HADOOP-14903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367308#comment-16367308
]
genericqa commented on HADOOP-14903:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
19s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
52s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
9s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
32s{color} | {color:green} branch-2 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
29s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
35s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
24s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 15s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 47m 54s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}143m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Unreaped Processes | hadoop-common:1 |
| | hadoop-hdfs:12 |
| Failed junit tests | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.TestRestartDFS |
| Timed out junit tests | org.apache.hadoop.log.TestLogLevel |
| | org.apache.hadoop.hdfs.TestDatanodeRegistration |
| | org.apache.hadoop.hdfs.TestDatanodeDeath |
| | org.apache.hadoop.hdfs.TestFileAppend2 |
| | org.apache.hadoop.hdfs.TestSeekBug |
| | org.apache.hadoop.hdfs.TestDFSInputStream |
| | org.apache.hadoop.hdfs.tools.TestGetGroups |
| | org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache |
| |
[jira] [Updated] (HADOOP-10571) Use Log.*(Object, Throwable) overload to log exceptions
[
https://issues.apache.org/jira/browse/HADOOP-10571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Bokor updated HADOOP-10571:
--
Attachment: HADOOP-10571-branch-3.0.001.patch
> Use Log.*(Object, Throwable) overload to log exceptions
> ---
>
> Key: HADOOP-10571
> URL: https://issues.apache.org/jira/browse/HADOOP-10571
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Arpit Agarwal
>Assignee: Andras Bokor
>Priority: Major
> Attachments: HADOOP-10571-branch-3.0.001.patch,
> HADOOP-10571.01.patch, HADOOP-10571.01.patch, HADOOP-10571.02.patch,
> HADOOP-10571.03.patch, HADOOP-10571.04.patch, HADOOP-10571.05.patch,
> HADOOP-10571.06.patch, HADOOP-10571.07.patch
>
>
> When logging an exception, we often convert the exception to string or call
> {{.getMessage}}. Instead we can use the log method overloads which take
> {{Throwable}} as a parameter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-10571) Use Log.*(Object, Throwable) overload to log exceptions
[
https://issues.apache.org/jira/browse/HADOOP-10571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Bokor updated HADOOP-10571:
--
Attachment: (was: HADOOP-10571-brach-3.0.01.patch)
> Use Log.*(Object, Throwable) overload to log exceptions
> ---
>
> Key: HADOOP-10571
> URL: https://issues.apache.org/jira/browse/HADOOP-10571
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Arpit Agarwal
>Assignee: Andras Bokor
>Priority: Major
> Attachments: HADOOP-10571-branch-3.0.01.patch, HADOOP-10571.01.patch,
> HADOOP-10571.01.patch, HADOOP-10571.02.patch, HADOOP-10571.03.patch,
> HADOOP-10571.04.patch, HADOOP-10571.05.patch, HADOOP-10571.06.patch,
> HADOOP-10571.07.patch
>
>
> When logging an exception, we often convert the exception to string or call
> {{.getMessage}}. Instead we can use the log method overloads which take
> {{Throwable}} as a parameter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-10571) Use Log.*(Object, Throwable) overload to log exceptions
[
https://issues.apache.org/jira/browse/HADOOP-10571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Bokor updated HADOOP-10571:
--
Attachment: HADOOP-10571-branch-3.0.01.patch
> Use Log.*(Object, Throwable) overload to log exceptions
> ---
>
> Key: HADOOP-10571
> URL: https://issues.apache.org/jira/browse/HADOOP-10571
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Arpit Agarwal
>Assignee: Andras Bokor
>Priority: Major
> Attachments: HADOOP-10571-branch-3.0.01.patch, HADOOP-10571.01.patch,
> HADOOP-10571.01.patch, HADOOP-10571.02.patch, HADOOP-10571.03.patch,
> HADOOP-10571.04.patch, HADOOP-10571.05.patch, HADOOP-10571.06.patch,
> HADOOP-10571.07.patch
>
>
> When logging an exception, we often convert the exception to string or call
> {{.getMessage}}. Instead we can use the log method overloads which take
> {{Throwable}} as a parameter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14507) extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key
[
https://issues.apache.org/jira/browse/HADOOP-14507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14507:
Status: Patch Available (was: Open)
Patch 008; sync up with trunk & a moved ITestAssumeRole. Once Yetus is happy,
this goes on.
Testing. S3 ireland w/ {{-Dparallel-tests -DtestsThreadCount=8 -Ds3guard
-Ddynamodb -Dauth}}
> extend per-bucket secret key config with explicit getPassword() on
> fs.s3a.$bucket.secret.key
>
>
> Key: HADOOP-14507
> URL: https://issues.apache.org/jira/browse/HADOOP-14507
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Attachments: HADOOP-14507-001.patch, HADOOP-14507-002.patch,
> HADOOP-14507-003.patch, HADOOP-14507-004.patch, HADOOP-14507-005.patch,
> HADOOP-14507-006.patch, HADOOP-14507-006.patch, HADOOP-14507-007.patch,
> HADOOP-14507-008.patch
>
>
> Per-bucket jceks support turns out to be complex as you have to manage
> multiple jecks files & configure the client to ask for the right one. This is
> because we're calling {{Configuration.getPassword{"fs,s3a.secret.key"}}.
> If before that, we do a check for the explict id, key, session key in the
> properties {{fs.s3a.$bucket.secret}} ( & c), we could have a single JCEKs
> file with all the secrets for different bucket. You would only need to
> explicitly point the base config to the secrets file, and the right
> credentials would be picked up, if set
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14507) extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key
[
https://issues.apache.org/jira/browse/HADOOP-14507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14507:
Attachment: HADOOP-14507-008.patch
> extend per-bucket secret key config with explicit getPassword() on
> fs.s3a.$bucket.secret.key
>
>
> Key: HADOOP-14507
> URL: https://issues.apache.org/jira/browse/HADOOP-14507
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Attachments: HADOOP-14507-001.patch, HADOOP-14507-002.patch,
> HADOOP-14507-003.patch, HADOOP-14507-004.patch, HADOOP-14507-005.patch,
> HADOOP-14507-006.patch, HADOOP-14507-006.patch, HADOOP-14507-007.patch,
> HADOOP-14507-008.patch
>
>
> Per-bucket jceks support turns out to be complex as you have to manage
> multiple jecks files & configure the client to ask for the right one. This is
> because we're calling {{Configuration.getPassword{"fs,s3a.secret.key"}}.
> If before that, we do a check for the explict id, key, session key in the
> properties {{fs.s3a.$bucket.secret}} ( & c), we could have a single JCEKs
> file with all the secrets for different bucket. You would only need to
> explicitly point the base config to the secrets file, and the right
> credentials would be picked up, if set
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14507) extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key
[
https://issues.apache.org/jira/browse/HADOOP-14507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14507:
Status: Open (was: Patch Available)
> extend per-bucket secret key config with explicit getPassword() on
> fs.s3a.$bucket.secret.key
>
>
> Key: HADOOP-14507
> URL: https://issues.apache.org/jira/browse/HADOOP-14507
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Attachments: HADOOP-14507-001.patch, HADOOP-14507-002.patch,
> HADOOP-14507-003.patch, HADOOP-14507-004.patch, HADOOP-14507-005.patch,
> HADOOP-14507-006.patch, HADOOP-14507-006.patch, HADOOP-14507-007.patch
>
>
> Per-bucket jceks support turns out to be complex as you have to manage
> multiple jecks files & configure the client to ask for the right one. This is
> because we're calling {{Configuration.getPassword{"fs,s3a.secret.key"}}.
> If before that, we do a check for the explict id, key, session key in the
> properties {{fs.s3a.$bucket.secret}} ( & c), we could have a single JCEKs
> file with all the secrets for different bucket. You would only need to
> explicitly point the base config to the secrets file, and the right
> credentials would be picked up, if set
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10571) Use Log.*(Object, Throwable) overload to log exceptions
[
https://issues.apache.org/jira/browse/HADOOP-10571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367289#comment-16367289
]
genericqa commented on HADOOP-10571:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HADOOP-10571 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-10571 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910925/HADOOP-10571-brach-3.0.01.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14154/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Use Log.*(Object, Throwable) overload to log exceptions
> ---
>
> Key: HADOOP-10571
> URL: https://issues.apache.org/jira/browse/HADOOP-10571
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Arpit Agarwal
>Assignee: Andras Bokor
>Priority: Major
> Attachments: HADOOP-10571-brach-3.0.01.patch, HADOOP-10571.01.patch,
> HADOOP-10571.01.patch, HADOOP-10571.02.patch, HADOOP-10571.03.patch,
> HADOOP-10571.04.patch, HADOOP-10571.05.patch, HADOOP-10571.06.patch,
> HADOOP-10571.07.patch
>
>
> When logging an exception, we often convert the exception to string or call
> {{.getMessage}}. Instead we can use the log method overloads which take
> {{Throwable}} as a parameter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-10571) Use Log.*(Object, Throwable) overload to log exceptions
[
https://issues.apache.org/jira/browse/HADOOP-10571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Bokor updated HADOOP-10571:
--
Attachment: HADOOP-10571-brach-3.0.01.patch
> Use Log.*(Object, Throwable) overload to log exceptions
> ---
>
> Key: HADOOP-10571
> URL: https://issues.apache.org/jira/browse/HADOOP-10571
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Arpit Agarwal
>Assignee: Andras Bokor
>Priority: Major
> Attachments: HADOOP-10571-brach-3.0.01.patch, HADOOP-10571.01.patch,
> HADOOP-10571.01.patch, HADOOP-10571.02.patch, HADOOP-10571.03.patch,
> HADOOP-10571.04.patch, HADOOP-10571.05.patch, HADOOP-10571.06.patch,
> HADOOP-10571.07.patch
>
>
> When logging an exception, we often convert the exception to string or call
> {{.getMessage}}. Instead we can use the log method overloads which take
> {{Throwable}} as a parameter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14507) extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key
[
https://issues.apache.org/jira/browse/HADOOP-14507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367147#comment-16367147
]
Steve Loughran commented on HADOOP-14507:
-
thx for the +1, some user called "stevel" has managed to break my own merge by
moving ITestAssumedRole in HADOOP-15786; I'll submit the final patch through
yetus for the final check before committing. Whoever this stevel is, I wished
they'd stop breaking my patches.
> extend per-bucket secret key config with explicit getPassword() on
> fs.s3a.$bucket.secret.key
>
>
> Key: HADOOP-14507
> URL: https://issues.apache.org/jira/browse/HADOOP-14507
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Attachments: HADOOP-14507-001.patch, HADOOP-14507-002.patch,
> HADOOP-14507-003.patch, HADOOP-14507-004.patch, HADOOP-14507-005.patch,
> HADOOP-14507-006.patch, HADOOP-14507-006.patch, HADOOP-14507-007.patch
>
>
> Per-bucket jceks support turns out to be complex as you have to manage
> multiple jecks files & configure the client to ask for the right one. This is
> because we're calling {{Configuration.getPassword{"fs,s3a.secret.key"}}.
> If before that, we do a check for the explict id, key, session key in the
> properties {{fs.s3a.$bucket.secret}} ( & c), we could have a single JCEKs
> file with all the secrets for different bucket. You would only need to
> explicitly point the base config to the secrets file, and the right
> credentials would be picked up, if set
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15243) test and document use of fs.s3a.signing-algorithm
Steve Loughran created HADOOP-15243:
---
Summary: test and document use of fs.s3a.signing-algorithm
Key: HADOOP-15243
URL: https://issues.apache.org/jira/browse/HADOOP-15243
Project: Hadoop Common
Issue Type: Sub-task
Components: documentation, fs/s3, test
Reporter: Steve Loughran
we suggest changing of {{fs.s3a.signing-algorithm}} as a way to deal with some
auth problems, but there's no standalone section of what is available, or any
tests trying to switch to different methods.
This is hampered by the fact that none of us have owned up to understanding all
the choices available and the AWS docs not covering it...all we have is the
source which sets up the options in >1 place
Proposed:
* add some tests of the actual valid options.
* document the values in the s3a docs
* and in core-default.xml
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15223) Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() when available
[
https://issues.apache.org/jira/browse/HADOOP-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
fang zhenyi updated HADOOP-15223:
-
Status: Patch Available (was: In Progress)
> Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap()
> when available
> -
>
> Key: HADOOP-15223
> URL: https://issues.apache.org/jira/browse/HADOOP-15223
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: fang zhenyi
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-15223.001.patch, HADOOP-15223.002.patch,
> HADOOP-15223.003.patch, HADOOP-15223.004.patch
>
>
> The use of {{Collections.EMPTY_SET}} and {{Collections.EMPTY_MAP}} often
> causes unchecked assignment and it should be replaced with
> {{Collections.emptySet()}} and {{Collections.emptyMap()}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15223) Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() when available
[
https://issues.apache.org/jira/browse/HADOOP-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
fang zhenyi updated HADOOP-15223:
-
Status: In Progress (was: Patch Available)
> Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap()
> when available
> -
>
> Key: HADOOP-15223
> URL: https://issues.apache.org/jira/browse/HADOOP-15223
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: fang zhenyi
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-15223.001.patch, HADOOP-15223.002.patch,
> HADOOP-15223.003.patch, HADOOP-15223.004.patch
>
>
> The use of {{Collections.EMPTY_SET}} and {{Collections.EMPTY_MAP}} often
> causes unchecked assignment and it should be replaced with
> {{Collections.emptySet()}} and {{Collections.emptyMap()}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15223) Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() when available
[
https://issues.apache.org/jira/browse/HADOOP-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366918#comment-16366918
]
fang zhenyi commented on HADOOP-15223:
--
Thanks [~ajisakaa] for review.
I have replaced {{Collections.EMPTY_MAP}} in
SingleConstraintAppPlacementAllocator.java.Hope you can review again, thanks a
lot.
> Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap()
> when available
> -
>
> Key: HADOOP-15223
> URL: https://issues.apache.org/jira/browse/HADOOP-15223
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: fang zhenyi
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-15223.001.patch, HADOOP-15223.002.patch,
> HADOOP-15223.003.patch, HADOOP-15223.004.patch
>
>
> The use of {{Collections.EMPTY_SET}} and {{Collections.EMPTY_MAP}} often
> causes unchecked assignment and it should be replaced with
> {{Collections.emptySet()}} and {{Collections.emptyMap()}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15223) Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() when available
[
https://issues.apache.org/jira/browse/HADOOP-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
fang zhenyi updated HADOOP-15223:
-
Status: Patch Available (was: In Progress)
> Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap()
> when available
> -
>
> Key: HADOOP-15223
> URL: https://issues.apache.org/jira/browse/HADOOP-15223
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: fang zhenyi
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-15223.001.patch, HADOOP-15223.002.patch,
> HADOOP-15223.003.patch, HADOOP-15223.004.patch
>
>
> The use of {{Collections.EMPTY_SET}} and {{Collections.EMPTY_MAP}} often
> causes unchecked assignment and it should be replaced with
> {{Collections.emptySet()}} and {{Collections.emptyMap()}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15223) Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() when available
[
https://issues.apache.org/jira/browse/HADOOP-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
fang zhenyi updated HADOOP-15223:
-
Attachment: HADOOP-15223.004.patch.patch
> Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap()
> when available
> -
>
> Key: HADOOP-15223
> URL: https://issues.apache.org/jira/browse/HADOOP-15223
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: fang zhenyi
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-15223.001.patch, HADOOP-15223.002.patch,
> HADOOP-15223.003.patch, HADOOP-15223.004.patch
>
>
> The use of {{Collections.EMPTY_SET}} and {{Collections.EMPTY_MAP}} often
> causes unchecked assignment and it should be replaced with
> {{Collections.emptySet()}} and {{Collections.emptyMap()}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15223) Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() when available
[
https://issues.apache.org/jira/browse/HADOOP-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
fang zhenyi updated HADOOP-15223:
-
Attachment: HADOOP-15223.004.patch
> Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap()
> when available
> -
>
> Key: HADOOP-15223
> URL: https://issues.apache.org/jira/browse/HADOOP-15223
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: fang zhenyi
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-15223.001.patch, HADOOP-15223.002.patch,
> HADOOP-15223.003.patch, HADOOP-15223.004.patch
>
>
> The use of {{Collections.EMPTY_SET}} and {{Collections.EMPTY_MAP}} often
> causes unchecked assignment and it should be replaced with
> {{Collections.emptySet()}} and {{Collections.emptyMap()}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15223) Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() when available
[
https://issues.apache.org/jira/browse/HADOOP-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
fang zhenyi updated HADOOP-15223:
-
Status: In Progress (was: Patch Available)
> Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap()
> when available
> -
>
> Key: HADOOP-15223
> URL: https://issues.apache.org/jira/browse/HADOOP-15223
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: fang zhenyi
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-15223.001.patch, HADOOP-15223.002.patch,
> HADOOP-15223.003.patch, HADOOP-15223.004.patch
>
>
> The use of {{Collections.EMPTY_SET}} and {{Collections.EMPTY_MAP}} often
> causes unchecked assignment and it should be replaced with
> {{Collections.emptySet()}} and {{Collections.emptyMap()}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15223) Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() when available
[
https://issues.apache.org/jira/browse/HADOOP-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
fang zhenyi updated HADOOP-15223:
-
Attachment: (was: HADOOP-15223.004.patch.patch)
> Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap()
> when available
> -
>
> Key: HADOOP-15223
> URL: https://issues.apache.org/jira/browse/HADOOP-15223
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: fang zhenyi
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-15223.001.patch, HADOOP-15223.002.patch,
> HADOOP-15223.003.patch, HADOOP-15223.004.patch
>
>
> The use of {{Collections.EMPTY_SET}} and {{Collections.EMPTY_MAP}} often
> causes unchecked assignment and it should be replaced with
> {{Collections.emptySet()}} and {{Collections.emptyMap()}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15129) Datanode caches namenode DNS lookup failure and cannot startup
[
https://issues.apache.org/jira/browse/HADOOP-15129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366892#comment-16366892
]
Steve Loughran commented on HADOOP-15129:
-
Could always add a specific failure call and wiki link here
> Datanode caches namenode DNS lookup failure and cannot startup
> --
>
> Key: HADOOP-15129
> URL: https://issues.apache.org/jira/browse/HADOOP-15129
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.8.2
> Environment: Google Compute Engine.
> I'm using Java 8, Debian 8, Hadoop 2.8.2.
>Reporter: Karthik Palaniappan
>Assignee: Karthik Palaniappan
>Priority: Minor
> Attachments: HADOOP-15129.001.patch, HADOOP-15129.002.patch
>
>
> On startup, the Datanode creates an InetSocketAddress to register with each
> namenode. Though there are retries on connection failure throughout the
> stack, the same InetSocketAddress is reused.
> InetSocketAddress is an interesting class, because it resolves DNS names to
> IP addresses on construction, and it is never refreshed. Hadoop re-creates an
> InetSocketAddress in some cases just in case the remote IP has changed for a
> particular DNS name: https://issues.apache.org/jira/browse/HADOOP-7472.
> Anyway, on startup, you cna see the Datanode log: "Namenode...remains
> unresolved" -- referring to the fact that DNS lookup failed.
> {code:java}
> 2017-11-02 16:01:55,115 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Refresh request received for nameservices: null
> 2017-11-02 16:01:55,153 WARN org.apache.hadoop.hdfs.DFSUtilClient: Namenode
> for null remains unresolved for ID null. Check your hdfs-site.xml file to
> ensure namenodes are configured properly.
> 2017-11-02 16:01:55,156 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Starting BPOfferServices for nameservices:
> 2017-11-02 16:01:55,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Block pool (Datanode Uuid unassigned) service to
> cluster-32f5-m:8020 starting to offer service
> {code}
> The Datanode then proceeds to use this unresolved address, as it may work if
> the DN is configured to use a proxy. Since I'm not using a proxy, it forever
> prints out this message:
> {code:java}
> 2017-12-15 00:13:40,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:45,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:50,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:55,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:14:00,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> {code}
> Unfortunately, the log doesn't contain the exception that triggered it, but
> the culprit is actually in IPC Client:
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L444.
> This line was introduced in https://issues.apache.org/jira/browse/HADOOP-487
> to give a clear error message when somebody mispells an address.
> However, the fix in HADOOP-7472 doesn't apply here, because that code happens
> in Client#getConnection after the Connection is constructed.
> My proposed fix (will attach a patch) is to move this exception out of the
> constructor and into a place that will trigger HADOOP-7472's logic to
> re-resolve addresses. If the DNS failure was temporary, this will allow the
> connection to succeed. If not, the connection will fail after ipc client
> retries (default 10 seconds worth of retries).
> I want to fix this in ipc client rather than just in Datanode startup, as
> this fixes temporary DNS issues for all of Hadoop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14507) extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key
[
https://issues.apache.org/jira/browse/HADOOP-14507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366891#comment-16366891
]
Steve Loughran commented on HADOOP-14507:
-
bq. If I understand properly that the fs.s3a.server-side-encryption.key is a
name into another KMS for the actual key than I am fine with it.
it can be both. We can support an explicit option for keys for SSE-C and
client-side in future
> extend per-bucket secret key config with explicit getPassword() on
> fs.s3a.$bucket.secret.key
>
>
> Key: HADOOP-14507
> URL: https://issues.apache.org/jira/browse/HADOOP-14507
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Attachments: HADOOP-14507-001.patch, HADOOP-14507-002.patch,
> HADOOP-14507-003.patch, HADOOP-14507-004.patch, HADOOP-14507-005.patch,
> HADOOP-14507-006.patch, HADOOP-14507-006.patch, HADOOP-14507-007.patch
>
>
> Per-bucket jceks support turns out to be complex as you have to manage
> multiple jecks files & configure the client to ask for the right one. This is
> because we're calling {{Configuration.getPassword{"fs,s3a.secret.key"}}.
> If before that, we do a check for the explict id, key, session key in the
> properties {{fs.s3a.$bucket.secret}} ( & c), we could have a single JCEKs
> file with all the secrets for different bucket. You would only need to
> explicitly point the base config to the secrets file, and the right
> credentials would be picked up, if set
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14903) Add json-smart explicitly to pom.xml
[
https://issues.apache.org/jira/browse/HADOOP-14903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366858#comment-16366858
]
Brahma Reddy Battula commented on HADOOP-14903:
---
Testcases are not ran.Uploading the patch to execute the testcases.
> Add json-smart explicitly to pom.xml
>
>
> Key: HADOOP-14903
> URL: https://issues.apache.org/jira/browse/HADOOP-14903
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.0.0-beta1
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>Priority: Major
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14903-003-branch-2.patch,
> HADOOP-14903-branch-2-003.patch,
> HADOOP-14903-branch-2-004-ForExecutingTests.patch,
> HADOOP-14903-branch-2-004.patch, HADOOP-14903.001.patch,
> HADOOP-14903.002.patch, HADOOP-14903.003.patch
>
>
> With the library update in HADOOP-14799, maven knows how to pull in
> net.minidev:json-smart for tests, but not for packaging. This needs to be
> added to the main project pom in order to avoid this warning:
> {noformat}
> [WARNING] The POM for net.minidev:json-smart:jar:2.3-SNAPSHOT is missing, no
> dependency information available
> {noformat}
> This is pulled in from a few places:
> {noformat}
> [INFO] | +- org.apache.hadoop:hadoop-auth:jar:3.1.0-SNAPSHOT:compile
> [INFO] | | +- com.nimbusds:nimbus-jose-jwt:jar:4.41.1:compile
> [INFO] | | | +- com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
> [INFO] | | | \- net.minidev:json-smart:jar:2.3:compile
> [INFO] | | \- org.apache.kerby:token-provider:jar:1.0.1:compile
> [INFO] | | \- com.nimbusds:nimbus-jose-jwt:jar:4.41.1:compile
> [INFO] | |+-
> com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
> [INFO] | |\- net.minidev:json-smart:jar:2.3:compile
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14903) Add json-smart explicitly to pom.xml
[
https://issues.apache.org/jira/browse/HADOOP-14903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HADOOP-14903:
--
Attachment: HADOOP-14903-branch-2-004-ForExecutingTests.patch
> Add json-smart explicitly to pom.xml
>
>
> Key: HADOOP-14903
> URL: https://issues.apache.org/jira/browse/HADOOP-14903
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.0.0-beta1
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>Priority: Major
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14903-003-branch-2.patch,
> HADOOP-14903-branch-2-003.patch,
> HADOOP-14903-branch-2-004-ForExecutingTests.patch,
> HADOOP-14903-branch-2-004.patch, HADOOP-14903.001.patch,
> HADOOP-14903.002.patch, HADOOP-14903.003.patch
>
>
> With the library update in HADOOP-14799, maven knows how to pull in
> net.minidev:json-smart for tests, but not for packaging. This needs to be
> added to the main project pom in order to avoid this warning:
> {noformat}
> [WARNING] The POM for net.minidev:json-smart:jar:2.3-SNAPSHOT is missing, no
> dependency information available
> {noformat}
> This is pulled in from a few places:
> {noformat}
> [INFO] | +- org.apache.hadoop:hadoop-auth:jar:3.1.0-SNAPSHOT:compile
> [INFO] | | +- com.nimbusds:nimbus-jose-jwt:jar:4.41.1:compile
> [INFO] | | | +- com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
> [INFO] | | | \- net.minidev:json-smart:jar:2.3:compile
> [INFO] | | \- org.apache.kerby:token-provider:jar:1.0.1:compile
> [INFO] | | \- com.nimbusds:nimbus-jose-jwt:jar:4.41.1:compile
> [INFO] | |+-
> com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
> [INFO] | |\- net.minidev:json-smart:jar:2.3:compile
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14163) Refactor existing hadoop site to use more usable static website generator
[
https://issues.apache.org/jira/browse/HADOOP-14163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366846#comment-16366846
]
genericqa commented on HADOOP-14163:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HADOOP-14163 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-14163 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910915/HADOOP-14163.009.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14151/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Refactor existing hadoop site to use more usable static website generator
> -
>
> Key: HADOOP-14163
> URL: https://issues.apache.org/jira/browse/HADOOP-14163
> Project: Hadoop Common
> Issue Type: Improvement
> Components: site
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HADOOP-14163-001.zip, HADOOP-14163-002.zip,
> HADOOP-14163-003.zip, HADOOP-14163.004.patch, HADOOP-14163.005.patch,
> HADOOP-14163.006.patch, HADOOP-14163.007.patch, HADOOP-14163.008.tar.gz,
> HADOOP-14163.009.patch, HADOOP-14163.009.tar.gz, hadoop-site.tar.gz,
> hadop-site-rendered.tar.gz
>
>
> From the dev mailing list:
> "Publishing can be attacked via a mix of scripting and revamping the darned
> website. Forrest is pretty bad compared to the newer static site generators
> out there (e.g. need to write XML instead of markdown, it's hard to review a
> staging site because of all the absolute links, hard to customize, did I
> mention XML?), and the look and feel of the site is from the 00s. We don't
> actually have that much site content, so it should be possible to migrate to
> a new system."
> This issue is find a solution to migrate the old site to a new modern static
> site generator using a more contemprary theme.
> Goals:
> * existing links should work (or at least redirected)
> * It should be easy to add more content required by a release automatically
> (most probably with creating separated markdown files)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14163) Refactor existing hadoop site to use more usable static website generator
[ https://issues.apache.org/jira/browse/HADOOP-14163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366843#comment-16366843 ] Elek, Marton commented on HADOOP-14163: --- I uploaded the latest version with the recent releases and receng commiter/pmc changes. Please let me know if you have any other concerns/questions. > Refactor existing hadoop site to use more usable static website generator > - > > Key: HADOOP-14163 > URL: https://issues.apache.org/jira/browse/HADOOP-14163 > Project: Hadoop Common > Issue Type: Improvement > Components: site >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-14163-001.zip, HADOOP-14163-002.zip, > HADOOP-14163-003.zip, HADOOP-14163.004.patch, HADOOP-14163.005.patch, > HADOOP-14163.006.patch, HADOOP-14163.007.patch, HADOOP-14163.008.tar.gz, > HADOOP-14163.009.patch, HADOOP-14163.009.tar.gz, hadoop-site.tar.gz, > hadop-site-rendered.tar.gz > > > From the dev mailing list: > "Publishing can be attacked via a mix of scripting and revamping the darned > website. Forrest is pretty bad compared to the newer static site generators > out there (e.g. need to write XML instead of markdown, it's hard to review a > staging site because of all the absolute links, hard to customize, did I > mention XML?), and the look and feel of the site is from the 00s. We don't > actually have that much site content, so it should be possible to migrate to > a new system." > This issue is find a solution to migrate the old site to a new modern static > site generator using a more contemprary theme. > Goals: > * existing links should work (or at least redirected) > * It should be easy to add more content required by a release automatically > (most probably with creating separated markdown files) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14163) Refactor existing hadoop site to use more usable static website generator
[ https://issues.apache.org/jira/browse/HADOOP-14163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-14163: -- Attachment: HDFS-13149-HDFS-7240.001.patch > Refactor existing hadoop site to use more usable static website generator > - > > Key: HADOOP-14163 > URL: https://issues.apache.org/jira/browse/HADOOP-14163 > Project: Hadoop Common > Issue Type: Improvement > Components: site >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-14163-001.zip, HADOOP-14163-002.zip, > HADOOP-14163-003.zip, HADOOP-14163.004.patch, HADOOP-14163.005.patch, > HADOOP-14163.006.patch, HADOOP-14163.007.patch, HADOOP-14163.008.tar.gz, > HADOOP-14163.009.patch, HADOOP-14163.009.tar.gz, hadoop-site.tar.gz, > hadop-site-rendered.tar.gz > > > From the dev mailing list: > "Publishing can be attacked via a mix of scripting and revamping the darned > website. Forrest is pretty bad compared to the newer static site generators > out there (e.g. need to write XML instead of markdown, it's hard to review a > staging site because of all the absolute links, hard to customize, did I > mention XML?), and the look and feel of the site is from the 00s. We don't > actually have that much site content, so it should be possible to migrate to > a new system." > This issue is find a solution to migrate the old site to a new modern static > site generator using a more contemprary theme. > Goals: > * existing links should work (or at least redirected) > * It should be easy to add more content required by a release automatically > (most probably with creating separated markdown files) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14163) Refactor existing hadoop site to use more usable static website generator
[ https://issues.apache.org/jira/browse/HADOOP-14163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-14163: -- Attachment: HADOOP-14163.009.patch > Refactor existing hadoop site to use more usable static website generator > - > > Key: HADOOP-14163 > URL: https://issues.apache.org/jira/browse/HADOOP-14163 > Project: Hadoop Common > Issue Type: Improvement > Components: site >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-14163-001.zip, HADOOP-14163-002.zip, > HADOOP-14163-003.zip, HADOOP-14163.004.patch, HADOOP-14163.005.patch, > HADOOP-14163.006.patch, HADOOP-14163.007.patch, HADOOP-14163.008.tar.gz, > HADOOP-14163.009.patch, HADOOP-14163.009.tar.gz, hadoop-site.tar.gz, > hadop-site-rendered.tar.gz > > > From the dev mailing list: > "Publishing can be attacked via a mix of scripting and revamping the darned > website. Forrest is pretty bad compared to the newer static site generators > out there (e.g. need to write XML instead of markdown, it's hard to review a > staging site because of all the absolute links, hard to customize, did I > mention XML?), and the look and feel of the site is from the 00s. We don't > actually have that much site content, so it should be possible to migrate to > a new system." > This issue is find a solution to migrate the old site to a new modern static > site generator using a more contemprary theme. > Goals: > * existing links should work (or at least redirected) > * It should be easy to add more content required by a release automatically > (most probably with creating separated markdown files) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14163) Refactor existing hadoop site to use more usable static website generator
[ https://issues.apache.org/jira/browse/HADOOP-14163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-14163: -- Attachment: (was: HDFS-13149-HDFS-7240.001.patch) > Refactor existing hadoop site to use more usable static website generator > - > > Key: HADOOP-14163 > URL: https://issues.apache.org/jira/browse/HADOOP-14163 > Project: Hadoop Common > Issue Type: Improvement > Components: site >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-14163-001.zip, HADOOP-14163-002.zip, > HADOOP-14163-003.zip, HADOOP-14163.004.patch, HADOOP-14163.005.patch, > HADOOP-14163.006.patch, HADOOP-14163.007.patch, HADOOP-14163.008.tar.gz, > HADOOP-14163.009.patch, HADOOP-14163.009.tar.gz, hadoop-site.tar.gz, > hadop-site-rendered.tar.gz > > > From the dev mailing list: > "Publishing can be attacked via a mix of scripting and revamping the darned > website. Forrest is pretty bad compared to the newer static site generators > out there (e.g. need to write XML instead of markdown, it's hard to review a > staging site because of all the absolute links, hard to customize, did I > mention XML?), and the look and feel of the site is from the 00s. We don't > actually have that much site content, so it should be possible to migrate to > a new system." > This issue is find a solution to migrate the old site to a new modern static > site generator using a more contemprary theme. > Goals: > * existing links should work (or at least redirected) > * It should be easy to add more content required by a release automatically > (most probably with creating separated markdown files) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14163) Refactor existing hadoop site to use more usable static website generator
[ https://issues.apache.org/jira/browse/HADOOP-14163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-14163: -- Attachment: HADOOP-14163.009.tar.gz > Refactor existing hadoop site to use more usable static website generator > - > > Key: HADOOP-14163 > URL: https://issues.apache.org/jira/browse/HADOOP-14163 > Project: Hadoop Common > Issue Type: Improvement > Components: site >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-14163-001.zip, HADOOP-14163-002.zip, > HADOOP-14163-003.zip, HADOOP-14163.004.patch, HADOOP-14163.005.patch, > HADOOP-14163.006.patch, HADOOP-14163.007.patch, HADOOP-14163.008.tar.gz, > HADOOP-14163.009.tar.gz, hadoop-site.tar.gz, hadop-site-rendered.tar.gz > > > From the dev mailing list: > "Publishing can be attacked via a mix of scripting and revamping the darned > website. Forrest is pretty bad compared to the newer static site generators > out there (e.g. need to write XML instead of markdown, it's hard to review a > staging site because of all the absolute links, hard to customize, did I > mention XML?), and the look and feel of the site is from the 00s. We don't > actually have that much site content, so it should be possible to migrate to > a new system." > This issue is find a solution to migrate the old site to a new modern static > site generator using a more contemprary theme. > Goals: > * existing links should work (or at least redirected) > * It should be easy to add more content required by a release automatically > (most probably with creating separated markdown files) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14903) Add json-smart explicitly to pom.xml
[
https://issues.apache.org/jira/browse/HADOOP-14903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366838#comment-16366838
]
genericqa commented on HADOOP-14903:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 21m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
34s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
12s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
10s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:17213a0 |
| JIRA Issue | HADOOP-14903 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910909/HADOOP-14903-branch-2-004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux df4255598e7f 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / fe044e6 |
| maven | version: Apache Maven 3.3.9
(bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00) |
| Default Java | 1.7.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14150/testReport/ |
| Max. process+thread count | 66 (vs. ulimit of 5500) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14150/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add json-smart explicitly to pom.xml
>
>
> Key: HADOOP-14903
> URL: https://issues.apache.org/jira/browse/HADOOP-14903
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.0.0-beta1
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>Priority: Major
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14903-003-branch-2.patch,
> HADOOP-14903-branch-2-003.patch, HADOOP-14903-branch-2-004.patch,
> HADOOP-14903.001.patch, HADOOP-14903.002.patch, HADOOP-14903.003.patch
>
>
> With the library update in HADOOP-14799, maven knows how to pull in
> net.minidev:json-smart for tests, but not for
[jira] [Commented] (HADOOP-14903) Add json-smart explicitly to pom.xml
[
https://issues.apache.org/jira/browse/HADOOP-14903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366829#comment-16366829
]
genericqa commented on HADOOP-14903:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 21m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
24s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
10s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:17213a0 |
| JIRA Issue | HADOOP-14903 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910909/HADOOP-14903-branch-2-004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux d19dce5bc0c5 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / fe044e6 |
| maven | version: Apache Maven 3.3.9
(bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00) |
| Default Java | 1.7.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14149/testReport/ |
| Max. process+thread count | 75 (vs. ulimit of 5500) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14149/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add json-smart explicitly to pom.xml
>
>
> Key: HADOOP-14903
> URL: https://issues.apache.org/jira/browse/HADOOP-14903
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.0.0-beta1
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>Priority: Major
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14903-003-branch-2.patch,
> HADOOP-14903-branch-2-003.patch, HADOOP-14903-branch-2-004.patch,
> HADOOP-14903.001.patch, HADOOP-14903.002.patch, HADOOP-14903.003.patch
>
>
> With the library update in HADOOP-14799, maven knows how to pull in
> net.minidev:json-smart for tests, but not for
[jira] [Commented] (HADOOP-14903) Add json-smart explicitly to pom.xml
[
https://issues.apache.org/jira/browse/HADOOP-14903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366803#comment-16366803
]
Brahma Reddy Battula commented on HADOOP-14903:
---
[~vinayrpet] thanks for taking a look into the issue.
bq.Instead of overriding json-smart to 1.1.1 (as previous in branch-2.x), I
think its safe to upgrade to closer version 1.3.1
Ok. Updated patch.
> Add json-smart explicitly to pom.xml
>
>
> Key: HADOOP-14903
> URL: https://issues.apache.org/jira/browse/HADOOP-14903
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.0.0-beta1
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>Priority: Major
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14903-003-branch-2.patch,
> HADOOP-14903-branch-2-003.patch, HADOOP-14903-branch-2-004.patch,
> HADOOP-14903.001.patch, HADOOP-14903.002.patch, HADOOP-14903.003.patch
>
>
> With the library update in HADOOP-14799, maven knows how to pull in
> net.minidev:json-smart for tests, but not for packaging. This needs to be
> added to the main project pom in order to avoid this warning:
> {noformat}
> [WARNING] The POM for net.minidev:json-smart:jar:2.3-SNAPSHOT is missing, no
> dependency information available
> {noformat}
> This is pulled in from a few places:
> {noformat}
> [INFO] | +- org.apache.hadoop:hadoop-auth:jar:3.1.0-SNAPSHOT:compile
> [INFO] | | +- com.nimbusds:nimbus-jose-jwt:jar:4.41.1:compile
> [INFO] | | | +- com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
> [INFO] | | | \- net.minidev:json-smart:jar:2.3:compile
> [INFO] | | \- org.apache.kerby:token-provider:jar:1.0.1:compile
> [INFO] | | \- com.nimbusds:nimbus-jose-jwt:jar:4.41.1:compile
> [INFO] | |+-
> com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
> [INFO] | |\- net.minidev:json-smart:jar:2.3:compile
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14903) Add json-smart explicitly to pom.xml
[
https://issues.apache.org/jira/browse/HADOOP-14903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HADOOP-14903:
--
Attachment: HADOOP-14903-branch-2-004.patch
> Add json-smart explicitly to pom.xml
>
>
> Key: HADOOP-14903
> URL: https://issues.apache.org/jira/browse/HADOOP-14903
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.0.0-beta1
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>Priority: Major
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14903-003-branch-2.patch,
> HADOOP-14903-branch-2-003.patch, HADOOP-14903-branch-2-004.patch,
> HADOOP-14903.001.patch, HADOOP-14903.002.patch, HADOOP-14903.003.patch
>
>
> With the library update in HADOOP-14799, maven knows how to pull in
> net.minidev:json-smart for tests, but not for packaging. This needs to be
> added to the main project pom in order to avoid this warning:
> {noformat}
> [WARNING] The POM for net.minidev:json-smart:jar:2.3-SNAPSHOT is missing, no
> dependency information available
> {noformat}
> This is pulled in from a few places:
> {noformat}
> [INFO] | +- org.apache.hadoop:hadoop-auth:jar:3.1.0-SNAPSHOT:compile
> [INFO] | | +- com.nimbusds:nimbus-jose-jwt:jar:4.41.1:compile
> [INFO] | | | +- com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
> [INFO] | | | \- net.minidev:json-smart:jar:2.3:compile
> [INFO] | | \- org.apache.kerby:token-provider:jar:1.0.1:compile
> [INFO] | | \- com.nimbusds:nimbus-jose-jwt:jar:4.41.1:compile
> [INFO] | |+-
> com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
> [INFO] | |\- net.minidev:json-smart:jar:2.3:compile
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14799) Update nimbus-jose-jwt to 4.41.1
[ https://issues.apache.org/jira/browse/HADOOP-14799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366768#comment-16366768 ] Vinayakumar B commented on HADOOP-14799: I think, its the problem of json-smart 2.3 jar which gets pulled after this change. In HADOOP-14903, upgrading json-smart to 1.3.1 (which also supported by jose-jwt-4.41.1) along with jose-jwt-4.41.1 should fix everything. > Update nimbus-jose-jwt to 4.41.1 > > > Key: HADOOP-14799 > URL: https://issues.apache.org/jira/browse/HADOOP-14799 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Ray Chiang >Assignee: Ray Chiang >Priority: Major > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-14799.001.patch, HADOOP-14799.002.patch, > HADOOP-14799.003.patch > > > Update the dependency > com.nimbusds:nimbus-jose-jwt:3.9 > to the latest (4.41.1) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14903) Add json-smart explicitly to pom.xml
[
https://issues.apache.org/jira/browse/HADOOP-14903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366766#comment-16366766
]
Vinayakumar B commented on HADOOP-14903:
nimbus-jose-jwt-4.41.1 depends on json-smart ranging from [1.3.1, 2.3]
Instead of overriding json-smart to 1.1.1 (as previous in branch-2.x), I think
its safe to upgrade to closer version 1.3.1 (closer to existing 1.1.1) without
breaking any feature of (unknown to us) jose-jwt.
> Add json-smart explicitly to pom.xml
>
>
> Key: HADOOP-14903
> URL: https://issues.apache.org/jira/browse/HADOOP-14903
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.0.0-beta1
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>Priority: Major
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14903-003-branch-2.patch,
> HADOOP-14903-branch-2-003.patch, HADOOP-14903.001.patch,
> HADOOP-14903.002.patch, HADOOP-14903.003.patch
>
>
> With the library update in HADOOP-14799, maven knows how to pull in
> net.minidev:json-smart for tests, but not for packaging. This needs to be
> added to the main project pom in order to avoid this warning:
> {noformat}
> [WARNING] The POM for net.minidev:json-smart:jar:2.3-SNAPSHOT is missing, no
> dependency information available
> {noformat}
> This is pulled in from a few places:
> {noformat}
> [INFO] | +- org.apache.hadoop:hadoop-auth:jar:3.1.0-SNAPSHOT:compile
> [INFO] | | +- com.nimbusds:nimbus-jose-jwt:jar:4.41.1:compile
> [INFO] | | | +- com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
> [INFO] | | | \- net.minidev:json-smart:jar:2.3:compile
> [INFO] | | \- org.apache.kerby:token-provider:jar:1.0.1:compile
> [INFO] | | \- com.nimbusds:nimbus-jose-jwt:jar:4.41.1:compile
> [INFO] | |+-
> com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
> [INFO] | |\- net.minidev:json-smart:jar:2.3:compile
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15230) org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
[
https://issues.apache.org/jira/browse/HADOOP-15230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366758#comment-16366758
]
genericqa commented on HADOOP-15230:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
15s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 37s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 9 new + 3 unchanged - 253 fixed = 12 total (was 256) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
23s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15230 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910882/HADOOP-15230.006.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 146f118dc091 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a1e05e0 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14148/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14148/testReport/ |
| Max. process+thread count | 1649 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14148/console |
| Powered by | Apache Yetus
