[jira] [Updated] (HADOOP-15230) org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
[
https://issues.apache.org/jira/browse/HADOOP-15230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Howard Yoo updated HADOOP-15230:
Attachment: HADOOP-15230.006.patch
> org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
>
>
> Key: HADOOP-15230
> URL: https://issues.apache.org/jira/browse/HADOOP-15230
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Reporter: Howard Yoo
>Priority: Major
> Labels: GraphiteSink,, metrics
> Attachments: HADOOP-15230.006.patch
>
>
> org.apache.hadoop.metrics2.GraphiteSink's implementation has certain problems
> that would make it to generate metrics incorrectly.
> The problem lies with line 77 ~ 84 of the GraphiteSink java:
> {code:java}
> for (MetricsTag tag : record.tags()) {
> if (tag.value() != null) {
> metricsPathPrefix.append(".");
> metricsPathPrefix.append(tag.name());
> metricsPathPrefix.append("=");
> metricsPathPrefix.append(tag.value());
> }
> }
> {code}
> It produces point tags having name=value pair in the metrics. However, notice
> how the tags are added with '.' as its delimiters. Rather than using the '.'
> character, it should follow the following convention mentioned in the latest
> graphite doc of using ';' character.
> [http://graphite.readthedocs.io/en/latest/tags.html]
> Also, the value is not properly being escaped, meaning that if the value has
> a '.' character in it, it will easily confuse Graphite to accept it as a
> delimiter, rather than the value. A really good prime example is when the
> value is a hostname or ip address,
> {code:java}
> metrics.example.Hostname=this.is.a.hostname.and.this.is.Metrics 10.0{code}
> In this example, the since the value of the hostname contains '.', it is
> extremely hard for the receiving end to determine which part is hostname and
> which part is the rest of the metrics name. A good strategy is to convert any
> '.' character in the value to be converted to other characters, such as '_'.
> However, the best way would be to follow the latest metrics convention of
> using ';'
> {code:java}
> metrics.example.and.this.is.Metrics;Hostname=this.is.a.hostname 10.0{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15230) org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
[
https://issues.apache.org/jira/browse/HADOOP-15230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Howard Yoo updated HADOOP-15230:
Attachment: (was: HADOOP-15230.005.patch)
> org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
>
>
> Key: HADOOP-15230
> URL: https://issues.apache.org/jira/browse/HADOOP-15230
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Reporter: Howard Yoo
>Priority: Major
> Labels: GraphiteSink,, metrics
>
> org.apache.hadoop.metrics2.GraphiteSink's implementation has certain problems
> that would make it to generate metrics incorrectly.
> The problem lies with line 77 ~ 84 of the GraphiteSink java:
> {code:java}
> for (MetricsTag tag : record.tags()) {
> if (tag.value() != null) {
> metricsPathPrefix.append(".");
> metricsPathPrefix.append(tag.name());
> metricsPathPrefix.append("=");
> metricsPathPrefix.append(tag.value());
> }
> }
> {code}
> It produces point tags having name=value pair in the metrics. However, notice
> how the tags are added with '.' as its delimiters. Rather than using the '.'
> character, it should follow the following convention mentioned in the latest
> graphite doc of using ';' character.
> [http://graphite.readthedocs.io/en/latest/tags.html]
> Also, the value is not properly being escaped, meaning that if the value has
> a '.' character in it, it will easily confuse Graphite to accept it as a
> delimiter, rather than the value. A really good prime example is when the
> value is a hostname or ip address,
> {code:java}
> metrics.example.Hostname=this.is.a.hostname.and.this.is.Metrics 10.0{code}
> In this example, the since the value of the hostname contains '.', it is
> extremely hard for the receiving end to determine which part is hostname and
> which part is the rest of the metrics name. A good strategy is to convert any
> '.' character in the value to be converted to other characters, such as '_'.
> However, the best way would be to follow the latest metrics convention of
> using ';'
> {code:java}
> metrics.example.and.this.is.Metrics;Hostname=this.is.a.hostname 10.0{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15223) Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap() when available
[
https://issues.apache.org/jira/browse/HADOOP-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366678#comment-16366678
]
Akira Ajisaka commented on HADOOP-15223:
Thanks [~zhenyi] for the update.
Would you replace {{Collections.EMPTY_MAP}} in
SingleConstraintAppPlacementAllocator.java? I'm +1 if that is addressed.
> Replace Collections.EMPTY_SET and EMPTY_MAP with emptySet() and emptyMap()
> when available
> -
>
> Key: HADOOP-15223
> URL: https://issues.apache.org/jira/browse/HADOOP-15223
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: fang zhenyi
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-15223.001.patch, HADOOP-15223.002.patch,
> HADOOP-15223.003.patch
>
>
> The use of {{Collections.EMPTY_SET}} and {{Collections.EMPTY_MAP}} often
> causes unchecked assignment and it should be replaced with
> {{Collections.emptySet()}} and {{Collections.emptyMap()}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[
https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366675#comment-16366675
]
genericqa commented on HADOOP-13761:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 48s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 14s{color} | {color:orange} hadoop-tools/hadoop-aws: The patch generated 38
new + 13 unchanged - 1 fixed = 51 total (was 14) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 16s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
42s{color} | {color:red} hadoop-tools/hadoop-aws generated 5 new + 0 unchanged
- 0 fixed = 5 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
42s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-tools/hadoop-aws |
| | Class org.apache.hadoop.fs.s3a.InconsistentS3Object defines non-transient
non-serializable instance field policy In InconsistentS3Object.java:instance
field policy In InconsistentS3Object.java |
| | Inconsistent synchronization of
org.apache.hadoop.fs.s3a.S3AInputStream.contentRangeFinish; locked 88% of time
Unsynchronized access at S3AInputStream.java:88% of time Unsynchronized access
at S3AInputStream.java:[line 296] |
| | Inconsistent synchronization of
org.apache.hadoop.fs.s3a.S3AInputStream.pos; locked 62% of time Unsynchronized
access at S3AInputStream.java:62% of time Unsynchronized access at
S3AInputStream.java:[line 241] |
| | Inconsistent synchronization of
org.apache.hadoop.fs.s3a.S3AInputStream.readahead; locked 83% of time
Unsynchronized access at S3AInputStream.java:83% of time Unsynchronized access
at S3AInputStream.java:[line 247] |
| | Inconsistent synchronization of
org.apache.hadoop.fs.s3a.S3AInputStream.wrappedStream; locked 57% of time
Unsynchronized access at S3AInputStream.java:57% of time Unsynchronized access
at S3AInputStream.java:[line
[jira] [Commented] (HADOOP-15230) org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
[
https://issues.apache.org/jira/browse/HADOOP-15230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1630#comment-1630
]
genericqa commented on HADOOP-15230:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 58s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
17s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 37s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 9 new + 3 unchanged - 253 fixed = 12 total (was 256) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 21s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
16s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15230 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910868/HADOOP-15230.005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 3d5952eaa9e2 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8013475 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14146/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14146/artifact/out/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14146/testReport/ |
| Max. process+thread count | 1675 (vs. ulimit of 5500) |
| modules | C:
[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13761: -- Attachment: HADOOP-13761-006.patch > S3Guard: implement retries for DDB failures and throttling; translate > exceptions > > > Key: HADOOP-13761 > URL: https://issues.apache.org/jira/browse/HADOOP-13761 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Blocker > Attachments: HADOOP-13761-004-to-005.patch, > HADOOP-13761-005-to-006-approx.diff.txt, HADOOP-13761-005.patch, > HADOOP-13761-006.patch, HADOOP-13761.001.patch, HADOOP-13761.002.patch, > HADOOP-13761.003.patch, HADOOP-13761.004.patch > > > Following the S3AFileSystem integration patch in HADOOP-13651, we need to add > retry logic. > In HADOOP-13651, I added TODO comments in most of the places retry loops are > needed, including: > - open(path). If MetadataStore reflects recent create/move of file path, but > we fail to read it from S3, retry. > - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the > file exists, retry. > - rename(src,dest). If source path is not visible in S3 yet, retry. > - listFiles(). Skip for now. Not currently implemented in S3Guard. I will > create a separate JIRA for this as it will likely require interface changes > (i.e. prefix or subtree scan). > We may miss some cases initially and we should do failure injection testing > to make sure we're covered. Failure injection tests can be a separate JIRA > to make this easier to review. > We also need basic configuration parameters around retry policy. There > should be a way to specify maximum retry duration, as some applications would > prefer to receive an error eventually, than waiting indefinitely. We should > also be keeping statistics when inconsistency is detected and we enter a > retry loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366647#comment-16366647 ] Aaron Fabbri commented on HADOOP-13761: --- About to post v6 patch. Changes from v5: - Run testOpenFailOnRead() in a loop with 0.5 failure probability. (My idea about making failpoint probing more deterministic wasn't worth the trouble.) It runs 20 times, but I tested it with 500 iterations. I no longer assert non-s3guard fails since it is now RNG. - Add status=404 failpoint to InconsistentAmazonS3Client#getObject(): that is the first GET AFAICT - Add FNFE failpoint to to skip() (which may use read() underneath to advance stream, I believe). - Add s3guard-specific retry() around lazySeek(). Now both callers of reopen() are covered in a retry, only when S3Guard is enabled. I'll add the v6 patch as well as a v5-to-v6 diff so you can skim the changes..(diff-diff is not exact: it contains a change or two from your v5 tweaks). > S3Guard: implement retries for DDB failures and throttling; translate > exceptions > > > Key: HADOOP-13761 > URL: https://issues.apache.org/jira/browse/HADOOP-13761 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Blocker > Attachments: HADOOP-13761-004-to-005.patch, > HADOOP-13761-005-to-006-approx.diff.txt, HADOOP-13761-005.patch, > HADOOP-13761.001.patch, HADOOP-13761.002.patch, HADOOP-13761.003.patch, > HADOOP-13761.004.patch > > > Following the S3AFileSystem integration patch in HADOOP-13651, we need to add > retry logic. > In HADOOP-13651, I added TODO comments in most of the places retry loops are > needed, including: > - open(path). If MetadataStore reflects recent create/move of file path, but > we fail to read it from S3, retry. > - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the > file exists, retry. > - rename(src,dest). If source path is not visible in S3 yet, retry. > - listFiles(). Skip for now. Not currently implemented in S3Guard. I will > create a separate JIRA for this as it will likely require interface changes > (i.e. prefix or subtree scan). > We may miss some cases initially and we should do failure injection testing > to make sure we're covered. Failure injection tests can be a separate JIRA > to make this easier to review. > We also need basic configuration parameters around retry policy. There > should be a way to specify maximum retry duration, as some applications would > prefer to receive an error eventually, than waiting indefinitely. We should > also be keeping statistics when inconsistency is detected and we enter a > retry loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13761: -- Attachment: (was: HADOOP-13761-006.diff) > S3Guard: implement retries for DDB failures and throttling; translate > exceptions > > > Key: HADOOP-13761 > URL: https://issues.apache.org/jira/browse/HADOOP-13761 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Blocker > Attachments: HADOOP-13761-004-to-005.patch, > HADOOP-13761-005-to-006-approx.diff.txt, HADOOP-13761-005.patch, > HADOOP-13761.001.patch, HADOOP-13761.002.patch, HADOOP-13761.003.patch, > HADOOP-13761.004.patch > > > Following the S3AFileSystem integration patch in HADOOP-13651, we need to add > retry logic. > In HADOOP-13651, I added TODO comments in most of the places retry loops are > needed, including: > - open(path). If MetadataStore reflects recent create/move of file path, but > we fail to read it from S3, retry. > - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the > file exists, retry. > - rename(src,dest). If source path is not visible in S3 yet, retry. > - listFiles(). Skip for now. Not currently implemented in S3Guard. I will > create a separate JIRA for this as it will likely require interface changes > (i.e. prefix or subtree scan). > We may miss some cases initially and we should do failure injection testing > to make sure we're covered. Failure injection tests can be a separate JIRA > to make this easier to review. > We also need basic configuration parameters around retry policy. There > should be a way to specify maximum retry duration, as some applications would > prefer to receive an error eventually, than waiting indefinitely. We should > also be keeping statistics when inconsistency is detected and we enter a > retry loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13761: -- Attachment: HADOOP-13761-006.diff > S3Guard: implement retries for DDB failures and throttling; translate > exceptions > > > Key: HADOOP-13761 > URL: https://issues.apache.org/jira/browse/HADOOP-13761 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Blocker > Attachments: HADOOP-13761-004-to-005.patch, > HADOOP-13761-005-to-006-approx.diff.txt, HADOOP-13761-005.patch, > HADOOP-13761.001.patch, HADOOP-13761.002.patch, HADOOP-13761.003.patch, > HADOOP-13761.004.patch > > > Following the S3AFileSystem integration patch in HADOOP-13651, we need to add > retry logic. > In HADOOP-13651, I added TODO comments in most of the places retry loops are > needed, including: > - open(path). If MetadataStore reflects recent create/move of file path, but > we fail to read it from S3, retry. > - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the > file exists, retry. > - rename(src,dest). If source path is not visible in S3 yet, retry. > - listFiles(). Skip for now. Not currently implemented in S3Guard. I will > create a separate JIRA for this as it will likely require interface changes > (i.e. prefix or subtree scan). > We may miss some cases initially and we should do failure injection testing > to make sure we're covered. Failure injection tests can be a separate JIRA > to make this easier to review. > We also need basic configuration parameters around retry policy. There > should be a way to specify maximum retry duration, as some applications would > prefer to receive an error eventually, than waiting indefinitely. We should > also be keeping statistics when inconsistency is detected and we enter a > retry loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13761: -- Attachment: HADOOP-13761-005-to-006-approx.diff.txt > S3Guard: implement retries for DDB failures and throttling; translate > exceptions > > > Key: HADOOP-13761 > URL: https://issues.apache.org/jira/browse/HADOOP-13761 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Blocker > Attachments: HADOOP-13761-004-to-005.patch, > HADOOP-13761-005-to-006-approx.diff.txt, HADOOP-13761-005.patch, > HADOOP-13761.001.patch, HADOOP-13761.002.patch, HADOOP-13761.003.patch, > HADOOP-13761.004.patch > > > Following the S3AFileSystem integration patch in HADOOP-13651, we need to add > retry logic. > In HADOOP-13651, I added TODO comments in most of the places retry loops are > needed, including: > - open(path). If MetadataStore reflects recent create/move of file path, but > we fail to read it from S3, retry. > - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the > file exists, retry. > - rename(src,dest). If source path is not visible in S3 yet, retry. > - listFiles(). Skip for now. Not currently implemented in S3Guard. I will > create a separate JIRA for this as it will likely require interface changes > (i.e. prefix or subtree scan). > We may miss some cases initially and we should do failure injection testing > to make sure we're covered. Failure injection tests can be a separate JIRA > to make this easier to review. > We also need basic configuration parameters around retry policy. There > should be a way to specify maximum retry duration, as some applications would > prefer to receive an error eventually, than waiting indefinitely. We should > also be keeping statistics when inconsistency is detected and we enter a > retry loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15230) org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
[
https://issues.apache.org/jira/browse/HADOOP-15230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Howard Yoo updated HADOOP-15230:
Attachment: (was: HADOOP-15230.004.patch)
> org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
>
>
> Key: HADOOP-15230
> URL: https://issues.apache.org/jira/browse/HADOOP-15230
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Reporter: Howard Yoo
>Priority: Major
> Labels: GraphiteSink,, metrics
> Attachments: HADOOP-15230.005.patch
>
>
> org.apache.hadoop.metrics2.GraphiteSink's implementation has certain problems
> that would make it to generate metrics incorrectly.
> The problem lies with line 77 ~ 84 of the GraphiteSink java:
> {code:java}
> for (MetricsTag tag : record.tags()) {
> if (tag.value() != null) {
> metricsPathPrefix.append(".");
> metricsPathPrefix.append(tag.name());
> metricsPathPrefix.append("=");
> metricsPathPrefix.append(tag.value());
> }
> }
> {code}
> It produces point tags having name=value pair in the metrics. However, notice
> how the tags are added with '.' as its delimiters. Rather than using the '.'
> character, it should follow the following convention mentioned in the latest
> graphite doc of using ';' character.
> [http://graphite.readthedocs.io/en/latest/tags.html]
> Also, the value is not properly being escaped, meaning that if the value has
> a '.' character in it, it will easily confuse Graphite to accept it as a
> delimiter, rather than the value. A really good prime example is when the
> value is a hostname or ip address,
> {code:java}
> metrics.example.Hostname=this.is.a.hostname.and.this.is.Metrics 10.0{code}
> In this example, the since the value of the hostname contains '.', it is
> extremely hard for the receiving end to determine which part is hostname and
> which part is the rest of the metrics name. A good strategy is to convert any
> '.' character in the value to be converted to other characters, such as '_'.
> However, the best way would be to follow the latest metrics convention of
> using ';'
> {code:java}
> metrics.example.and.this.is.Metrics;Hostname=this.is.a.hostname 10.0{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15230) org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
[
https://issues.apache.org/jira/browse/HADOOP-15230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Howard Yoo updated HADOOP-15230:
Attachment: HADOOP-15230.005.patch
> org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
>
>
> Key: HADOOP-15230
> URL: https://issues.apache.org/jira/browse/HADOOP-15230
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Reporter: Howard Yoo
>Priority: Major
> Labels: GraphiteSink,, metrics
> Attachments: HADOOP-15230.005.patch
>
>
> org.apache.hadoop.metrics2.GraphiteSink's implementation has certain problems
> that would make it to generate metrics incorrectly.
> The problem lies with line 77 ~ 84 of the GraphiteSink java:
> {code:java}
> for (MetricsTag tag : record.tags()) {
> if (tag.value() != null) {
> metricsPathPrefix.append(".");
> metricsPathPrefix.append(tag.name());
> metricsPathPrefix.append("=");
> metricsPathPrefix.append(tag.value());
> }
> }
> {code}
> It produces point tags having name=value pair in the metrics. However, notice
> how the tags are added with '.' as its delimiters. Rather than using the '.'
> character, it should follow the following convention mentioned in the latest
> graphite doc of using ';' character.
> [http://graphite.readthedocs.io/en/latest/tags.html]
> Also, the value is not properly being escaped, meaning that if the value has
> a '.' character in it, it will easily confuse Graphite to accept it as a
> delimiter, rather than the value. A really good prime example is when the
> value is a hostname or ip address,
> {code:java}
> metrics.example.Hostname=this.is.a.hostname.and.this.is.Metrics 10.0{code}
> In this example, the since the value of the hostname contains '.', it is
> extremely hard for the receiving end to determine which part is hostname and
> which part is the rest of the metrics name. A good strategy is to convert any
> '.' character in the value to be converted to other characters, such as '_'.
> However, the best way would be to follow the latest metrics convention of
> using ';'
> {code:java}
> metrics.example.and.this.is.Metrics;Hostname=this.is.a.hostname 10.0{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[
https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366622#comment-16366622
]
Aaron Fabbri commented on HADOOP-13761:
---
[~ste...@apache.org] was this part of your diff intentional or accident?
{noformat}
diff --git
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ARetryPolicy.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ARetryPolicy.java
index 61ce68441e3..514a5e606e8 100644
---
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ARetryPolicy.java
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ARetryPolicy.java
@@ -153,7 +153,6 @@ public S3ARetryPolicy(Configuration conf) {
policyMap.put(InterruptedException.class, fail);
// note this does not pick up subclasses (like socket timeout)
policyMap.put(InterruptedIOException.class, fail);
-policyMap.put(AWSRedirectException.class, fail);
{noformat}
> S3Guard: implement retries for DDB failures and throttling; translate
> exceptions
>
>
> Key: HADOOP-13761
> URL: https://issues.apache.org/jira/browse/HADOOP-13761
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Aaron Fabbri
>Assignee: Aaron Fabbri
>Priority: Blocker
> Attachments: HADOOP-13761-004-to-005.patch, HADOOP-13761-005.patch,
> HADOOP-13761.001.patch, HADOOP-13761.002.patch, HADOOP-13761.003.patch,
> HADOOP-13761.004.patch
>
>
> Following the S3AFileSystem integration patch in HADOOP-13651, we need to add
> retry logic.
> In HADOOP-13651, I added TODO comments in most of the places retry loops are
> needed, including:
> - open(path). If MetadataStore reflects recent create/move of file path, but
> we fail to read it from S3, retry.
> - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the
> file exists, retry.
> - rename(src,dest). If source path is not visible in S3 yet, retry.
> - listFiles(). Skip for now. Not currently implemented in S3Guard. I will
> create a separate JIRA for this as it will likely require interface changes
> (i.e. prefix or subtree scan).
> We may miss some cases initially and we should do failure injection testing
> to make sure we're covered. Failure injection tests can be a separate JIRA
> to make this easier to review.
> We also need basic configuration parameters around retry policy. There
> should be a way to specify maximum retry duration, as some applications would
> prefer to receive an error eventually, than waiting indefinitely. We should
> also be keeping statistics when inconsistency is detected and we enter a
> retry loop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13837) Always get unable to kill error message even the hadoop process was successfully killed
[
https://issues.apache.org/jira/browse/HADOOP-13837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366608#comment-16366608
]
genericqa commented on HADOOP-13837:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 30s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
4s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
11s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 29s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
15s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-13837 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12900444/HADOOP-13837.05.patch
|
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux 69217260963f 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8013475 |
| maven | version: Apache Maven 3.3.9 |
| shellcheck | v0.4.6 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14145/testReport/ |
| Max. process+thread count | 341 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14145/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Always get unable to kill error message even the hadoop process was
> successfully killed
> ---
>
> Key: HADOOP-13837
> URL: https://issues.apache.org/jira/browse/HADOOP-13837
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Attachments: HADOOP-13837.01.patch, HADOOP-13837.02.patch,
> HADOOP-13837.03.patch, HADOOP-13837.04.patch, HADOOP-13837.05.patch,
> check_proc.sh
>
>
> *Reproduce steps*
> # Setup a hadoop cluster
> # Stop resource manager : yarn --daemon stop resourcemanager
> # Stop node manager : yarn --daemon stop nodemanager
> WARNING: nodemanager did not stop gracefully after 5 seconds: Trying to kill
> with kill -9
> ERROR: Unable to kill 20325
> it always gets "Unable to kill " error message, this gives user
> impression there is something wrong with the node manager process because it
> was not able to be
[jira] [Commented] (HADOOP-15203) Support composite trusted channel resolver that supports both whitelist and blacklist
[
https://issues.apache.org/jira/browse/HADOOP-15203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366605#comment-16366605
]
genericqa commented on HADOOP-15203:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 38s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 58s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}118m 0s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}169m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestTruncateQuotaUpdate |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15203 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910840/HADOOP-15203.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 4e58c800ee76 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8013475 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14143/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14143/testReport/ |
| Max. process+thread count | 3231 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Updated] (HADOOP-15200) Missing DistCpOptions constructor breaks downstream DistCp projects in 3.0
[ https://issues.apache.org/jira/browse/HADOOP-15200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated HADOOP-15200: Target Version/s: 3.2.0, 3.0.2 (was: 3.1.0, 3.0.2) > Missing DistCpOptions constructor breaks downstream DistCp projects in 3.0 > -- > > Key: HADOOP-15200 > URL: https://issues.apache.org/jira/browse/HADOOP-15200 > Project: Hadoop Common > Issue Type: Bug > Components: tools/distcp >Affects Versions: 3.0.0 >Reporter: Kuhu Shukla >Priority: Critical > > Post HADOOP-14267, the constructor for DistCpOptions was removed and will > break any project using it for java based implementation/usage of DistCp. > This JIRA would track next steps required to reconcile/fix this > incompatibility. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14060) HTTP servlet /logs should require authentication and authorization
[
https://issues.apache.org/jira/browse/HADOOP-14060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated HADOOP-14060:
Target Version/s: 3.0.1, 3.2.0 (was: 3.1.0, 3.0.1)
> HTTP servlet /logs should require authentication and authorization
> --
>
> Key: HADOOP-14060
> URL: https://issues.apache.org/jira/browse/HADOOP-14060
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 3.0.0-alpha4
>Reporter: John Zhuge
>Priority: Critical
> Attachments: HADOOP-14060-tmp.001.patch
>
>
> HADOOP-14047 makes KMS call {{HttpServer2#setACL}}. Access control works fine
> for /conf, /jmx, /logLevel, and /stacks, but not for /logs.
> The code in {{AdminAuthorizedServlet#doGet}} for /logs and
> {{ConfServlet#doGet}} for /conf are quite similar. This makes me believe that
> /logs should subject to the same access control as intended by the original
> developer.
> IMHO this could either be my misconfiguration or there is a bug somewhere in
> {{HttpServer2}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14060) HTTP servlet /logs should require authentication and authorization
[
https://issues.apache.org/jira/browse/HADOOP-14060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366584#comment-16366584
]
Wangda Tan commented on HADOOP-14060:
-
Thanks [~jzhuge], moved to 3.2.0.
> HTTP servlet /logs should require authentication and authorization
> --
>
> Key: HADOOP-14060
> URL: https://issues.apache.org/jira/browse/HADOOP-14060
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 3.0.0-alpha4
>Reporter: John Zhuge
>Priority: Critical
> Attachments: HADOOP-14060-tmp.001.patch
>
>
> HADOOP-14047 makes KMS call {{HttpServer2#setACL}}. Access control works fine
> for /conf, /jmx, /logLevel, and /stacks, but not for /logs.
> The code in {{AdminAuthorizedServlet#doGet}} for /logs and
> {{ConfServlet#doGet}} for /conf are quite similar. This makes me believe that
> /logs should subject to the same access control as intended by the original
> developer.
> IMHO this could either be my misconfiguration or there is a bug somewhere in
> {{HttpServer2}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14606) S3AInputStream: Handle http stream skip(n) skipping < n bytes in a forward seek
[
https://issues.apache.org/jira/browse/HADOOP-14606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated HADOOP-14606:
Target Version/s: 3.2.0 (was: 3.1.0)
> S3AInputStream: Handle http stream skip(n) skipping < n bytes in a forward
> seek
> ---
>
> Key: HADOOP-14606
> URL: https://issues.apache.org/jira/browse/HADOOP-14606
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Priority: Critical
>
> There's some hints in the InputStream docs that {{skip(n)}} may skip bytes. Codepaths only seem to do this if read() returns -1, meaning end of
> stream is reached.
> If that happens when doing a forward seek via skip, then we have got our
> numbers wrong and are in trouble. Look for a negative response, log @ ERROR
> and revert to a close/reopen seek to an absolute position.
> *I have no evidence of this acutally occurring*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14606) S3AInputStream: Handle http stream skip(n) skipping < n bytes in a forward seek
[
https://issues.apache.org/jira/browse/HADOOP-14606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366583#comment-16366583
]
Wangda Tan commented on HADOOP-14606:
-
Moved to 3.2.0, please revert if you disagree.
> S3AInputStream: Handle http stream skip(n) skipping < n bytes in a forward
> seek
> ---
>
> Key: HADOOP-14606
> URL: https://issues.apache.org/jira/browse/HADOOP-14606
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Priority: Critical
>
> There's some hints in the InputStream docs that {{skip(n)}} may skip bytes. Codepaths only seem to do this if read() returns -1, meaning end of
> stream is reached.
> If that happens when doing a forward seek via skip, then we have got our
> numbers wrong and are in trouble. Look for a negative response, log @ ERROR
> and revert to a close/reopen seek to an absolute position.
> *I have no evidence of this acutally occurring*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15137) ClassNotFoundException: org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using hadoop-client-minicluster
[
https://issues.apache.org/jira/browse/HADOOP-15137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366581#comment-16366581
]
Wangda Tan commented on HADOOP-15137:
-
[~rohithsharma], could you help to check this patch? Since this is targeted to
3.1.0 and we plan to finish all blockers/critical by end of this weekend.
Otherwise I have to move this Jira to 3.2.0.
> ClassNotFoundException:
> org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using
> hadoop-client-minicluster
> --
>
> Key: HADOOP-15137
> URL: https://issues.apache.org/jira/browse/HADOOP-15137
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Jeff Zhang
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HADOOP-15137.01.patch, YARN-7673.00.patch
>
>
> I'd like to use hadoop-client-minicluster for hadoop downstream project, but
> I encounter the following exception when starting hadoop minicluster. And I
> check the hadoop-client-minicluster, it indeed does not have this class. Is
> this something that is missing when packaging the published jar ?
> {code}
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/yarn/server/api/DistributedSchedulingAMProtocol
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.createResourceManager(MiniYARNCluster.java:851)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.serviceInit(MiniYARNCluster.java:285)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15230) org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
[
https://issues.apache.org/jira/browse/HADOOP-15230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366580#comment-16366580
]
genericqa commented on HADOOP-15230:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 21s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
30s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 48s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 10 new + 126 unchanged - 130 fixed = 136 total (was 256) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 29s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
48s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 89m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15230 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910851/HADOOP-15230.004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux b1c2d81b99a3 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8013475 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14144/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14144/artifact/out/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14144/testReport/ |
| Max. process+thread count | 1352 (vs. ulimit of 5500) |
| modules | C:
[jira] [Commented] (HADOOP-14875) Create end user documentation from the compatibility guidelines
[ https://issues.apache.org/jira/browse/HADOOP-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366579#comment-16366579 ] Wangda Tan commented on HADOOP-14875: - [~templedf], could you check my latest comments? I plan to move it to 3.2.0 by end of this weekend. > Create end user documentation from the compatibility guidelines > --- > > Key: HADOOP-14875 > URL: https://issues.apache.org/jira/browse/HADOOP-14875 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 3.0.0-beta1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: AdminCompatibilityGuide.pdf, Compatibility.htm, > HADOOP-14875.001.patch, HADOOP-14875.002.patch, HADOOP-14875.003.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14238) [Umbrella] Rechecking Guava's object is not exposed to user-facing API
[ https://issues.apache.org/jira/browse/HADOOP-14238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366578#comment-16366578 ] Wangda Tan commented on HADOOP-14238: - [~xiaochen]/[~ozawa], moved to 3.2.0, please revert if you disagree. Thanks > [Umbrella] Rechecking Guava's object is not exposed to user-facing API > -- > > Key: HADOOP-14238 > URL: https://issues.apache.org/jira/browse/HADOOP-14238 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Tsuyoshi Ozawa >Priority: Critical > > This is reported by [~hitesh] on HADOOP-10101. > At least, AMRMClient#waitFor takes Guava's Supplier instance as an instance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14238) [Umbrella] Rechecking Guava's object is not exposed to user-facing API
[ https://issues.apache.org/jira/browse/HADOOP-14238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated HADOOP-14238: Target Version/s: 3.2.0 (was: 3.1.0) > [Umbrella] Rechecking Guava's object is not exposed to user-facing API > -- > > Key: HADOOP-14238 > URL: https://issues.apache.org/jira/browse/HADOOP-14238 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Tsuyoshi Ozawa >Priority: Critical > > This is reported by [~hitesh] on HADOOP-10101. > At least, AMRMClient#waitFor takes Guava's Supplier instance as an instance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14295) Authentication proxy filter may fail authorization because of getRemoteAddr
[
https://issues.apache.org/jira/browse/HADOOP-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366577#comment-16366577
]
Wangda Tan commented on HADOOP-14295:
-
[~jeffreyr97]/[~jojochuang], I saw the Jenkins gave +1, is it ready to be
committed or should we move it to 3.2.0?
> Authentication proxy filter may fail authorization because of getRemoteAddr
> ---
>
> Key: HADOOP-14295
> URL: https://issues.apache.org/jira/browse/HADOOP-14295
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.7.4, 3.0.0-alpha2, 2.8.1
>Reporter: Jeffrey E Rodriguez
>Assignee: Jeffrey E Rodriguez
>Priority: Critical
> Attachments: HADOOP-14295.002.patch, HADOOP-14295.003.patch,
> HADOOP-14295.004.patch, hadoop-14295.001.patch
>
>
> When we turn on Hadoop UI Kerberos and try to access Datanode /logs the proxy
> (Knox) would get an Authorization failure and it hosts would should as
> 127.0.0.1 even though Knox wasn't in local host to Datanode, error message:
> {quote}
> "2017-04-08 07:01:23,029 ERROR security.AuthenticationWithProxyUserFilter
> (AuthenticationWithProxyUserFilter.java:getRemoteUser(94)) - Unable to verify
> proxy user: Unauthorized connection for super-user: knox from IP 127.0.0.1"
> {quote}
> We were able to figure out that Datanode have Jetty listening on localhost
> and that Netty is used to server request to DataNode, this was a measure to
> improve performance because of Netty Async NIO design.
> I propose to add a check for x-forwarded-for header since proxys usually
> inject that header before we do a getRemoteAddr
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14873) force cmake to exit if it can't detect a C99 compiler
[ https://issues.apache.org/jira/browse/HADOOP-14873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366575#comment-16366575 ] Wangda Tan commented on HADOOP-14873: - Moved to 3.2.0, please revert if you disagree. > force cmake to exit if it can't detect a C99 compiler > - > > Key: HADOOP-14873 > URL: https://issues.apache.org/jira/browse/HADOOP-14873 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.0.0-beta1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Critical > Attachments: HADOOP-14873.00.patch > > > C99 mode isn't getting invoked for containter-executor on CentOS 6.6 + gcc > 4.4. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14873) force cmake to exit if it can't detect a C99 compiler
[ https://issues.apache.org/jira/browse/HADOOP-14873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated HADOOP-14873: Target Version/s: 3.2.0 (was: 3.1.0) > force cmake to exit if it can't detect a C99 compiler > - > > Key: HADOOP-14873 > URL: https://issues.apache.org/jira/browse/HADOOP-14873 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.0.0-beta1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Critical > Attachments: HADOOP-14873.00.patch > > > C99 mode isn't getting invoked for containter-executor on CentOS 6.6 + gcc > 4.4. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13837) Always get unable to kill error message even the hadoop process was successfully killed
[
https://issues.apache.org/jira/browse/HADOOP-13837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366574#comment-16366574
]
Wangda Tan commented on HADOOP-13837:
-
Moved to 3.2.0, please revert if you disagree.
> Always get unable to kill error message even the hadoop process was
> successfully killed
> ---
>
> Key: HADOOP-13837
> URL: https://issues.apache.org/jira/browse/HADOOP-13837
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Attachments: HADOOP-13837.01.patch, HADOOP-13837.02.patch,
> HADOOP-13837.03.patch, HADOOP-13837.04.patch, HADOOP-13837.05.patch,
> check_proc.sh
>
>
> *Reproduce steps*
> # Setup a hadoop cluster
> # Stop resource manager : yarn --daemon stop resourcemanager
> # Stop node manager : yarn --daemon stop nodemanager
> WARNING: nodemanager did not stop gracefully after 5 seconds: Trying to kill
> with kill -9
> ERROR: Unable to kill 20325
> it always gets "Unable to kill " error message, this gives user
> impression there is something wrong with the node manager process because it
> was not able to be forcibly killed. But in fact, the kill command works as
> expected.
> This was because hadoop-functions.sh did not check process existence after
> kill properly. Currently it checks the process liveness right after the kill
> command
> {code}
> ...
> kill -9 "${pid}" >/dev/null 2>&1
> if ps -p "${pid}" > /dev/null 2>&1; then
> hadoop_error "ERROR: Unable to kill ${pid}"
> ...
> {code}
> when resource manager stopped before node managers, it always takes some
> additional time until the process completely terminates. I tried to print
> output of {{ps -p }} in a while loop after kill -9, and found
> following
> {noformat}
> 16212 ?00:00:11 java
> 0
> PID TTY TIME CMD
> 16212 ?00:00:11 java
> 0
> PID TTY TIME CMD
> 16212 ?00:00:11 java
> 0
> PID TTY TIME CMD
> 1
> PID TTY TIME CMD
> 1
> PID TTY TIME CMD
> 1
> PID TTY TIME CMD
> ...
> {noformat}
> in the first 3 times of the loop, the process did not terminate so the exit
> code of {{ps -p}} are still {{0}}
> *Proposal of a fix*
> Firstly I was thinking to add a more comprehensive pid check, it checks the
> pid liveness until reaches the HADOOP_STOP_TIMEOUT, but this seems to add too
> much complexity. Second fix was to simply add a {{sleep 3}} after {{kill
> -9}}, it should fix the error in most cases with relative small changes to
> the script.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15200) Missing DistCpOptions constructor breaks downstream DistCp projects in 3.0
[ https://issues.apache.org/jira/browse/HADOOP-15200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366572#comment-16366572 ] Wangda Tan commented on HADOOP-15200: - Moved to 3.2.0, please revert if you disagree. > Missing DistCpOptions constructor breaks downstream DistCp projects in 3.0 > -- > > Key: HADOOP-15200 > URL: https://issues.apache.org/jira/browse/HADOOP-15200 > Project: Hadoop Common > Issue Type: Bug > Components: tools/distcp >Affects Versions: 3.0.0 >Reporter: Kuhu Shukla >Priority: Critical > > Post HADOOP-14267, the constructor for DistCpOptions was removed and will > break any project using it for java based implementation/usage of DistCp. > This JIRA would track next steps required to reconcile/fix this > incompatibility. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13837) Always get unable to kill error message even the hadoop process was successfully killed
[
https://issues.apache.org/jira/browse/HADOOP-13837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated HADOOP-13837:
Target Version/s: 3.2.0 (was: 3.1.0)
> Always get unable to kill error message even the hadoop process was
> successfully killed
> ---
>
> Key: HADOOP-13837
> URL: https://issues.apache.org/jira/browse/HADOOP-13837
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Attachments: HADOOP-13837.01.patch, HADOOP-13837.02.patch,
> HADOOP-13837.03.patch, HADOOP-13837.04.patch, HADOOP-13837.05.patch,
> check_proc.sh
>
>
> *Reproduce steps*
> # Setup a hadoop cluster
> # Stop resource manager : yarn --daemon stop resourcemanager
> # Stop node manager : yarn --daemon stop nodemanager
> WARNING: nodemanager did not stop gracefully after 5 seconds: Trying to kill
> with kill -9
> ERROR: Unable to kill 20325
> it always gets "Unable to kill " error message, this gives user
> impression there is something wrong with the node manager process because it
> was not able to be forcibly killed. But in fact, the kill command works as
> expected.
> This was because hadoop-functions.sh did not check process existence after
> kill properly. Currently it checks the process liveness right after the kill
> command
> {code}
> ...
> kill -9 "${pid}" >/dev/null 2>&1
> if ps -p "${pid}" > /dev/null 2>&1; then
> hadoop_error "ERROR: Unable to kill ${pid}"
> ...
> {code}
> when resource manager stopped before node managers, it always takes some
> additional time until the process completely terminates. I tried to print
> output of {{ps -p }} in a while loop after kill -9, and found
> following
> {noformat}
> 16212 ?00:00:11 java
> 0
> PID TTY TIME CMD
> 16212 ?00:00:11 java
> 0
> PID TTY TIME CMD
> 16212 ?00:00:11 java
> 0
> PID TTY TIME CMD
> 1
> PID TTY TIME CMD
> 1
> PID TTY TIME CMD
> 1
> PID TTY TIME CMD
> ...
> {noformat}
> in the first 3 times of the loop, the process did not terminate so the exit
> code of {{ps -p}} are still {{0}}
> *Proposal of a fix*
> Firstly I was thinking to add a more comprehensive pid check, it checks the
> pid liveness until reaches the HADOOP_STOP_TIMEOUT, but this seems to add too
> much complexity. Second fix was to simply add a {{sleep 3}} after {{kill
> -9}}, it should fix the error in most cases with relative small changes to
> the script.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13916) Document how downstream clients should make use of the new shaded client artifacts
[ https://issues.apache.org/jira/browse/HADOOP-13916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366567#comment-16366567 ] Wangda Tan commented on HADOOP-13916: - Moved to 3.2.0 given lack of activities on this effort, please revert if you have different opinions. > Document how downstream clients should make use of the new shaded client > artifacts > -- > > Key: HADOOP-13916 > URL: https://issues.apache.org/jira/browse/HADOOP-13916 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha2 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > > provide a quickstart that walks through using the new shaded dependencies > with Maven to create a simple downstream project. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13916) Document how downstream clients should make use of the new shaded client artifacts
[ https://issues.apache.org/jira/browse/HADOOP-13916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated HADOOP-13916: Target Version/s: 3.2.0 (was: 3.1.0) > Document how downstream clients should make use of the new shaded client > artifacts > -- > > Key: HADOOP-13916 > URL: https://issues.apache.org/jira/browse/HADOOP-13916 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha2 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > > provide a quickstart that walks through using the new shaded dependencies > with Maven to create a simple downstream project. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15007) Stabilize and document Configuration element
[ https://issues.apache.org/jira/browse/HADOOP-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated HADOOP-15007: Target Version/s: 3.2.0 (was: 3.1.0) > Stabilize and document Configuration element > -- > > Key: HADOOP-15007 > URL: https://issues.apache.org/jira/browse/HADOOP-15007 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Ajay Kumar >Priority: Blocker > Attachments: HADOOP-15007.000.patch, HADOOP-15007.001.patch > > > HDFS-12350 (moved to HADOOP-15005). Adds the ability to tag properties with a > value. > We need to make sure that this feature is backwards compatible & usable in > production. That's docs, testing, marshalling etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15010) hadoop-resourceestimator's assembly buries it
[
https://issues.apache.org/jira/browse/HADOOP-15010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated HADOOP-15010:
Target Version/s: 3.2.0 (was: 3.1.0)
> hadoop-resourceestimator's assembly buries it
> -
>
> Key: HADOOP-15010
> URL: https://issues.apache.org/jira/browse/HADOOP-15010
> Project: Hadoop Common
> Issue Type: Bug
> Components: build, tools
>Affects Versions: 2.9.0, 3.1.0
>Reporter: Allen Wittenauer
>Priority: Blocker
>
> There's zero reason for this layout:
> {code}
> hadoop-3.1.0-SNAPSHOT/share/hadoop/tools/resourceestimator
> - bin
> - conf
> - data
> {code}
> Buried that far back, it might as well not exist.
> Propose:
> a) HADOOP-15009 to eliminate bin
> b) Move conf file into etc/hadoop
> c) keep data where it's at
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15010) hadoop-resourceestimator's assembly buries it
[
https://issues.apache.org/jira/browse/HADOOP-15010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366555#comment-16366555
]
Wangda Tan commented on HADOOP-15010:
-
Moved to 3.2.0.
> hadoop-resourceestimator's assembly buries it
> -
>
> Key: HADOOP-15010
> URL: https://issues.apache.org/jira/browse/HADOOP-15010
> Project: Hadoop Common
> Issue Type: Bug
> Components: build, tools
>Affects Versions: 2.9.0, 3.1.0
>Reporter: Allen Wittenauer
>Priority: Blocker
>
> There's zero reason for this layout:
> {code}
> hadoop-3.1.0-SNAPSHOT/share/hadoop/tools/resourceestimator
> - bin
> - conf
> - data
> {code}
> Buried that far back, it might as well not exist.
> Propose:
> a) HADOOP-15009 to eliminate bin
> b) Move conf file into etc/hadoop
> c) keep data where it's at
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15009) hadoop-resourceestimator's shell scripts are a mess
[ https://issues.apache.org/jira/browse/HADOOP-15009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated HADOOP-15009: Target Version/s: 3.2.0 (was: 3.1.0) > hadoop-resourceestimator's shell scripts are a mess > --- > > Key: HADOOP-15009 > URL: https://issues.apache.org/jira/browse/HADOOP-15009 > Project: Hadoop Common > Issue Type: Bug > Components: scripts, tools >Affects Versions: 3.1.0 >Reporter: Allen Wittenauer >Assignee: Ajay Kumar >Priority: Blocker > Attachments: HADOOP-15009.001.patch, Screen Shot 2017-12-12 at > 11.16.23 AM.png > > > #1: > There's no reason for estimator.sh to exist. Just make it a subcommand under > yarn or whatever. > #2: > In it's current form, it's missing a BUNCH of boilerplate that makes certain > functionality completely fail. > #3 > start/stop-estimator.sh is full of copypasta that doesn't actually do > anything/work correctly. Additionally, if estimator.sh doesn't exist, > neither does this since yarn --daemon start/stop will do everything as > necessary. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15009) hadoop-resourceestimator's shell scripts are a mess
[ https://issues.apache.org/jira/browse/HADOOP-15009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366554#comment-16366554 ] Wangda Tan commented on HADOOP-15009: - Moved to 3.2.0. > hadoop-resourceestimator's shell scripts are a mess > --- > > Key: HADOOP-15009 > URL: https://issues.apache.org/jira/browse/HADOOP-15009 > Project: Hadoop Common > Issue Type: Bug > Components: scripts, tools >Affects Versions: 3.1.0 >Reporter: Allen Wittenauer >Assignee: Ajay Kumar >Priority: Blocker > Attachments: HADOOP-15009.001.patch, Screen Shot 2017-12-12 at > 11.16.23 AM.png > > > #1: > There's no reason for estimator.sh to exist. Just make it a subcommand under > yarn or whatever. > #2: > In it's current form, it's missing a BUNCH of boilerplate that makes certain > functionality completely fail. > #3 > start/stop-estimator.sh is full of copypasta that doesn't actually do > anything/work correctly. Additionally, if estimator.sh doesn't exist, > neither does this since yarn --daemon start/stop will do everything as > necessary. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15129) Datanode caches namenode DNS lookup failure and cannot startup
[
https://issues.apache.org/jira/browse/HADOOP-15129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366550#comment-16366550
]
Karthik Palaniappan commented on HADOOP-15129:
--
[~arpitagarwal] yes both tests are necessary. The HostResolutionTimeout test
proves that if the host is always unresolvable, the code still does throw an
exception. TheFlakyHostResolution test shows that if the host is only
temporarily resolvable, ipc client can recover. Other tests of course prove the
final case that if the host is immediately resolvable it's fine.
> Datanode caches namenode DNS lookup failure and cannot startup
> --
>
> Key: HADOOP-15129
> URL: https://issues.apache.org/jira/browse/HADOOP-15129
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.8.2
> Environment: Google Compute Engine.
> I'm using Java 8, Debian 8, Hadoop 2.8.2.
>Reporter: Karthik Palaniappan
>Assignee: Karthik Palaniappan
>Priority: Minor
> Attachments: HADOOP-15129.001.patch, HADOOP-15129.002.patch
>
>
> On startup, the Datanode creates an InetSocketAddress to register with each
> namenode. Though there are retries on connection failure throughout the
> stack, the same InetSocketAddress is reused.
> InetSocketAddress is an interesting class, because it resolves DNS names to
> IP addresses on construction, and it is never refreshed. Hadoop re-creates an
> InetSocketAddress in some cases just in case the remote IP has changed for a
> particular DNS name: https://issues.apache.org/jira/browse/HADOOP-7472.
> Anyway, on startup, you cna see the Datanode log: "Namenode...remains
> unresolved" -- referring to the fact that DNS lookup failed.
> {code:java}
> 2017-11-02 16:01:55,115 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Refresh request received for nameservices: null
> 2017-11-02 16:01:55,153 WARN org.apache.hadoop.hdfs.DFSUtilClient: Namenode
> for null remains unresolved for ID null. Check your hdfs-site.xml file to
> ensure namenodes are configured properly.
> 2017-11-02 16:01:55,156 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Starting BPOfferServices for nameservices:
> 2017-11-02 16:01:55,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Block pool (Datanode Uuid unassigned) service to
> cluster-32f5-m:8020 starting to offer service
> {code}
> The Datanode then proceeds to use this unresolved address, as it may work if
> the DN is configured to use a proxy. Since I'm not using a proxy, it forever
> prints out this message:
> {code:java}
> 2017-12-15 00:13:40,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:45,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:50,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:55,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:14:00,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> {code}
> Unfortunately, the log doesn't contain the exception that triggered it, but
> the culprit is actually in IPC Client:
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L444.
> This line was introduced in https://issues.apache.org/jira/browse/HADOOP-487
> to give a clear error message when somebody mispells an address.
> However, the fix in HADOOP-7472 doesn't apply here, because that code happens
> in Client#getConnection after the Connection is constructed.
> My proposed fix (will attach a patch) is to move this exception out of the
> constructor and into a place that will trigger HADOOP-7472's logic to
> re-resolve addresses. If the DNS failure was temporary, this will allow the
> connection to succeed. If not, the connection will fail after ipc client
> retries (default 10 seconds worth of retries).
> I want to fix this in ipc client rather than just in Datanode startup, as
> this fixes temporary DNS issues for all of Hadoop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15230) org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
[
https://issues.apache.org/jira/browse/HADOOP-15230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Howard Yoo updated HADOOP-15230:
Attachment: (was: HADOOP-15230.003.patch)
> org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
>
>
> Key: HADOOP-15230
> URL: https://issues.apache.org/jira/browse/HADOOP-15230
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Reporter: Howard Yoo
>Priority: Major
> Labels: GraphiteSink,, metrics
> Attachments: HADOOP-15230.004.patch
>
>
> org.apache.hadoop.metrics2.GraphiteSink's implementation has certain problems
> that would make it to generate metrics incorrectly.
> The problem lies with line 77 ~ 84 of the GraphiteSink java:
> {code:java}
> for (MetricsTag tag : record.tags()) {
> if (tag.value() != null) {
> metricsPathPrefix.append(".");
> metricsPathPrefix.append(tag.name());
> metricsPathPrefix.append("=");
> metricsPathPrefix.append(tag.value());
> }
> }
> {code}
> It produces point tags having name=value pair in the metrics. However, notice
> how the tags are added with '.' as its delimiters. Rather than using the '.'
> character, it should follow the following convention mentioned in the latest
> graphite doc of using ';' character.
> [http://graphite.readthedocs.io/en/latest/tags.html]
> Also, the value is not properly being escaped, meaning that if the value has
> a '.' character in it, it will easily confuse Graphite to accept it as a
> delimiter, rather than the value. A really good prime example is when the
> value is a hostname or ip address,
> {code:java}
> metrics.example.Hostname=this.is.a.hostname.and.this.is.Metrics 10.0{code}
> In this example, the since the value of the hostname contains '.', it is
> extremely hard for the receiving end to determine which part is hostname and
> which part is the rest of the metrics name. A good strategy is to convert any
> '.' character in the value to be converted to other characters, such as '_'.
> However, the best way would be to follow the latest metrics convention of
> using ';'
> {code:java}
> metrics.example.and.this.is.Metrics;Hostname=this.is.a.hostname 10.0{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15230) org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
[
https://issues.apache.org/jira/browse/HADOOP-15230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Howard Yoo updated HADOOP-15230:
Attachment: HADOOP-15230.004.patch
> org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
>
>
> Key: HADOOP-15230
> URL: https://issues.apache.org/jira/browse/HADOOP-15230
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Reporter: Howard Yoo
>Priority: Major
> Labels: GraphiteSink,, metrics
> Attachments: HADOOP-15230.004.patch
>
>
> org.apache.hadoop.metrics2.GraphiteSink's implementation has certain problems
> that would make it to generate metrics incorrectly.
> The problem lies with line 77 ~ 84 of the GraphiteSink java:
> {code:java}
> for (MetricsTag tag : record.tags()) {
> if (tag.value() != null) {
> metricsPathPrefix.append(".");
> metricsPathPrefix.append(tag.name());
> metricsPathPrefix.append("=");
> metricsPathPrefix.append(tag.value());
> }
> }
> {code}
> It produces point tags having name=value pair in the metrics. However, notice
> how the tags are added with '.' as its delimiters. Rather than using the '.'
> character, it should follow the following convention mentioned in the latest
> graphite doc of using ';' character.
> [http://graphite.readthedocs.io/en/latest/tags.html]
> Also, the value is not properly being escaped, meaning that if the value has
> a '.' character in it, it will easily confuse Graphite to accept it as a
> delimiter, rather than the value. A really good prime example is when the
> value is a hostname or ip address,
> {code:java}
> metrics.example.Hostname=this.is.a.hostname.and.this.is.Metrics 10.0{code}
> In this example, the since the value of the hostname contains '.', it is
> extremely hard for the receiving end to determine which part is hostname and
> which part is the rest of the metrics name. A good strategy is to convert any
> '.' character in the value to be converted to other characters, such as '_'.
> However, the best way would be to follow the latest metrics convention of
> using ';'
> {code:java}
> metrics.example.and.this.is.Metrics;Hostname=this.is.a.hostname 10.0{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15007) Stabilize and document Configuration element
[
https://issues.apache.org/jira/browse/HADOOP-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366512#comment-16366512
]
Anu Engineer commented on HADOOP-15007:
---
[~ajayydv] Overall it looks very good. Some minor comments/questions.
# Since we have the hadoop.system.tags in core-default.xml, do we need {{public
static final String HADOOP_SYSTEM_TAGS_DEFAULT =}} in
{{CommonConfigurationKeysPublic.java}}
# Not sure why we callthe {{removeUndeclaredTags}}, if we don't do that we get
custom tag support right? Am I missing something here?
> Stabilize and document Configuration element
> --
>
> Key: HADOOP-15007
> URL: https://issues.apache.org/jira/browse/HADOOP-15007
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Ajay Kumar
>Priority: Blocker
> Attachments: HADOOP-15007.000.patch, HADOOP-15007.001.patch
>
>
> HDFS-12350 (moved to HADOOP-15005). Adds the ability to tag properties with a
> value.
> We need to make sure that this feature is backwards compatible & usable in
> production. That's docs, testing, marshalling etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14396) Add builder interface to FileContext
[
https://issues.apache.org/jira/browse/HADOOP-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366477#comment-16366477
]
Lei (Eddy) Xu commented on HADOOP-14396:
{{TestCopyPreserveFlag}} is not relevant, as it does not use the builder API,
and it passed locally on my laptop.
> Add builder interface to FileContext
>
>
> Key: HADOOP-14396
> URL: https://issues.apache.org/jira/browse/HADOOP-14396
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 2.9.0, 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Major
> Attachments: HADOOP-14396.00.patch, HADOOP-14396.01.patch,
> HADOOP-14396.02.patch
>
>
> Add builder interface for {{FileContext#create}} and {{FileContext#append}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14396) Add builder interface to FileContext
[
https://issues.apache.org/jira/browse/HADOOP-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366463#comment-16366463
]
genericqa commented on HADOOP-14396:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 58s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 0s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 43s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 87m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.shell.TestCopyPreserveFlag |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14396 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910818/HADOOP-14396.02.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux c811cf83964e 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 0b489e5 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14141/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14141/testReport/ |
| Max. process+thread count | 1383 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14141/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT
[jira] [Commented] (HADOOP-15230) org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
[
https://issues.apache.org/jira/browse/HADOOP-15230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366456#comment-16366456
]
genericqa commented on HADOOP-15230:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 54s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
17s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 46s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 11 new + 247 unchanged - 9 fixed = 258 total (was 256) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 6s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
51s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15230 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910803/HADOOP-15230.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux b4cdbc17ff0c 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 0b489e5 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14140/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14140/testReport/ |
| Max. process+thread count | 1717 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14140/console |
| Powered by | Apache Yetus
[jira] [Commented] (HADOOP-15007) Stabilize and document Configuration element
[
https://issues.apache.org/jira/browse/HADOOP-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366454#comment-16366454
]
genericqa commented on HADOOP-15007:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 18s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
9s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 50s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 6 new + 352 unchanged - 1 fixed = 358 total (was 353) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 30s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 22s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestRaceWhenRelogin |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15007 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910826/HADOOP-15007.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux 29681e9ecea6 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 0b489e5 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14142/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
[jira] [Commented] (HADOOP-15203) Support composite trusted channel resolver that supports both whitelist and blacklist
[ https://issues.apache.org/jira/browse/HADOOP-15203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366443#comment-16366443 ] Ajay Kumar commented on HADOOP-15203: - Patch v2 to remove checkstyle issue. Tests failures are unrelated. > Support composite trusted channel resolver that supports both whitelist and > blacklist > - > > Key: HADOOP-15203 > URL: https://issues.apache.org/jira/browse/HADOOP-15203 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Labels: security > Attachments: HADOOP-15203.000.patch, HADOOP-15203.001.patch, > HADOOP-15203.002.patch > > > support composite trusted channel resolver that supports both whitelist and > blacklist -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15203) Support composite trusted channel resolver that supports both whitelist and blacklist
[ https://issues.apache.org/jira/browse/HADOOP-15203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HADOOP-15203: Attachment: HADOOP-15203.002.patch > Support composite trusted channel resolver that supports both whitelist and > blacklist > - > > Key: HADOOP-15203 > URL: https://issues.apache.org/jira/browse/HADOOP-15203 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Labels: security > Attachments: HADOOP-15203.000.patch, HADOOP-15203.001.patch, > HADOOP-15203.002.patch > > > support composite trusted channel resolver that supports both whitelist and > blacklist -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15203) Support composite trusted channel resolver that supports both whitelist and blacklist
[ https://issues.apache.org/jira/browse/HADOOP-15203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HADOOP-15203: Attachment: (was: HDFS-13081.002.patch) > Support composite trusted channel resolver that supports both whitelist and > blacklist > - > > Key: HADOOP-15203 > URL: https://issues.apache.org/jira/browse/HADOOP-15203 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Labels: security > Attachments: HADOOP-15203.000.patch, HADOOP-15203.001.patch > > > support composite trusted channel resolver that supports both whitelist and > blacklist -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15203) Support composite trusted channel resolver that supports both whitelist and blacklist
[ https://issues.apache.org/jira/browse/HADOOP-15203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HADOOP-15203: Attachment: HDFS-13081.002.patch > Support composite trusted channel resolver that supports both whitelist and > blacklist > - > > Key: HADOOP-15203 > URL: https://issues.apache.org/jira/browse/HADOOP-15203 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Labels: security > Attachments: HADOOP-15203.000.patch, HADOOP-15203.001.patch > > > support composite trusted channel resolver that supports both whitelist and > blacklist -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15210) Handle FNFE from S3Guard.getMetadataStore() in S3A initialize()
[
https://issues.apache.org/jira/browse/HADOOP-15210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366439#comment-16366439
]
Aaron Fabbri commented on HADOOP-15210:
---
Seems like some customers *would* want the fail-fast-and-hard behavior we have
today (don't do anything without S3Guard).. especially since successful S3Guard
usage requires all bucket modifying processes to participate.
If we agree on that, the question becomes: Should we have a "run degraded on
S3Guard failure" config option?
> Handle FNFE from S3Guard.getMetadataStore() in S3A initialize()
> ---
>
> Key: HADOOP-15210
> URL: https://issues.apache.org/jira/browse/HADOOP-15210
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Priority: Minor
>
> {{S3Guard.getMetadataStore()}} throws FileNotFoundExceptions up, as the
> comments say " rely on callers to catch and treat specially"
> S3A Filesystem doesn't do that, instead it will just fail
> FileSystem.initialize; the FNFE is generated by DynamoDBMetadataStore.
> Are we happy with this?
> Downgrading has some appeal: if you don't have the table, it will keep going.
> But failures could be a sign of bad config, so maybe silent recovery is bad.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14759) S3GuardTool prune to prune specific bucket entries
[
https://issues.apache.org/jira/browse/HADOOP-14759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366435#comment-16366435
]
Aaron Fabbri commented on HADOOP-14759:
---
I thought we already did this for some reason.
Anyways, I'd suggest you can actually
(1) specify a subdirectory tree or bucket (basically, any path prefix that ends
in slash)
> S3GuardTool prune to prune specific bucket entries
> --
>
> Key: HADOOP-14759
> URL: https://issues.apache.org/jira/browse/HADOOP-14759
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Priority: Minor
>
> Users may think that when you provide a URI to a bucket, you are pruning all
> entries in the table *for that bucket*. In fact you are purging all entries
> across all buckets in the table:
> {code}
> hadoop s3guard prune -days 7 s3a://ireland-1
> {code}
> It should be restricted to that bucket, unless you specify otherwise
> +maybe also add a hard date rather than a relative one
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[
https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366433#comment-16366433
]
Aaron Fabbri commented on HADOOP-13761:
---
[~ste...@apache.org] on your question about changing the retry() to enclose
lazySeek() instead of around stream.read(), that is not sufficient with the
current failure model (i.e. how I'm injecting failures). I think the failure
injection needs work.
{noformat}
2018-02-15 15:27:00,907 [JUnit-testOpenFailOnRead] ERROR
s3a.AbstractS3ATestBase (ITestS3AInconsistency.java:testOpenFailOnRead(129)) -
Error:
java.io.FileNotFoundException: read(b, 0, 4) on key
test/ancestor/file-to-read-DELAY_LISTING_ME failed: injecting error 3/5 for
test.
at
org.apache.hadoop.fs.s3a.InconsistentS3Object$InconsistentS3InputStream.readFailpoint(InconsistentS3Object.java:169)
at
org.apache.hadoop.fs.s3a.InconsistentS3Object$InconsistentS3InputStream.read(InconsistentS3Object.java:159)
at
org.apache.hadoop.fs.s3a.S3AInputStream.lambda$read$56(S3AInputStream.java:426)
at
org.apache.hadoop.fs.s3a.S3AInputStream$$Lambda$31/1279551328.execute(Unknown
Source)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$7(Invoker.java:260)
at
org.apache.hadoop.fs.s3a.Invoker$$Lambda$12/89609.execute(Unknown Source)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:317)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:256)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:231)
at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:438)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.fs.s3a.ITestS3AInconsistency.testOpenFailOnRead(ITestS3AInconsistency.java:126)
{noformat}
If we can agree on where to inject failures I think I can come up with a good
solution.
Maybe:
- We need both lazySeek() and the stream.read() retries?
- The failure injection for InconsistentS3InputStream() should also have a
failpoint in skip(), which would have exposed the lack of retry in lazySeek().
- AmazonS3Client.getObject() currently does not fail, but returns an
InconsistentS3Object with the read/skip/etc. failpoints mentioned. Seems like
getObject() needs to fail as, looking at the SDK code, it actually does the GET
request I believe.
Does this sound right?
Also: My current approach of failing read 5 times then succeeding (5<20 which
is retry max) is not going to work to expose all the codepaths that fail. I
need a loop runs multiple tests and either (1) increases max failures, or
failure offset, by one each duration (how traceroute uses TTLs to probe router
hops; I use to probe failure points) or (2) uses randomization to be likely to
hit different failure points.
#1 actually seems more deterministic.
> S3Guard: implement retries for DDB failures and throttling; translate
> exceptions
>
>
> Key: HADOOP-13761
> URL: https://issues.apache.org/jira/browse/HADOOP-13761
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Aaron Fabbri
>Assignee: Aaron Fabbri
>Priority: Blocker
> Attachments: HADOOP-13761-004-to-005.patch, HADOOP-13761-005.patch,
> HADOOP-13761.001.patch, HADOOP-13761.002.patch, HADOOP-13761.003.patch,
> HADOOP-13761.004.patch
>
>
> Following the S3AFileSystem integration patch in HADOOP-13651, we need to add
> retry logic.
> In HADOOP-13651, I added TODO comments in most of the places retry loops are
> needed, including:
> - open(path). If MetadataStore reflects recent create/move of file path, but
> we fail to read it from S3, retry.
> - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the
> file exists, retry.
> - rename(src,dest). If source path is not visible in S3 yet, retry.
> - listFiles(). Skip for now. Not currently implemented in S3Guard. I will
> create a separate JIRA for this as it will likely require interface changes
> (i.e. prefix or subtree scan).
> We may miss some cases initially and we should do failure injection testing
> to make sure we're covered. Failure injection tests can be a separate JIRA
> to make this easier to review.
> We also need basic configuration parameters around retry policy. There
> should be a way to specify maximum retry duration, as some applications would
> prefer to receive an error eventually, than waiting indefinitely. We should
> also be keeping statistics when inconsistency is detected and we enter a
> retry loop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HADOOP-15007) Stabilize and document Configuration element
[ https://issues.apache.org/jira/browse/HADOOP-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HADOOP-15007: Attachment: HADOOP-15007.001.patch > Stabilize and document Configuration element > -- > > Key: HADOOP-15007 > URL: https://issues.apache.org/jira/browse/HADOOP-15007 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Ajay Kumar >Priority: Blocker > Attachments: HADOOP-15007.000.patch, HADOOP-15007.001.patch > > > HDFS-12350 (moved to HADOOP-15005). Adds the ability to tag properties with a > value. > We need to make sure that this feature is backwards compatible & usable in > production. That's docs, testing, marshalling etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15007) Stabilize and document Configuration element
[ https://issues.apache.org/jira/browse/HADOOP-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366393#comment-16366393 ] Ajay Kumar commented on HADOOP-15007: - [~leftnoteasy] moved it for 3.2 [~elek],[~anu],Uploaded new patch which picks Tags from config file instead of Enum,Strings > Stabilize and document Configuration element > -- > > Key: HADOOP-15007 > URL: https://issues.apache.org/jira/browse/HADOOP-15007 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Ajay Kumar >Priority: Blocker > Attachments: HADOOP-15007.000.patch, HADOOP-15007.001.patch > > > HDFS-12350 (moved to HADOOP-15005). Adds the ability to tag properties with a > value. > We need to make sure that this feature is backwards compatible & usable in > production. That's docs, testing, marshalling etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15007) Stabilize and document Configuration element
[ https://issues.apache.org/jira/browse/HADOOP-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HADOOP-15007: Affects Version/s: (was: 3.1.0) 3.2.0 > Stabilize and document Configuration element > -- > > Key: HADOOP-15007 > URL: https://issues.apache.org/jira/browse/HADOOP-15007 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Ajay Kumar >Priority: Blocker > Attachments: HADOOP-15007.000.patch > > > HDFS-12350 (moved to HADOOP-15005). Adds the ability to tag properties with a > value. > We need to make sure that this feature is backwards compatible & usable in > production. That's docs, testing, marshalling etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15206) BZip2 drops and duplicates records when input split size is small
[
https://issues.apache.org/jira/browse/HADOOP-15206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366339#comment-16366339
]
genericqa commented on HADOOP-15206:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 23s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 44s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
11s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}126m 38s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}226m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapreduce.v2.TestMRJobs |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15206 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910782/HADOOP-15206.008.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 62fa7bbdb8e3 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / da59acd |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
[jira] [Commented] (HADOOP-14396) Add builder interface to FileContext
[
https://issues.apache.org/jira/browse/HADOOP-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366309#comment-16366309
]
Lei (Eddy) Xu commented on HADOOP-14396:
Updated the patch to address checkstyle and javadoc warnings.
> Add builder interface to FileContext
>
>
> Key: HADOOP-14396
> URL: https://issues.apache.org/jira/browse/HADOOP-14396
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 2.9.0, 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Major
> Attachments: HADOOP-14396.00.patch, HADOOP-14396.01.patch,
> HADOOP-14396.02.patch
>
>
> Add builder interface for {{FileContext#create}} and {{FileContext#append}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14396) Add builder interface to FileContext
[
https://issues.apache.org/jira/browse/HADOOP-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HADOOP-14396:
---
Attachment: HADOOP-14396.02.patch
> Add builder interface to FileContext
>
>
> Key: HADOOP-14396
> URL: https://issues.apache.org/jira/browse/HADOOP-14396
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 2.9.0, 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Major
> Attachments: HADOOP-14396.00.patch, HADOOP-14396.01.patch,
> HADOOP-14396.02.patch
>
>
> Add builder interface for {{FileContext#create}} and {{FileContext#append}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14396) Add builder interface to FileContext
[
https://issues.apache.org/jira/browse/HADOOP-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366295#comment-16366295
]
genericqa commented on HADOOP-14396:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 56s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
28s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 48s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 157 unchanged - 0 fixed = 158 total (was 157) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 19s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
3s{color} | {color:red} hadoop-common-project_hadoop-common generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 15s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}104m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.shell.TestCopyPreserveFlag |
| | hadoop.ha.TestZKFailoverController |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14396 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910790/HADOOP-14396.01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux e0d742a92acb 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / da59acd |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14139/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14139/artifact/out/diff-javadoc-javadoc-hadoop-common-project_hadoop-common.txt
|
| unit |
[jira] [Comment Edited] (HADOOP-14507) extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key
[
https://issues.apache.org/jira/browse/HADOOP-14507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366289#comment-16366289
]
Larry McCay edited comment on HADOOP-14507 at 2/15/18 9:43 PM:
---
{quote}bq. Note that the value of {{fs.s3a.server-side-encryption.key}} can be
a simple string to a KMS key, which is the best way to manage keys. The only
time you'd have a number in there is for SSE-C encryption, where every client
must supply a key. That's not an easy one to work with...my stance is "just use
AWS KMS" If there's an API explicitly for managing the real encryption keys,
that's only relevant for SSE-C and future client-side encryption. In which
case, we could treat those keys differently
{quote}
Ahh, I think that is where my confusion was.
If I understand properly that the fs.s3a.server-side-encryption.key is a name
into another KMS for the actual key than I am fine with it.
Any future management of actual encryption keys should definitely consider the
Key Provider API at that time.
LGTM.
+1
was (Author: lmccay):
{quote}bq. Note that the value of {{fs.s3a.server-side-encryption.key}} can be
a simple string to a KMS key, which is the best way to manage keys. The only
time you'd have a number in there is for SSE-C encryption, where every client
must supply a key. That's not an easy one to work with...my stance is "just use
AWS KMS"
If there's an API explicitly for managing the real encryption keys, that's only
relevant for SSE-C and future client-side encryption. In which case, we could
treat those keys differently
{quote}
Ahh, I think that is where my confusion was.
If I understand properly that the fs.s3a.server-side-encryption.key is a name
into another KMS for the actual key than I am fine with it.
Any future management of actual encryption keys should definitely consider the
Key Provider API at that time.
LGTM.
+1
> extend per-bucket secret key config with explicit getPassword() on
> fs.s3a.$bucket.secret.key
>
>
> Key: HADOOP-14507
> URL: https://issues.apache.org/jira/browse/HADOOP-14507
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Attachments: HADOOP-14507-001.patch, HADOOP-14507-002.patch,
> HADOOP-14507-003.patch, HADOOP-14507-004.patch, HADOOP-14507-005.patch,
> HADOOP-14507-006.patch, HADOOP-14507-006.patch, HADOOP-14507-007.patch
>
>
> Per-bucket jceks support turns out to be complex as you have to manage
> multiple jecks files & configure the client to ask for the right one. This is
> because we're calling {{Configuration.getPassword{"fs,s3a.secret.key"}}.
> If before that, we do a check for the explict id, key, session key in the
> properties {{fs.s3a.$bucket.secret}} ( & c), we could have a single JCEKs
> file with all the secrets for different bucket. You would only need to
> explicitly point the base config to the secrets file, and the right
> credentials would be picked up, if set
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14507) extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key
[
https://issues.apache.org/jira/browse/HADOOP-14507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366289#comment-16366289
]
Larry McCay edited comment on HADOOP-14507 at 2/15/18 9:42 PM:
---
{quote}bq. Note that the value of {{fs.s3a.server-side-encryption.key}} can be
a simple string to a KMS key, which is the best way to manage keys. The only
time you'd have a number in there is for SSE-C encryption, where every client
must supply a key. That's not an easy one to work with...my stance is "just use
AWS KMS"
If there's an API explicitly for managing the real encryption keys, that's only
relevant for SSE-C and future client-side encryption. In which case, we could
treat those keys differently
{quote}
Ahh, I think that is where my confusion was.
If I understand properly that the fs.s3a.server-side-encryption.key is a name
into another KMS for the actual key than I am fine with it.
Any future management of actual encryption keys should definitely consider the
Key Provider API at that time.
LGTM.
+1
was (Author: lmccay):
{quote}bq. Note that the value of {{fs.s3a.server-side-encryption.key}} can be
a simple string to a KMS key, which is the best way to manage keys. The only
time you'd have a number in there is for SSE-C encryption, where every client
must supply a key. That's not an easy one to work with...my stance is "just use
AWS KMS"
If there's an API explicitly for managing the real encryption keys, that's only
relevant for SSE-C and future client-side encryption. In which case, we could
treat those keys differently
{quote}
Ahh, I think that is where my confusion was.
If I understand properly that the fs.s3a.server-side-encryption.key is a name
into another KMS for the actual key than I am fine with it.
Any future management of actual encryption keys should definitely consider the
Key Provider API at that time.
LGTM.
+1
> extend per-bucket secret key config with explicit getPassword() on
> fs.s3a.$bucket.secret.key
>
>
> Key: HADOOP-14507
> URL: https://issues.apache.org/jira/browse/HADOOP-14507
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Attachments: HADOOP-14507-001.patch, HADOOP-14507-002.patch,
> HADOOP-14507-003.patch, HADOOP-14507-004.patch, HADOOP-14507-005.patch,
> HADOOP-14507-006.patch, HADOOP-14507-006.patch, HADOOP-14507-007.patch
>
>
> Per-bucket jceks support turns out to be complex as you have to manage
> multiple jecks files & configure the client to ask for the right one. This is
> because we're calling {{Configuration.getPassword{"fs,s3a.secret.key"}}.
> If before that, we do a check for the explict id, key, session key in the
> properties {{fs.s3a.$bucket.secret}} ( & c), we could have a single JCEKs
> file with all the secrets for different bucket. You would only need to
> explicitly point the base config to the secrets file, and the right
> credentials would be picked up, if set
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14507) extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key
[
https://issues.apache.org/jira/browse/HADOOP-14507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366289#comment-16366289
]
Larry McCay commented on HADOOP-14507:
--
{quote}bq. Note that the value of {{fs.s3a.server-side-encryption.key}} can be
a simple string to a KMS key, which is the best way to manage keys. The only
time you'd have a number in there is for SSE-C encryption, where every client
must supply a key. That's not an easy one to work with...my stance is "just use
AWS KMS"
If there's an API explicitly for managing the real encryption keys, that's only
relevant for SSE-C and future client-side encryption. In which case, we could
treat those keys differently
{quote}
Ahh, I think that is where my confusion was.
If I understand properly that the fs.s3a.server-side-encryption.key is a name
into another KMS for the actual key than I am fine with it.
Any future management of actual encryption keys should definitely consider the
Key Provider API at that time.
LGTM.
+1
> extend per-bucket secret key config with explicit getPassword() on
> fs.s3a.$bucket.secret.key
>
>
> Key: HADOOP-14507
> URL: https://issues.apache.org/jira/browse/HADOOP-14507
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Critical
> Attachments: HADOOP-14507-001.patch, HADOOP-14507-002.patch,
> HADOOP-14507-003.patch, HADOOP-14507-004.patch, HADOOP-14507-005.patch,
> HADOOP-14507-006.patch, HADOOP-14507-006.patch, HADOOP-14507-007.patch
>
>
> Per-bucket jceks support turns out to be complex as you have to manage
> multiple jecks files & configure the client to ask for the right one. This is
> because we're calling {{Configuration.getPassword{"fs,s3a.secret.key"}}.
> If before that, we do a check for the explict id, key, session key in the
> properties {{fs.s3a.$bucket.secret}} ( & c), we could have a single JCEKs
> file with all the secrets for different bucket. You would only need to
> explicitly point the base config to the secrets file, and the right
> credentials would be picked up, if set
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15230) org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
[
https://issues.apache.org/jira/browse/HADOOP-15230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Howard Yoo updated HADOOP-15230:
Attachment: HADOOP-15230.003.patch
> org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
>
>
> Key: HADOOP-15230
> URL: https://issues.apache.org/jira/browse/HADOOP-15230
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Reporter: Howard Yoo
>Priority: Major
> Labels: GraphiteSink,, metrics
> Attachments: HADOOP-15230.003.patch
>
>
> org.apache.hadoop.metrics2.GraphiteSink's implementation has certain problems
> that would make it to generate metrics incorrectly.
> The problem lies with line 77 ~ 84 of the GraphiteSink java:
> {code:java}
> for (MetricsTag tag : record.tags()) {
> if (tag.value() != null) {
> metricsPathPrefix.append(".");
> metricsPathPrefix.append(tag.name());
> metricsPathPrefix.append("=");
> metricsPathPrefix.append(tag.value());
> }
> }
> {code}
> It produces point tags having name=value pair in the metrics. However, notice
> how the tags are added with '.' as its delimiters. Rather than using the '.'
> character, it should follow the following convention mentioned in the latest
> graphite doc of using ';' character.
> [http://graphite.readthedocs.io/en/latest/tags.html]
> Also, the value is not properly being escaped, meaning that if the value has
> a '.' character in it, it will easily confuse Graphite to accept it as a
> delimiter, rather than the value. A really good prime example is when the
> value is a hostname or ip address,
> {code:java}
> metrics.example.Hostname=this.is.a.hostname.and.this.is.Metrics 10.0{code}
> In this example, the since the value of the hostname contains '.', it is
> extremely hard for the receiving end to determine which part is hostname and
> which part is the rest of the metrics name. A good strategy is to convert any
> '.' character in the value to be converted to other characters, such as '_'.
> However, the best way would be to follow the latest metrics convention of
> using ';'
> {code:java}
> metrics.example.and.this.is.Metrics;Hostname=this.is.a.hostname 10.0{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15230) org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
[
https://issues.apache.org/jira/browse/HADOOP-15230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Howard Yoo updated HADOOP-15230:
Attachment: (was: HADOOP-15230.003.patch)
> org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
>
>
> Key: HADOOP-15230
> URL: https://issues.apache.org/jira/browse/HADOOP-15230
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Reporter: Howard Yoo
>Priority: Major
> Labels: GraphiteSink,, metrics
>
> org.apache.hadoop.metrics2.GraphiteSink's implementation has certain problems
> that would make it to generate metrics incorrectly.
> The problem lies with line 77 ~ 84 of the GraphiteSink java:
> {code:java}
> for (MetricsTag tag : record.tags()) {
> if (tag.value() != null) {
> metricsPathPrefix.append(".");
> metricsPathPrefix.append(tag.name());
> metricsPathPrefix.append("=");
> metricsPathPrefix.append(tag.value());
> }
> }
> {code}
> It produces point tags having name=value pair in the metrics. However, notice
> how the tags are added with '.' as its delimiters. Rather than using the '.'
> character, it should follow the following convention mentioned in the latest
> graphite doc of using ';' character.
> [http://graphite.readthedocs.io/en/latest/tags.html]
> Also, the value is not properly being escaped, meaning that if the value has
> a '.' character in it, it will easily confuse Graphite to accept it as a
> delimiter, rather than the value. A really good prime example is when the
> value is a hostname or ip address,
> {code:java}
> metrics.example.Hostname=this.is.a.hostname.and.this.is.Metrics 10.0{code}
> In this example, the since the value of the hostname contains '.', it is
> extremely hard for the receiving end to determine which part is hostname and
> which part is the rest of the metrics name. A good strategy is to convert any
> '.' character in the value to be converted to other characters, such as '_'.
> However, the best way would be to follow the latest metrics convention of
> using ';'
> {code:java}
> metrics.example.and.this.is.Metrics;Hostname=this.is.a.hostname 10.0{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15230) org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
[
https://issues.apache.org/jira/browse/HADOOP-15230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Howard Yoo updated HADOOP-15230:
Attachment: (was: HADOOP-15230.002.patch)
> org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
>
>
> Key: HADOOP-15230
> URL: https://issues.apache.org/jira/browse/HADOOP-15230
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Reporter: Howard Yoo
>Priority: Major
> Labels: GraphiteSink,, metrics
> Attachments: HADOOP-15230.003.patch
>
>
> org.apache.hadoop.metrics2.GraphiteSink's implementation has certain problems
> that would make it to generate metrics incorrectly.
> The problem lies with line 77 ~ 84 of the GraphiteSink java:
> {code:java}
> for (MetricsTag tag : record.tags()) {
> if (tag.value() != null) {
> metricsPathPrefix.append(".");
> metricsPathPrefix.append(tag.name());
> metricsPathPrefix.append("=");
> metricsPathPrefix.append(tag.value());
> }
> }
> {code}
> It produces point tags having name=value pair in the metrics. However, notice
> how the tags are added with '.' as its delimiters. Rather than using the '.'
> character, it should follow the following convention mentioned in the latest
> graphite doc of using ';' character.
> [http://graphite.readthedocs.io/en/latest/tags.html]
> Also, the value is not properly being escaped, meaning that if the value has
> a '.' character in it, it will easily confuse Graphite to accept it as a
> delimiter, rather than the value. A really good prime example is when the
> value is a hostname or ip address,
> {code:java}
> metrics.example.Hostname=this.is.a.hostname.and.this.is.Metrics 10.0{code}
> In this example, the since the value of the hostname contains '.', it is
> extremely hard for the receiving end to determine which part is hostname and
> which part is the rest of the metrics name. A good strategy is to convert any
> '.' character in the value to be converted to other characters, such as '_'.
> However, the best way would be to follow the latest metrics convention of
> using ';'
> {code:java}
> metrics.example.and.this.is.Metrics;Hostname=this.is.a.hostname 10.0{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15230) org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
[
https://issues.apache.org/jira/browse/HADOOP-15230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Howard Yoo updated HADOOP-15230:
Attachment: HADOOP-15230.003.patch
> org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
>
>
> Key: HADOOP-15230
> URL: https://issues.apache.org/jira/browse/HADOOP-15230
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Reporter: Howard Yoo
>Priority: Major
> Labels: GraphiteSink,, metrics
> Attachments: HADOOP-15230.003.patch
>
>
> org.apache.hadoop.metrics2.GraphiteSink's implementation has certain problems
> that would make it to generate metrics incorrectly.
> The problem lies with line 77 ~ 84 of the GraphiteSink java:
> {code:java}
> for (MetricsTag tag : record.tags()) {
> if (tag.value() != null) {
> metricsPathPrefix.append(".");
> metricsPathPrefix.append(tag.name());
> metricsPathPrefix.append("=");
> metricsPathPrefix.append(tag.value());
> }
> }
> {code}
> It produces point tags having name=value pair in the metrics. However, notice
> how the tags are added with '.' as its delimiters. Rather than using the '.'
> character, it should follow the following convention mentioned in the latest
> graphite doc of using ';' character.
> [http://graphite.readthedocs.io/en/latest/tags.html]
> Also, the value is not properly being escaped, meaning that if the value has
> a '.' character in it, it will easily confuse Graphite to accept it as a
> delimiter, rather than the value. A really good prime example is when the
> value is a hostname or ip address,
> {code:java}
> metrics.example.Hostname=this.is.a.hostname.and.this.is.Metrics 10.0{code}
> In this example, the since the value of the hostname contains '.', it is
> extremely hard for the receiving end to determine which part is hostname and
> which part is the rest of the metrics name. A good strategy is to convert any
> '.' character in the value to be converted to other characters, such as '_'.
> However, the best way would be to follow the latest metrics convention of
> using ';'
> {code:java}
> metrics.example.and.this.is.Metrics;Hostname=this.is.a.hostname 10.0{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15242) Fix typos in hadoop-functions.sh
[
https://issues.apache.org/jira/browse/HADOOP-15242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366239#comment-16366239
]
genericqa commented on HADOOP-15242:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 24s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
3s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
9s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 14s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
14s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15242 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910791/HADOOP-15242.001.patch
|
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux 30b915ee2f01 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / da59acd |
| maven | version: Apache Maven 3.3.9 |
| shellcheck | v0.4.6 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14138/testReport/ |
| Max. process+thread count | 304 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14138/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Fix typos in hadoop-functions.sh
>
>
> Key: HADOOP-15242
> URL: https://issues.apache.org/jira/browse/HADOOP-15242
> Project: Hadoop Common
> Issue Type: Task
>Affects Versions: 3.0.0
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>Priority: Trivial
> Labels: newbie
> Attachments: HADOOP-15242.001.patch
>
>
> In the comments of this file, there is a reference to
> "hadoop-layout{color:#d04437}s{color}.sh", but all the other references in
> the actual code are for a "hadoop-layout.sh" file. Similarly, the example
> file is named "hadoop-layout.sh.example".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15206) BZip2 drops and duplicates records when input split size is small
[
https://issues.apache.org/jira/browse/HADOOP-15206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366237#comment-16366237
]
Jason Lowe commented on HADOOP-15206:
-
bq. Since this is the first time to apply a patch to the community, I apologize
for having bothered you.
No worries whatsoever. It is very common to go back and forth on a number of
patches before anything is committed, so this is simply development as usual
from my perspective. I deeply appreciate your contribution on this subtle and
tricky issue!
+1 for the latest patch, pending Jenkins.
> BZip2 drops and duplicates records when input split size is small
> -
>
> Key: HADOOP-15206
> URL: https://issues.apache.org/jira/browse/HADOOP-15206
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.3, 3.0.0
>Reporter: Aki Tanaka
>Assignee: Aki Tanaka
>Priority: Major
> Attachments: HADOOP-15206-test.patch, HADOOP-15206.001.patch,
> HADOOP-15206.002.patch, HADOOP-15206.003.patch, HADOOP-15206.004.patch,
> HADOOP-15206.005.patch, HADOOP-15206.006.patch, HADOOP-15206.007.patch,
> HADOOP-15206.008.patch
>
>
> BZip2 can drop and duplicate record when input split file is small. I
> confirmed that this issue happens when the input split size is between 1byte
> and 4bytes.
> I am seeing the following 2 problem behaviors.
>
> 1. Drop record:
> BZip2 skips the first record in the input file when the input split size is
> small
>
> Set the split size to 3 and tested to load 100 records (0, 1, 2..99)
> {code:java}
> 2018-02-01 10:52:33,502 INFO [Thread-17] mapred.TestTextInputFormat
> (TestTextInputFormat.java:verifyPartitions(317)) -
> splits[1]=file:/work/count-mismatch2/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/TestTextInputFormat/test.bz2:3+3
> count=99{code}
> > The input format read only 99 records but not 100 records
>
> 2. Duplicate Record:
> 2 input splits has same BZip2 records when the input split size is small
>
> Set the split size to 1 and tested to load 100 records (0, 1, 2..99)
>
> {code:java}
> 2018-02-01 11:18:49,309 INFO [Thread-17] mapred.TestTextInputFormat
> (TestTextInputFormat.java:verifyPartitions(318)) - splits[3]=file
> /work/count-mismatch2/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/TestTextInputFormat/test.bz2:3+1
> count=99
> 2018-02-01 11:18:49,310 WARN [Thread-17] mapred.TestTextInputFormat
> (TestTextInputFormat.java:verifyPartitions(308)) - conflict with 1 in split 4
> at position 8
> {code}
>
> I experienced this error when I execute Spark (SparkSQL) job under the
> following conditions:
> * The file size of the input files are small (around 1KB)
> * Hadoop cluster has many slave nodes (able to launch many executor tasks)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366211#comment-16366211 ] Aaron Fabbri commented on HADOOP-13761: --- Your changes all look straightforward, thanks. I can play with moving the retry this evening. > S3Guard: implement retries for DDB failures and throttling; translate > exceptions > > > Key: HADOOP-13761 > URL: https://issues.apache.org/jira/browse/HADOOP-13761 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Blocker > Attachments: HADOOP-13761-004-to-005.patch, HADOOP-13761-005.patch, > HADOOP-13761.001.patch, HADOOP-13761.002.patch, HADOOP-13761.003.patch, > HADOOP-13761.004.patch > > > Following the S3AFileSystem integration patch in HADOOP-13651, we need to add > retry logic. > In HADOOP-13651, I added TODO comments in most of the places retry loops are > needed, including: > - open(path). If MetadataStore reflects recent create/move of file path, but > we fail to read it from S3, retry. > - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the > file exists, retry. > - rename(src,dest). If source path is not visible in S3 yet, retry. > - listFiles(). Skip for now. Not currently implemented in S3Guard. I will > create a separate JIRA for this as it will likely require interface changes > (i.e. prefix or subtree scan). > We may miss some cases initially and we should do failure injection testing > to make sure we're covered. Failure injection tests can be a separate JIRA > to make this easier to review. > We also need basic configuration parameters around retry policy. There > should be a way to specify maximum retry duration, as some applications would > prefer to receive an error eventually, than waiting indefinitely. We should > also be keeping statistics when inconsistency is detected and we enter a > retry loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[
https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366200#comment-16366200
]
genericqa commented on HADOOP-13761:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 34s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 12s{color} | {color:orange} hadoop-tools/hadoop-aws: The patch generated 38
new + 13 unchanged - 1 fixed = 51 total (was 14) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
42s{color} | {color:red} hadoop-tools/hadoop-aws generated 4 new + 0 unchanged
- 0 fixed = 4 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
29s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 29s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-tools/hadoop-aws |
| | org.apache.hadoop.fs.s3a.InconsistentS3Object.readFailureCounter should
be package protected At InconsistentS3Object.java: At
InconsistentS3Object.java:[line 42] |
| | Class org.apache.hadoop.fs.s3a.InconsistentS3Object defines non-transient
non-serializable instance field policy In InconsistentS3Object.java:instance
field policy In InconsistentS3Object.java |
| | Format-string method String.format(String, Object[]) called with format
string "read(b, %d, %d) on key %s failed:with format string "read(b, %d, %d) on
key %s failed: injecting error {}/{} for test." wants 3 arguments but is given
5 in
org.apache.hadoop.fs.s3a.InconsistentS3Object$InconsistentS3InputStream.readFailpoint(int,
int) At InconsistentS3Object.java:[line 169] |
| | Inconsistent synchronization of
org.apache.hadoop.fs.s3a.S3AInputStream.wrappedStream; locked 88% of time
Unsynchronized access at S3AInputStream.java:88% of time Unsynchronized access
at S3AInputStream.java:[line 409] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA
[jira] [Comment Edited] (HADOOP-14396) Add builder interface to FileContext
[
https://issues.apache.org/jira/browse/HADOOP-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366189#comment-16366189
]
Lei (Eddy) Xu edited comment on HADOOP-14396 at 2/15/18 7:58 PM:
-
Updated the patch to address reviews:
bq. FSDataOutputStreamBuilder line 79: fs = null when built from file context.
Later in getFS() there is Preconditions.checkNotNull(fs); So, the expectation
is no one should be calling getFS() when it is constructed from FileContext ?
If using {{FileContext}}, the fs field is not used. Yes, no one should call
{{getFS()}} if it is constructed from FileContext.
bq. Why do we need an additional "donotCreateParent" option?
It is not necessary indeed. Removed in the new patch.
bq. Can you please extend the test with few build() options like recursive /
progress added ?
Done
bq. There is an annotation @Nonnull FileContext fc and also later
Preconditions.checkNotNull(fc); Is the later needed in the constructor?
My understand is that @Nonnull is for compile time check, but Precondition is
for runtime check. Giving this is a public API, it prevents other call hdfs
client as library.
The checkstyle and warnings were deleted from Yetus. Will fix them if any in
the new build.
was (Author: eddyxu):
Updated the patch to address reviews:
bq. FSDataOutputStreamBuilder line 79: fs = null when built from file context.
Later in getFS() there is Preconditions.checkNotNull(fs); So, the expectation
is no one should be calling getFS() when it is constructed from FileContext ?
If using {{FileContext}}, the fs field is not used. Yes, no one should call
{{getFS()}} if it is constructed from FileContext.
bq. Why do we need an additional "donotCreateParent" option?
It is not necessary indeed. Removed in the new patch.
bq. Can you please extend the test with few build() options like recursive /
progress added ?
Done
bq. There is an annotation @Nonnull FileContext fc and also later
Preconditions.checkNotNull(fc); Is the later needed in the constructor?
My understand is that @Nonnull is for compile time check, but Precondition is
for runtime check. Giving this is a public API, it prevents other call hdfs
client as library.
> Add builder interface to FileContext
>
>
> Key: HADOOP-14396
> URL: https://issues.apache.org/jira/browse/HADOOP-14396
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 2.9.0, 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Major
> Attachments: HADOOP-14396.00.patch, HADOOP-14396.01.patch
>
>
> Add builder interface for {{FileContext#create}} and {{FileContext#append}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14396) Add builder interface to FileContext
[
https://issues.apache.org/jira/browse/HADOOP-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HADOOP-14396:
---
Status: Patch Available (was: Open)
Updated the patch to address reviews:
bq. FSDataOutputStreamBuilder line 79: fs = null when built from file context.
Later in getFS() there is Preconditions.checkNotNull(fs); So, the expectation
is no one should be calling getFS() when it is constructed from FileContext ?
If using {{FileContext}}, the fs field is not used. Yes, no one should call
{{getFS()}} if it is constructed from FileContext.
bq. Why do we need an additional "donotCreateParent" option?
It is not necessary indeed. Removed in the new patch.
bq. Can you please extend the test with few build() options like recursive /
progress added ?
Done
bq. There is an annotation @Nonnull FileContext fc and also later
Preconditions.checkNotNull(fc); Is the later needed in the constructor?
My understand is that @Nonnull is for compile time check, but Precondition is
for runtime check. Giving this is a public API, it prevents other call hdfs
client as library.
> Add builder interface to FileContext
>
>
> Key: HADOOP-14396
> URL: https://issues.apache.org/jira/browse/HADOOP-14396
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 3.0.0-alpha3, 2.9.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Major
> Attachments: HADOOP-14396.00.patch, HADOOP-14396.01.patch
>
>
> Add builder interface for {{FileContext#create}} and {{FileContext#append}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15242) Fix typos in hadoop-functions.sh
[
https://issues.apache.org/jira/browse/HADOOP-15242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ray Chiang updated HADOOP-15242:
Attachment: HADOOP-15242.001.patch
> Fix typos in hadoop-functions.sh
>
>
> Key: HADOOP-15242
> URL: https://issues.apache.org/jira/browse/HADOOP-15242
> Project: Hadoop Common
> Issue Type: Task
>Affects Versions: 3.0.0
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>Priority: Trivial
> Labels: newbie
> Attachments: HADOOP-15242.001.patch
>
>
> In the comments of this file, there is a reference to
> "hadoop-layout{color:#d04437}s{color}.sh", but all the other references in
> the actual code are for a "hadoop-layout.sh" file. Similarly, the example
> file is named "hadoop-layout.sh.example".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15242) Fix typos in hadoop-functions.sh
[
https://issues.apache.org/jira/browse/HADOOP-15242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ray Chiang updated HADOOP-15242:
Status: Patch Available (was: Open)
> Fix typos in hadoop-functions.sh
>
>
> Key: HADOOP-15242
> URL: https://issues.apache.org/jira/browse/HADOOP-15242
> Project: Hadoop Common
> Issue Type: Task
>Affects Versions: 3.0.0
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>Priority: Trivial
> Labels: newbie
> Attachments: HADOOP-15242.001.patch
>
>
> In the comments of this file, there is a reference to
> "hadoop-layout{color:#d04437}s{color}.sh", but all the other references in
> the actual code are for a "hadoop-layout.sh" file. Similarly, the example
> file is named "hadoop-layout.sh.example".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15242) Fix typos in hadoop-functions.sh
Ray Chiang created HADOOP-15242:
---
Summary: Fix typos in hadoop-functions.sh
Key: HADOOP-15242
URL: https://issues.apache.org/jira/browse/HADOOP-15242
Project: Hadoop Common
Issue Type: Task
Affects Versions: 3.0.0
Reporter: Ray Chiang
Assignee: Ray Chiang
In the comments of this file, there is a reference to
"hadoop-layout{color:#d04437}s{color}.sh", but all the other references in the
actual code are for a "hadoop-layout.sh" file. Similarly, the example file is
named "hadoop-layout.sh.example".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15182) Support to change back to signature version 2 of AWS SDK
[
https://issues.apache.org/jira/browse/HADOOP-15182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366171#comment-16366171
]
Steve Loughran commented on HADOOP-15182:
-
following up on this: I'm going to document the property properly.
Now, "S3SignerType" isn't in that enum, is it? But it works
> Support to change back to signature version 2 of AWS SDK
>
>
> Key: HADOOP-15182
> URL: https://issues.apache.org/jira/browse/HADOOP-15182
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.9.0
> Environment:
>
>
>Reporter: Yonger
>Priority: Minor
>
> Current s3a depend on aws-java-sdk-bundle-1.11.199 which use signature v4.
> So for some s3-compatible system(Ceph) which still using v2, Hadoop can't
> work on them.
> s3cmd can use v2 via specify option like :
> {code:java}
> s3cmd --signature-v2 ls s3://xxx/{code}
>
> maybe we can add a parameter to allow back to use signature v2 in s3a.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14396) Add builder interface to FileContext
[
https://issues.apache.org/jira/browse/HADOOP-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HADOOP-14396:
---
Attachment: HADOOP-14396.01.patch
> Add builder interface to FileContext
>
>
> Key: HADOOP-14396
> URL: https://issues.apache.org/jira/browse/HADOOP-14396
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 2.9.0, 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Major
> Attachments: HADOOP-14396.00.patch, HADOOP-14396.01.patch
>
>
> Add builder interface for {{FileContext#create}} and {{FileContext#append}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14396) Add builder interface to FileContext
[
https://issues.apache.org/jira/browse/HADOOP-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366175#comment-16366175
]
Steve Loughran commented on HADOOP-14396:
-
I haven't done anything, but I've also pulled it out of the 3.1 list to keep
Wangda happy. I'd have liked it in, but can live without it as the
counterpoint: the same for open() is something I'll only be doing for 3.2
> Add builder interface to FileContext
>
>
> Key: HADOOP-14396
> URL: https://issues.apache.org/jira/browse/HADOOP-14396
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 2.9.0, 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Major
> Attachments: HADOOP-14396.00.patch
>
>
> Add builder interface for {{FileContext#create}} and {{FileContext#append}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14396) Add builder interface to FileContext
[
https://issues.apache.org/jira/browse/HADOOP-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14396:
Priority: Major (was: Blocker)
> Add builder interface to FileContext
>
>
> Key: HADOOP-14396
> URL: https://issues.apache.org/jira/browse/HADOOP-14396
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 2.9.0, 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Major
> Attachments: HADOOP-14396.00.patch
>
>
> Add builder interface for {{FileContext#create}} and {{FileContext#append}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14396) Add builder interface to FileContext
[
https://issues.apache.org/jira/browse/HADOOP-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14396:
Target Version/s: 3.2.0 (was: 3.1.0, 2.10.0)
Status: Open (was: Patch Available)
> Add builder interface to FileContext
>
>
> Key: HADOOP-14396
> URL: https://issues.apache.org/jira/browse/HADOOP-14396
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 3.0.0-alpha3, 2.9.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Blocker
> Attachments: HADOOP-14396.00.patch
>
>
> Add builder interface for {{FileContext#create}} and {{FileContext#append}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9747) Reduce unnecessary UGI synchronization
[ https://issues.apache.org/jira/browse/HADOOP-9747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366159#comment-16366159 ] Daryn Sharp commented on HADOOP-9747: - I'm back on it today. > Reduce unnecessary UGI synchronization > -- > > Key: HADOOP-9747 > URL: https://issues.apache.org/jira/browse/HADOOP-9747 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0-alpha1 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Attachments: HADOOP-9747-trunk-03.patch, HADOOP-9747-trunk.01.patch, > HADOOP-9747-trunk.02.patch, HADOOP-9747.2.branch-2.patch, > HADOOP-9747.2.trunk.patch, HADOOP-9747.branch-2.patch, HADOOP-9747.trunk.patch > > > Jstacks of heavily loaded NNs show up to dozens of threads blocking in the > UGI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13761: Status: Patch Available (was: Open) > S3Guard: implement retries for DDB failures and throttling; translate > exceptions > > > Key: HADOOP-13761 > URL: https://issues.apache.org/jira/browse/HADOOP-13761 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Blocker > Attachments: HADOOP-13761-004-to-005.patch, HADOOP-13761-005.patch, > HADOOP-13761.001.patch, HADOOP-13761.002.patch, HADOOP-13761.003.patch, > HADOOP-13761.004.patch > > > Following the S3AFileSystem integration patch in HADOOP-13651, we need to add > retry logic. > In HADOOP-13651, I added TODO comments in most of the places retry loops are > needed, including: > - open(path). If MetadataStore reflects recent create/move of file path, but > we fail to read it from S3, retry. > - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the > file exists, retry. > - rename(src,dest). If source path is not visible in S3 yet, retry. > - listFiles(). Skip for now. Not currently implemented in S3Guard. I will > create a separate JIRA for this as it will likely require interface changes > (i.e. prefix or subtree scan). > We may miss some cases initially and we should do failure injection testing > to make sure we're covered. Failure injection tests can be a separate JIRA > to make this easier to review. > We also need basic configuration parameters around retry policy. There > should be a way to specify maximum retry duration, as some applications would > prefer to receive an error eventually, than waiting indefinitely. We should > also be keeping statistics when inconsistency is detected and we enter a > retry loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366140#comment-16366140 ] Steve Loughran commented on HADOOP-13761: - submitted. Other than the big q around the read() retry, all LGTM > S3Guard: implement retries for DDB failures and throttling; translate > exceptions > > > Key: HADOOP-13761 > URL: https://issues.apache.org/jira/browse/HADOOP-13761 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Blocker > Attachments: HADOOP-13761-004-to-005.patch, HADOOP-13761-005.patch, > HADOOP-13761.001.patch, HADOOP-13761.002.patch, HADOOP-13761.003.patch, > HADOOP-13761.004.patch > > > Following the S3AFileSystem integration patch in HADOOP-13651, we need to add > retry logic. > In HADOOP-13651, I added TODO comments in most of the places retry loops are > needed, including: > - open(path). If MetadataStore reflects recent create/move of file path, but > we fail to read it from S3, retry. > - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the > file exists, retry. > - rename(src,dest). If source path is not visible in S3 yet, retry. > - listFiles(). Skip for now. Not currently implemented in S3Guard. I will > create a separate JIRA for this as it will likely require interface changes > (i.e. prefix or subtree scan). > We may miss some cases initially and we should do failure injection testing > to make sure we're covered. Failure injection tests can be a separate JIRA > to make this easier to review. > We also need basic configuration parameters around retry policy. There > should be a way to specify maximum retry duration, as some applications would > prefer to receive an error eventually, than waiting indefinitely. We should > also be keeping statistics when inconsistency is detected and we enter a > retry loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13761: Attachment: HADOOP-13761-005.patch > S3Guard: implement retries for DDB failures and throttling; translate > exceptions > > > Key: HADOOP-13761 > URL: https://issues.apache.org/jira/browse/HADOOP-13761 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Blocker > Attachments: HADOOP-13761-004-to-005.patch, HADOOP-13761-005.patch, > HADOOP-13761.001.patch, HADOOP-13761.002.patch, HADOOP-13761.003.patch, > HADOOP-13761.004.patch > > > Following the S3AFileSystem integration patch in HADOOP-13651, we need to add > retry logic. > In HADOOP-13651, I added TODO comments in most of the places retry loops are > needed, including: > - open(path). If MetadataStore reflects recent create/move of file path, but > we fail to read it from S3, retry. > - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the > file exists, retry. > - rename(src,dest). If source path is not visible in S3 yet, retry. > - listFiles(). Skip for now. Not currently implemented in S3Guard. I will > create a separate JIRA for this as it will likely require interface changes > (i.e. prefix or subtree scan). > We may miss some cases initially and we should do failure injection testing > to make sure we're covered. Failure injection tests can be a separate JIRA > to make this easier to review. > We also need basic configuration parameters around retry policy. There > should be a way to specify maximum retry duration, as some applications would > prefer to receive an error eventually, than waiting indefinitely. We should > also be keeping statistics when inconsistency is detected and we enter a > retry loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13761: Attachment: HADOOP-13761-004-to-005.patch > S3Guard: implement retries for DDB failures and throttling; translate > exceptions > > > Key: HADOOP-13761 > URL: https://issues.apache.org/jira/browse/HADOOP-13761 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Blocker > Attachments: HADOOP-13761-004-to-005.patch, HADOOP-13761.001.patch, > HADOOP-13761.002.patch, HADOOP-13761.003.patch, HADOOP-13761.004.patch > > > Following the S3AFileSystem integration patch in HADOOP-13651, we need to add > retry logic. > In HADOOP-13651, I added TODO comments in most of the places retry loops are > needed, including: > - open(path). If MetadataStore reflects recent create/move of file path, but > we fail to read it from S3, retry. > - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the > file exists, retry. > - rename(src,dest). If source path is not visible in S3 yet, retry. > - listFiles(). Skip for now. Not currently implemented in S3Guard. I will > create a separate JIRA for this as it will likely require interface changes > (i.e. prefix or subtree scan). > We may miss some cases initially and we should do failure injection testing > to make sure we're covered. Failure injection tests can be a separate JIRA > to make this easier to review. > We also need basic configuration parameters around retry policy. There > should be a way to specify maximum retry duration, as some applications would > prefer to receive an error eventually, than waiting indefinitely. We should > also be keeping statistics when inconsistency is detected and we enter a > retry loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366138#comment-16366138 ] Steve Loughran commented on HADOOP-13761: - I think you forgot to hit the submit button. I do that all too often. Here's my review. I've done it in the IDE and am attaching (a) the new patch and (b) the diff between yours and mine. That'll simplify your reviewing my changes. One thing I've not done is the change in S3AInputStream invoke on read, where I have three comments # I think the S3Guard invoker should be around/in lazySeek(nextReadPos, len) -that is where the GET is initiated. The read() is just trying to work with an existing stream. # what happens in the state that the file really isn't there? That is: how rapidly do we expect to timeout on retries for it? # context.dstFileStatus.getPath().toString() => move to a const to avoid recreating it on every read() call. The key issues is #1: is the retry in the right place? I don't think so. h2. changes in the patch h3. S3AFileSystem * L83 remove unused import * L704 reformatted new input stream constructor invocation for clarity; do like the new structure BTW. * L2106 reinstated the log debug statement, unless you really don't want it. h3. ITestS3AInconsistency * L120: this stream needs closing in a try-with-resources call * L131 the fail() call should nest the inner exception as the cause * move waitUntilDeleted to java 8 lambda h3. Other * InconsistentS3Object: IDE complains that the LOG is non serializable; mark as transient to get it to quieten down * AbstractCommitITest, ITestDynamoDBMetadataStoreScale, FailureInjectionPolicy, S3AOpContext, S3GuardExistsRetryPolicy : import ordering * InconsistentS3Object: import ordering & tell IDE to stop complaining about serialization issues on this subclass * S3ARetryPolicy - fixed up something wrong in the original code; duplicate AWSRedirectException in map > S3Guard: implement retries for DDB failures and throttling; translate > exceptions > > > Key: HADOOP-13761 > URL: https://issues.apache.org/jira/browse/HADOOP-13761 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Blocker > Attachments: HADOOP-13761-004-to-005.patch, HADOOP-13761.001.patch, > HADOOP-13761.002.patch, HADOOP-13761.003.patch, HADOOP-13761.004.patch > > > Following the S3AFileSystem integration patch in HADOOP-13651, we need to add > retry logic. > In HADOOP-13651, I added TODO comments in most of the places retry loops are > needed, including: > - open(path). If MetadataStore reflects recent create/move of file path, but > we fail to read it from S3, retry. > - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the > file exists, retry. > - rename(src,dest). If source path is not visible in S3 yet, retry. > - listFiles(). Skip for now. Not currently implemented in S3Guard. I will > create a separate JIRA for this as it will likely require interface changes > (i.e. prefix or subtree scan). > We may miss some cases initially and we should do failure injection testing > to make sure we're covered. Failure injection tests can be a separate JIRA > to make this easier to review. > We also need basic configuration parameters around retry policy. There > should be a way to specify maximum retry duration, as some applications would > prefer to receive an error eventually, than waiting indefinitely. We should > also be keeping statistics when inconsistency is detected and we enter a > retry loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration
[
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366130#comment-16366130
]
John Zhuge commented on HADOOP-13972:
-
Thanks [~ste...@apache.org] !
> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: Sharad Sonker
>Priority: Major
> Fix For: 3.1.0, 2.10.0, 3.0.2
>
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names,
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted
> first, then fall back to the global keys.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14060) HTTP servlet /logs should require authentication and authorization
[
https://issues.apache.org/jira/browse/HADOOP-14060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366129#comment-16366129
]
John Zhuge commented on HADOOP-14060:
-
Sorry, don't have time to work on it now. Unassigned the JIRA.
> HTTP servlet /logs should require authentication and authorization
> --
>
> Key: HADOOP-14060
> URL: https://issues.apache.org/jira/browse/HADOOP-14060
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 3.0.0-alpha4
>Reporter: John Zhuge
>Priority: Critical
> Attachments: HADOOP-14060-tmp.001.patch
>
>
> HADOOP-14047 makes KMS call {{HttpServer2#setACL}}. Access control works fine
> for /conf, /jmx, /logLevel, and /stacks, but not for /logs.
> The code in {{AdminAuthorizedServlet#doGet}} for /logs and
> {{ConfServlet#doGet}} for /conf are quite similar. This makes me believe that
> /logs should subject to the same access control as intended by the original
> developer.
> IMHO this could either be my misconfiguration or there is a bug somewhere in
> {{HttpServer2}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14060) HTTP servlet /logs should require authentication and authorization
[
https://issues.apache.org/jira/browse/HADOOP-14060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge reassigned HADOOP-14060:
---
Assignee: (was: John Zhuge)
> HTTP servlet /logs should require authentication and authorization
> --
>
> Key: HADOOP-14060
> URL: https://issues.apache.org/jira/browse/HADOOP-14060
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 3.0.0-alpha4
>Reporter: John Zhuge
>Priority: Critical
> Attachments: HADOOP-14060-tmp.001.patch
>
>
> HADOOP-14047 makes KMS call {{HttpServer2#setACL}}. Access control works fine
> for /conf, /jmx, /logLevel, and /stacks, but not for /logs.
> The code in {{AdminAuthorizedServlet#doGet}} for /logs and
> {{ConfServlet#doGet}} for /conf are quite similar. This makes me believe that
> /logs should subject to the same access control as intended by the original
> developer.
> IMHO this could either be my misconfiguration or there is a bug somewhere in
> {{HttpServer2}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15230) org.apache.hadoop.metrics2.GraphiteSink is not implemented correctly
[
https://issues.apache.org/jira/browse/HADOOP-15230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366092#comment-16366092
]
genericqa commented on HADOOP-15230:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
27s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 39s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 11 new + 247 unchanged - 9 fixed = 258 total (was 256) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 9s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 45s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 86m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.metrics2.impl.TestGraphiteMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15230 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910772/HADOOP-15230.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 3e4356cc4921 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / da59acd |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14135/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14135/artifact/out/whitespace-eol.txt
|
| unit |
[jira] [Updated] (HADOOP-15206) BZip2 drops and duplicates records when input split size is small
[
https://issues.apache.org/jira/browse/HADOOP-15206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aki Tanaka updated HADOOP-15206:
Attachment: HADOOP-15206.008.patch
> BZip2 drops and duplicates records when input split size is small
> -
>
> Key: HADOOP-15206
> URL: https://issues.apache.org/jira/browse/HADOOP-15206
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.3, 3.0.0
>Reporter: Aki Tanaka
>Assignee: Aki Tanaka
>Priority: Major
> Attachments: HADOOP-15206-test.patch, HADOOP-15206.001.patch,
> HADOOP-15206.002.patch, HADOOP-15206.003.patch, HADOOP-15206.004.patch,
> HADOOP-15206.005.patch, HADOOP-15206.006.patch, HADOOP-15206.007.patch,
> HADOOP-15206.008.patch
>
>
> BZip2 can drop and duplicate record when input split file is small. I
> confirmed that this issue happens when the input split size is between 1byte
> and 4bytes.
> I am seeing the following 2 problem behaviors.
>
> 1. Drop record:
> BZip2 skips the first record in the input file when the input split size is
> small
>
> Set the split size to 3 and tested to load 100 records (0, 1, 2..99)
> {code:java}
> 2018-02-01 10:52:33,502 INFO [Thread-17] mapred.TestTextInputFormat
> (TestTextInputFormat.java:verifyPartitions(317)) -
> splits[1]=file:/work/count-mismatch2/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/TestTextInputFormat/test.bz2:3+3
> count=99{code}
> > The input format read only 99 records but not 100 records
>
> 2. Duplicate Record:
> 2 input splits has same BZip2 records when the input split size is small
>
> Set the split size to 1 and tested to load 100 records (0, 1, 2..99)
>
> {code:java}
> 2018-02-01 11:18:49,309 INFO [Thread-17] mapred.TestTextInputFormat
> (TestTextInputFormat.java:verifyPartitions(318)) - splits[3]=file
> /work/count-mismatch2/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/TestTextInputFormat/test.bz2:3+1
> count=99
> 2018-02-01 11:18:49,310 WARN [Thread-17] mapred.TestTextInputFormat
> (TestTextInputFormat.java:verifyPartitions(308)) - conflict with 1 in split 4
> at position 8
> {code}
>
> I experienced this error when I execute Spark (SparkSQL) job under the
> following conditions:
> * The file size of the input files are small (around 1KB)
> * Hadoop cluster has many slave nodes (able to launch many executor tasks)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15206) BZip2 drops and duplicates records when input split size is small
[
https://issues.apache.org/jira/browse/HADOOP-15206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366061#comment-16366061
]
Aki Tanaka commented on HADOOP-15206:
-
Thank you. Since this is the first time to apply a patch to the community, I
apologize for having bothered you.
Update the patch, I think I fixed the points you pointed out, please let me
know if there is anything I should do.
> BZip2 drops and duplicates records when input split size is small
> -
>
> Key: HADOOP-15206
> URL: https://issues.apache.org/jira/browse/HADOOP-15206
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.3, 3.0.0
>Reporter: Aki Tanaka
>Assignee: Aki Tanaka
>Priority: Major
> Attachments: HADOOP-15206-test.patch, HADOOP-15206.001.patch,
> HADOOP-15206.002.patch, HADOOP-15206.003.patch, HADOOP-15206.004.patch,
> HADOOP-15206.005.patch, HADOOP-15206.006.patch, HADOOP-15206.007.patch,
> HADOOP-15206.008.patch
>
>
> BZip2 can drop and duplicate record when input split file is small. I
> confirmed that this issue happens when the input split size is between 1byte
> and 4bytes.
> I am seeing the following 2 problem behaviors.
>
> 1. Drop record:
> BZip2 skips the first record in the input file when the input split size is
> small
>
> Set the split size to 3 and tested to load 100 records (0, 1, 2..99)
> {code:java}
> 2018-02-01 10:52:33,502 INFO [Thread-17] mapred.TestTextInputFormat
> (TestTextInputFormat.java:verifyPartitions(317)) -
> splits[1]=file:/work/count-mismatch2/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/TestTextInputFormat/test.bz2:3+3
> count=99{code}
> > The input format read only 99 records but not 100 records
>
> 2. Duplicate Record:
> 2 input splits has same BZip2 records when the input split size is small
>
> Set the split size to 1 and tested to load 100 records (0, 1, 2..99)
>
> {code:java}
> 2018-02-01 11:18:49,309 INFO [Thread-17] mapred.TestTextInputFormat
> (TestTextInputFormat.java:verifyPartitions(318)) - splits[3]=file
> /work/count-mismatch2/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/TestTextInputFormat/test.bz2:3+1
> count=99
> 2018-02-01 11:18:49,310 WARN [Thread-17] mapred.TestTextInputFormat
> (TestTextInputFormat.java:verifyPartitions(308)) - conflict with 1 in split 4
> at position 8
> {code}
>
> I experienced this error when I execute Spark (SparkSQL) job under the
> following conditions:
> * The file size of the input files are small (around 1KB)
> * Hadoop cluster has many slave nodes (able to launch many executor tasks)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14060) HTTP servlet /logs should require authentication and authorization
[
https://issues.apache.org/jira/browse/HADOOP-14060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HADOOP-14060:
---
Priority: Critical (was: Blocker)
> HTTP servlet /logs should require authentication and authorization
> --
>
> Key: HADOOP-14060
> URL: https://issues.apache.org/jira/browse/HADOOP-14060
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 3.0.0-alpha4
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Critical
> Attachments: HADOOP-14060-tmp.001.patch
>
>
> HADOOP-14047 makes KMS call {{HttpServer2#setACL}}. Access control works fine
> for /conf, /jmx, /logLevel, and /stacks, but not for /logs.
> The code in {{AdminAuthorizedServlet#doGet}} for /logs and
> {{ConfServlet#doGet}} for /conf are quite similar. This makes me believe that
> /logs should subject to the same access control as intended by the original
> developer.
> IMHO this could either be my misconfiguration or there is a bug somewhere in
> {{HttpServer2}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14396) Add builder interface to FileContext
[
https://issues.apache.org/jira/browse/HADOOP-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366046#comment-16366046
]
Lei (Eddy) Xu commented on HADOOP-14396:
[~leftnoteasy] sure I can do it.
[~ste...@apache.org] Have you started to work on it? or you will give a review
on it? I'd like to make sure that our efforts dont overlap :)
> Add builder interface to FileContext
>
>
> Key: HADOOP-14396
> URL: https://issues.apache.org/jira/browse/HADOOP-14396
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 2.9.0, 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Blocker
> Attachments: HADOOP-14396.00.patch
>
>
> Add builder interface for {{FileContext#create}} and {{FileContext#append}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration
[
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365985#comment-16365985
]
Sharad Sonker commented on HADOOP-13972:
Thank [~ste...@apache.org] for committing this and also for helping me out with
the process.
> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: Sharad Sonker
>Priority: Major
> Fix For: 3.1.0, 2.10.0, 3.0.2
>
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names,
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted
> first, then fall back to the global keys.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14831) Über-jira: S3a phase IV: Hadoop 3.1 features
[ https://issues.apache.org/jira/browse/HADOOP-14831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365981#comment-16365981 ] Steve Loughran commented on HADOOP-14831: - Wangda. moved pretty much everything out which isn't ready to go in. There's one bug HADOOP-14937 which I need to look at to see if I can replicate > Über-jira: S3a phase IV: Hadoop 3.1 features > > > Key: HADOOP-14831 > URL: https://issues.apache.org/jira/browse/HADOOP-14831 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Priority: Blocker > > All the S3/S3A features targeting Hadoop 3.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15231) WavefrontSink for Hadoop Metrics2
[ https://issues.apache.org/jira/browse/HADOOP-15231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365956#comment-16365956 ] Howard Yoo edited comment on HADOOP-15231 at 2/15/18 5:26 PM: -- Right, extending and re-using the code is always a good option. However, if you take a look at the GraphiteSink, you'd see these issues: # Much of its internal variables are defined as 'private' members, making anything extending it hard to use those. # Log messages have 'graphite' which makes it harder for others to change that, unless overwriting pretty much everything. So, I can extend the class as you suggested, but at the end, I would have to overwrite most of the existing methods anyway which will probably make it end up in a similar situation. So, generally agree that extending something is useful, but I don't see too much values in this case. Also, I feel that I need to create a unit-tests for this. Will it make it much better if I also add unit test for Wavefront format? Makes sense, and seems it is something that's required also. was (Author: howardyoo): Agreed, Will change the code to extend the GraphiteSink instead of copying it... Also, I believe I need to add unit tests as well - would it be better if I added the unit tests? > WavefrontSink for Hadoop Metrics2 > - > > Key: HADOOP-15231 > URL: https://issues.apache.org/jira/browse/HADOOP-15231 > Project: Hadoop Common > Issue Type: New Feature > Components: metrics >Reporter: Howard Yoo >Priority: Minor > Labels: metrics, wavefront > > Wavefront is a SaaS based large scale real time metrics monitoring and > analytic system capable of monitoring many different source systems. There > are several Sinks available in Hadoop in order to capture various metrics to > external system. The Wavefront data format follows a similar format as that > of Graphite, with the addition of native point tag support and source value. > The details are outlined here: > [https://docs.wavefront.com/wavefront_data_format.html] > It would be greatly helpful for both Hadoop and Wavefront to have a native > integration using Wavefront Sink to send Hadoop related metrics to Wavefront > Server. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13551) hook up AwsSdkMetrics to hadoop metrics
[
https://issues.apache.org/jira/browse/HADOOP-13551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13551:
Priority: Major (was: Critical)
> hook up AwsSdkMetrics to hadoop metrics
> ---
>
> Key: HADOOP-13551
> URL: https://issues.apache.org/jira/browse/HADOOP-13551
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Sean Mackrory
>Priority: Major
>
> There's an API in {{com.amazonaws.metrics.AwsSdkMetrics}} to give access to
> the internal metrics of the AWS libraries. We might want to get at those
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13551) hook up AwsSdkMetrics to hadoop metrics
[
https://issues.apache.org/jira/browse/HADOOP-13551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13551:
Parent Issue: HADOOP-15220 (was: HADOOP-14831)
> hook up AwsSdkMetrics to hadoop metrics
> ---
>
> Key: HADOOP-13551
> URL: https://issues.apache.org/jira/browse/HADOOP-13551
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Sean Mackrory
>Priority: Critical
>
> There's an API in {{com.amazonaws.metrics.AwsSdkMetrics}} to give access to
> the internal metrics of the AWS libraries. We might want to get at those
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
