[jira] [Commented] (HADOOP-14199) TestFsShellList.testList fails on windows: illegal filenames
[
https://issues.apache.org/jira/browse/HADOOP-14199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932178#comment-15932178
]
John Zhuge commented on HADOOP-14199:
-
In [MSDN Naming Files, Paths, and
Namespaces|https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#naming_conventions],
control character {{\b}} and {{\t}} are not explicitly forbidden, so I think
they are allowed. I don't have any Windows host, will verify once I set up one.
If they are allowed, could the issue be the filename with escape passed to
NativeIO? Windows does NOT allow backslash.
> TestFsShellList.testList fails on windows: illegal filenames
>
>
> Key: HADOOP-14199
> URL: https://issues.apache.org/jira/browse/HADOOP-14199
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Affects Versions: 2.8.0
> Environment: win64
>Reporter: Steve Loughran
>Priority: Minor
>
> {{TestFsShellList.testList}} fails setting up the files to test against
> {code}
> org.apache.hadoop.io.nativeio.NativeIOException: The filename, directory
> name, or volume label syntax is incorrect.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14199) TestFsShellList.testList fails on windows: illegal filenames
[
https://issues.apache.org/jira/browse/HADOOP-14199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge reassigned HADOOP-14199:
---
Assignee: John Zhuge
> TestFsShellList.testList fails on windows: illegal filenames
>
>
> Key: HADOOP-14199
> URL: https://issues.apache.org/jira/browse/HADOOP-14199
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Affects Versions: 2.8.0
> Environment: win64
>Reporter: Steve Loughran
>Assignee: John Zhuge
>Priority: Minor
>
> {{TestFsShellList.testList}} fails setting up the files to test against
> {code}
> org.apache.hadoop.io.nativeio.NativeIOException: The filename, directory
> name, or volume label syntax is incorrect.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13811) s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to sanitize XML document destined for handler class
[
https://issues.apache.org/jira/browse/HADOOP-13811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Fabbri reassigned HADOOP-13811:
-
Assignee: (was: Aaron Fabbri)
> s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to
> sanitize XML document destined for handler class
> -
>
> Key: HADOOP-13811
> URL: https://issues.apache.org/jira/browse/HADOOP-13811
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0, 2.7.3
>Reporter: Steve Loughran
>
> Sometimes, occasionally, getFileStatus() fails with a stack trace starting
> with {{com.amazonaws.AmazonClientException: Failed to sanitize XML document
> destined for handler class}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14098) Cache object meta info in client side to avoid frequent requests to OSS/S3
[ https://issues.apache.org/jira/browse/HADOOP-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932071#comment-15932071 ] Genmao Yu commented on HADOOP-14098: [~drankye] We have being testing it locally. We will provide performance data when it really has positive effect. > Cache object meta info in client side to avoid frequent requests to OSS/S3 > -- > > Key: HADOOP-14098 > URL: https://issues.apache.org/jira/browse/HADOOP-14098 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 3.0.0-alpha2 >Reporter: Genmao Yu > > Open this JIRA to research and address the potential request performance > issue. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14038) Rename ADLS credential properties
[
https://issues.apache.org/jira/browse/HADOOP-14038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932061#comment-15932061
]
John Zhuge commented on HADOOP-14038:
-
[~ste...@apache.org] Will include your comment in the next patch.
[~vishwajeet.dusane] Thanks for the review. Totally agree with you on that we
should be careful not to break any existing code.
It is a good idea to add what you suggested. It will cover one use case:
{{Configuration#set}} is called with an old key, then conf is read with the new
key.
There is another use case when a config file with old keys is loaded. These are
2 different code paths in Configuration class. I will add an unit test for it
as well.
For TestValidateConfiguration, I will keep it then since HADOOP-13037 reviewed
this class already even though IMHO the accidental modification is already
mitigated by 3 measures:
# The properties are in a class called {{AdlConfKeys}} which indicates these
are conf keys.
# The properties all have {{_KEY}} suffix to indicate these are conf keys.
# The property values are in the format of {{aa.bb.cc.dd}}, somewhat a hint of
property names.
> Rename ADLS credential properties
> -
>
> Key: HADOOP-14038
> URL: https://issues.apache.org/jira/browse/HADOOP-14038
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/adl
>Affects Versions: 3.0.0-alpha3
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Attachments: HADOOP-14038.001.patch, HADOOP-14038.002.patch,
> HADOOP-14038.003.patch, HADOOP-14038.004.patch, HADOOP-14038.005.patch,
> HADOOP-14038.006.patch
>
>
> Add ADLS credential configuration properties to {{core-default.xml}}.
> Set/document the default value for
> {{dfs.adls.oauth2.access.token.provider.type}} to {{ClientCredential}}.
> Fix {{AdlFileSystem#getAccessTokenProvider}} which implies the provider type
> is {{Custom}}.
> Fix several unit tests that set {{dfs.adls.oauth2.access.token.provider}} but
> does not set {{dfs.adls.oauth2.access.token.provider.type}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13665) Erasure Coding codec should support fallback coder
[
https://issues.apache.org/jira/browse/HADOOP-13665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932055#comment-15932055
]
Kai Zheng commented on HADOOP-13665:
Thanks all for the discussion and I took a quick look.
{code}
+conf.set(CodecUtil.IO_ERASURECODE_CODEC_CODERS,
+ErasureCodeConstants.RS_CODEC_NAME + "," +
+ErasureCodeConstants.XOR_CODEC_NAME);
{code}
It doesn't make sense to mix and configure something for RS codec and XOR codec
together. Instead, it should be to allow configuring more coders for the same
codec in order, IIUC.
For example, for the RS codec, we have the pure Java coder and also the ISA-L
based native coder. As Chuang proposed, what we want to do is to allow
configuring the two coders using the property like
{{io.erasurecode.codec.rs.rawcoders}} in addition to what we have already as
below
{code}
/** Raw coder factory for the RS codec. */
public static final String IO_ERASURECODE_CODEC_RS_RAWCODER_KEY =
"io.erasurecode.codec.rs.rawcoder";
public static final String IO_ERASURECODE_CODEC_RS_RAWCODER_DEFAULT =
RSRawErasureCoderFactory.class.getCanonicalName();
/** Raw coder factory for the RS legacy codec. */
public static final String IO_ERASURECODE_CODEC_RS_LEGACY_RAWCODER_KEY =
"io.erasurecode.codec.rs-legacy.rawcoder";
public static final String IO_ERASURECODE_CODEC_RS_LEGACY_RAWCODER_DEFAULT =
RSRawErasureCoderFactoryLegacy.class.getCanonicalName();
/** Raw coder factory for the XOR codec. */
public static final String IO_ERASURECODE_CODEC_XOR_RAWCODER_KEY =
"io.erasurecode.codec.xor.rawcoder";
public static final String IO_ERASURECODE_CODEC_XOR_RAWCODER_DEFAULT =
XORRawErasureCoderFactory.class.getCanonicalName();
{code}
> Erasure Coding codec should support fallback coder
> --
>
> Key: HADOOP-13665
> URL: https://issues.apache.org/jira/browse/HADOOP-13665
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: io
>Reporter: Wei-Chiu Chuang
>Assignee: Kai Sasaki
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-13665.01.patch, HADOOP-13665.02.patch,
> HADOOP-13665.03.patch, HADOOP-13665.04.patch, HADOOP-13665.05.patch
>
>
> The current EC codec supports a single coder only (by default pure Java
> implementation). If the native coder is specified but is unavailable, it
> should fallback to pure Java implementation.
> One possible solution is to follow the convention of existing Hadoop native
> codec, such as transport encryption (see {{CryptoCodec.java}}). It supports
> fallback by specifying two or multiple coders as the value of property, and
> loads coders in order.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14201) Fix some failing tests on windows
[
https://issues.apache.org/jira/browse/HADOOP-14201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932034#comment-15932034
]
Hadoop QA commented on HADOOP-14201:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HADOOP-14201 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-14201 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12859474/HADOOP-14201-001.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11853/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Fix some failing tests on windows
> -
>
> Key: HADOOP-14201
> URL: https://issues.apache.org/jira/browse/HADOOP-14201
> Project: Hadoop Common
> Issue Type: Task
> Components: test
>Affects Versions: 2.8.0
> Environment: Windows Server 2012.
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-14201-001.patch
>
>
> Some of the 2.8.0 tests are failing locally, without much in the way of
> diagnostics. They may be false alarms related to system, VM setup,
> performance, or they may be a sign of a problem.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14202) fix jsvc/secure user inconsistencies
Allen Wittenauer created HADOOP-14202: - Summary: fix jsvc/secure user inconsistencies Key: HADOOP-14202 URL: https://issues.apache.org/jira/browse/HADOOP-14202 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0-alpha2 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Post-HADOOP-13341 and (more importantly) HADOOP-13673, there has been a major effort on making the configuration environment variables consistent among all the projects. The vast majority of vars now look like (command)_(subcommand)_(etc). Two hold outs are HADOOP_SECURE_DN_USER and HADOOP_PRIVILEGED_NFS_USER. Additionally, there is * no generic handling * no documentation for anyone * no safety checks to make sure things are defined In order to fix all of this, we should: * deprecate the previous vars using the deprecation function, updating the HDFS documentation that references them * add generic (command)_(subcommand)_SECURE_USER support * add some verification for the previously mentioned var * add some docs to UnixShellGuide.md -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14202) fix jsvc/secure user var inconsistencies
[ https://issues.apache.org/jira/browse/HADOOP-14202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-14202: -- Summary: fix jsvc/secure user var inconsistencies (was: fix jsvc/secure user inconsistencies) > fix jsvc/secure user var inconsistencies > > > Key: HADOOP-14202 > URL: https://issues.apache.org/jira/browse/HADOOP-14202 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 3.0.0-alpha2 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > > Post-HADOOP-13341 and (more importantly) HADOOP-13673, there has been a major > effort on making the configuration environment variables consistent among all > the projects. The vast majority of vars now look like > (command)_(subcommand)_(etc). Two hold outs are HADOOP_SECURE_DN_USER and > HADOOP_PRIVILEGED_NFS_USER. > Additionally, there is > * no generic handling > * no documentation for anyone > * no safety checks to make sure things are defined > In order to fix all of this, we should: > * deprecate the previous vars using the deprecation function, updating the > HDFS documentation that references them > * add generic (command)_(subcommand)_SECURE_USER support > * add some verification for the previously mentioned var > * add some docs to UnixShellGuide.md -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14201) Fix some failing tests on windows
[ https://issues.apache.org/jira/browse/HADOOP-14201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931885#comment-15931885 ] Steve Loughran commented on HADOOP-14201: - + I'm not looking at the gzip failures here either > Fix some failing tests on windows > - > > Key: HADOOP-14201 > URL: https://issues.apache.org/jira/browse/HADOOP-14201 > Project: Hadoop Common > Issue Type: Task > Components: test >Affects Versions: 2.8.0 > Environment: Windows Server 2012. >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-14201-001.patch > > > Some of the 2.8.0 tests are failing locally, without much in the way of > diagnostics. They may be false alarms related to system, VM setup, > performance, or they may be a sign of a problem. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14201) Fix some failing tests on windows
[ https://issues.apache.org/jira/browse/HADOOP-14201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14201: Attachment: HADOOP-14201-001.patch Patch 001 this adds diagnostics to help find why tests are failing, * StatsDSink: binding diags of HADOOP-14200 * TestFsShellList: fixes of HADOOP-14199 * TestShellBasedUnixGroupsMapping don't run the tests windows doesn't like * TestLambdaTestUtils allow for slower machines by not worrying about the retry count, other than a retry must have happened. * TestWinUtils: more diagnostics of failures TestWinUtils was still failing;in the "insufficient memory," test; the fix for that appears to be "ask for much more heap". Maybe JRE changes to heap allocation caused the JVM to not ask for that full -Xmx amount, so hide the problem. My windows test VM is still having problems with hostnames and things; I've given up worrying about that. > Fix some failing tests on windows > - > > Key: HADOOP-14201 > URL: https://issues.apache.org/jira/browse/HADOOP-14201 > Project: Hadoop Common > Issue Type: Task > Components: test >Affects Versions: 2.8.0 > Environment: Windows Server 2012. >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-14201-001.patch > > > Some of the 2.8.0 tests are failing locally, without much in the way of > diagnostics. They may be false alarms related to system, VM setup, > performance, or they may be a sign of a problem. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14201) Fix some failing tests on windows
[ https://issues.apache.org/jira/browse/HADOOP-14201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14201: Status: Patch Available (was: Open) > Fix some failing tests on windows > - > > Key: HADOOP-14201 > URL: https://issues.apache.org/jira/browse/HADOOP-14201 > Project: Hadoop Common > Issue Type: Task > Components: test >Affects Versions: 2.8.0 > Environment: Windows Server 2012. >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-14201-001.patch > > > Some of the 2.8.0 tests are failing locally, without much in the way of > diagnostics. They may be false alarms related to system, VM setup, > performance, or they may be a sign of a problem. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14200) StatsDSink error reporting to use NetUtils for failure diagnostics
[
https://issues.apache.org/jira/browse/HADOOP-14200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931860#comment-15931860
]
Steve Loughran commented on HADOOP-14200:
-
new messages are better
{code}
Running org.apache.hadoop.metrics2.impl.TestStatsDMetrics
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.468 sec <<<
FAILURE! - in org.apache.hadoop.metrics2.i
mpl.TestStatsDMetrics
testPutMetrics2(org.apache.hadoop.metrics2.impl.TestStatsDMetrics) Time
elapsed: 0.421 sec <<< ERROR!
org.apache.hadoop.metrics2.MetricsException: Error writing metric to StatsD:
java.net.BindException: Problem binding to
[localhost:0] java.net.BindException: Cannot assign requested address: Datagram
send failed; For more details see: http
://wiki.apache.org/hadoop/BindException
at java.net.TwoStacksPlainDatagramSocketImpl.send(Native Method)
at java.net.DatagramSocket.send(DatagramSocket.java:693)
at
org.apache.hadoop.metrics2.sink.StatsDSink$StatsD.write(StatsDSink.java:204)
at
org.apache.hadoop.metrics2.sink.StatsDSink.writeMetric(StatsDSink.java:151)
at
org.apache.hadoop.metrics2.sink.StatsDSink.putMetrics(StatsDSink.java:144)
at
org.apache.hadoop.metrics2.impl.TestStatsDMetrics.testPutMetrics2(TestStatsDMetrics.java:109)
testPutMetrics(org.apache.hadoop.metrics2.impl.TestStatsDMetrics) Time
elapsed: 0 sec <<< ERROR!
org.apache.hadoop.metrics2.MetricsException: Error writing metric to StatsD:
java.net.BindException: Problem binding to
[localhost:0] java.net.BindException: Cannot assign requested address: Datagram
send failed; For more details see: http
://wiki.apache.org/hadoop/BindException
at java.net.TwoStacksPlainDatagramSocketImpl.send(Native Method)
at java.net.DatagramSocket.send(DatagramSocket.java:693)
at
org.apache.hadoop.metrics2.sink.StatsDSink$StatsD.write(StatsDSink.java:204)
at
org.apache.hadoop.metrics2.sink.StatsDSink.writeMetric(StatsDSink.java:151)
at
org.apache.hadoop.metrics2.sink.StatsDSink.putMetrics(StatsDSink.java:144)
at
org.apache.hadoop.metrics2.impl.TestStatsDMetrics.testPutMetrics(TestStatsDMetrics.java:74)
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
support was removed in 8.0
Results :
Tests in error:
TestStatsDMetrics.testPutMetrics2:109 » Metrics Error writing metric to
StatsD...
TestStatsDMetrics.testPutMetrics:74 » Metrics Error writing metric to StatsD:
...
{code}
> StatsDSink error reporting to use NetUtils for failure diagnostics
> --
>
> Key: HADOOP-14200
> URL: https://issues.apache.org/jira/browse/HADOOP-14200
> Project: Hadoop Common
> Issue Type: Improvement
> Components: metrics
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>
> When statsd publisher fails, it doesn't include the hostname/port information
> needed to begin debugging the problem. That's what NetUtils is for
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14201) Fix some failing tests on windows
[
https://issues.apache.org/jira/browse/HADOOP-14201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931841#comment-15931841
]
Steve Loughran edited comment on HADOOP-14201 at 3/19/17 5:32 PM:
--
h2. TestShellBasedBasedUnixGroupsMapping:
This shouldn't be running on Windows. Modified test results
{code}
Tests run: 4, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 0.656 sec <<<
FAILURE! - in org.apache.hadoop.security.T
estShellBasedUnixGroupsMapping
testGetNumericGroupsResolvable(org.apache.hadoop.security.TestShellBasedUnixGroupsMapping)
Time elapsed: 0.375 sec <<<
FAILURE!
java.lang.AssertionError: wrong group size [23,23 groupname zzz] expected:<3>
but was:<2>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.security.TestShellBasedUnixGroupsMapping.assertGroupValues(TestShellBasedUnixGroupsMapping.
java:207)
at
org.apache.hadoop.security.TestShellBasedUnixGroupsMapping.testGetNumericGroupsResolvable(TestShellBasedUnixG
roupsMapping.java:159)
testGetGroupsNotResolvable(org.apache.hadoop.security.TestShellBasedUnixGroupsMapping)
Time elapsed: 0 sec <<< FAILURE
!
java.lang.AssertionError: wrong group size [] expected:<2> but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.security.TestShellBasedUnixGroupsMapping.assertGroupValues(TestShellBasedUnixGroupsMapping.
java:207)
at
org.apache.hadoop.security.TestShellBasedUnixGroupsMapping.testGetGroupsNotResolvable(TestShellBasedUnixGroup
sMapping.java:113)
testGetGroupsResolvable(org.apache.hadoop.security.TestShellBasedUnixGroupsMapping)
Time elapsed: 0.023 sec <<< FAILUR
E!
java.lang.AssertionError: wrong group size [abc,def abc hij] expected:<3> but
was:<2>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.security.TestShellBasedUnixGroupsMapping.assertGroupValues(TestShellBasedUnixGroupsMapping.
java:207)
at
org.apache.hadoop.security.TestShellBasedUnixGroupsMapping.testGetGroupsResolvable(TestShellBasedUnixGroupsMa
pping.java:202)
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
support was removed in 8.0
Results :
Failed tests:
TestShellBasedUnixGroupsMapping.testGetNumericGroupsResolvable:159->assertGroupValues:207
wrong group size [23,23 grou
pname zzz] expected:<3> but was:<2>
TestShellBasedUnixGroupsMapping.testGetGroupsNotResolvable:113->assertGroupValues:207
wrong group size [] expected:<2>
but was:<0>
TestShellBasedUnixGroupsMapping.testGetGroupsResolvable:202->assertGroupValues:207
wrong group size [abc,def abc hij]
expected:<3> but was:<2>
{code}
logs:
{code}
2017-03-19 17:21:22,047 WARN security.ShellBasedUnixGroupsMapping
(ShellBasedUnixGroupsMapping.java:getUnixGroups(136)) - unable to return groups
for user foobarusernotexist
PartialGroupNameException Does not support partial group name resolution on
Windows. id: foobarusernotexist: No such user
at
org.apache.hadoop.security.ShellBasedUnixGroupsMapping.resolvePartialGroupNames(ShellBasedUnixGroupsMapping.java:208)
at
org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:133)
at
org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:72)
at
org.apache.hadoop.security.TestShellBasedUnixGroupsMapping.testGetGroupsNonexistentUser(TestShellBasedUnixGroupsMapping.java:69)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
[jira] [Commented] (HADOOP-14201) Fix some failing tests on windows
[
https://issues.apache.org/jira/browse/HADOOP-14201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931841#comment-15931841
]
Steve Loughran commented on HADOOP-14201:
-
h2. TestShellBasedBasedUnixGroupsMapping:
This shouldn't be running on Windows. Modified test results
{code}
Tests run: 4, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 0.656 sec <<<
FAILURE! - in org.apache.hadoop.security.T
estShellBasedUnixGroupsMapping
testGetNumericGroupsResolvable(org.apache.hadoop.security.TestShellBasedUnixGroupsMapping)
Time elapsed: 0.375 sec <<<
FAILURE!
java.lang.AssertionError: wrong group size [23,23 groupname zzz] expected:<3>
but was:<2>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.security.TestShellBasedUnixGroupsMapping.assertGroupValues(TestShellBasedUnixGroupsMapping.
java:207)
at
org.apache.hadoop.security.TestShellBasedUnixGroupsMapping.testGetNumericGroupsResolvable(TestShellBasedUnixG
roupsMapping.java:159)
testGetGroupsNotResolvable(org.apache.hadoop.security.TestShellBasedUnixGroupsMapping)
Time elapsed: 0 sec <<< FAILURE
!
java.lang.AssertionError: wrong group size [] expected:<2> but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.security.TestShellBasedUnixGroupsMapping.assertGroupValues(TestShellBasedUnixGroupsMapping.
java:207)
at
org.apache.hadoop.security.TestShellBasedUnixGroupsMapping.testGetGroupsNotResolvable(TestShellBasedUnixGroup
sMapping.java:113)
testGetGroupsResolvable(org.apache.hadoop.security.TestShellBasedUnixGroupsMapping)
Time elapsed: 0.023 sec <<< FAILUR
E!
java.lang.AssertionError: wrong group size [abc,def abc hij] expected:<3> but
was:<2>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.security.TestShellBasedUnixGroupsMapping.assertGroupValues(TestShellBasedUnixGroupsMapping.
java:207)
at
org.apache.hadoop.security.TestShellBasedUnixGroupsMapping.testGetGroupsResolvable(TestShellBasedUnixGroupsMa
pping.java:202)
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
support was removed in 8.0
Results :
Failed tests:
TestShellBasedUnixGroupsMapping.testGetNumericGroupsResolvable:159->assertGroupValues:207
wrong group size [23,23 grou
pname zzz] expected:<3> but was:<2>
TestShellBasedUnixGroupsMapping.testGetGroupsNotResolvable:113->assertGroupValues:207
wrong group size [] expected:<2>
but was:<0>
TestShellBasedUnixGroupsMapping.testGetGroupsResolvable:202->assertGroupValues:207
wrong group size [abc,def abc hij]
expected:<3> but was:<2>
{code}
logs:
{code}
2017-03-19 17:21:22,047 WARN security.ShellBasedUnixGroupsMapping
(ShellBasedUnixGroupsMapping.java:getUnixGroups(136)) - unable to return groups
for user foobarusernotexist
PartialGroupNameException Does not support partial group name resolution on
Windows. id: foobarusernotexist: No such user
at
org.apache.hadoop.security.ShellBasedUnixGroupsMapping.resolvePartialGroupNames(ShellBasedUnixGroupsMapping.java:208)
at
org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:133)
at
org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:72)
at
org.apache.hadoop.security.TestShellBasedUnixGroupsMapping.testGetGroupsNonexistentUser(TestShellBasedUnixGroupsMapping.java:69)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
[jira] [Commented] (HADOOP-14201) Fix some failing tests on windows
[
https://issues.apache.org/jira/browse/HADOOP-14201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931828#comment-15931828
]
Steve Loughran commented on HADOOP-14201:
-
{code}
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.555 sec <<<
FAILURE! - in org.apache.hadoop.fs.TestFsS
hellList
testList(org.apache.hadoop.fs.TestFsShellList) Time elapsed: 0.095 sec <<<
ERROR!
org.apache.hadoop.io.nativeio.NativeIOException: The filename, directory name,
or volume label syntax is incorrect.
at
org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Native
Method)
at
org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileOutputStreamWithMode(NativeIO.java:556)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:231)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:221)
at
org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:319)
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:308)
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:339)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.(ChecksumFileSystem.java:399)
at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:462)
at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:441)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:928)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:806)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:795)
at
org.apache.hadoop.fs.TestFsShellList.createFile(TestFsShellList.java:57)
at
org.apache.hadoop.fs.TestFsShellList.testList(TestFsShellList.java:69)
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
support was removed in 8.0
Running org.apache.hadoop.fs.TestDU
Tests run: 3, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 17.349 sec <<<
FAILURE! - in org.apache.hadoop.fs.TestDU
testDU(org.apache.hadoop.fs.TestDU) Time elapsed: 11.058 sec <<< FAILURE!
junit.framework.AssertionFailedError: Invalid on-disk size
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.assertTrue(Assert.java:22)
at junit.framework.TestCase.assertTrue(TestCase.java:192)
at org.apache.hadoop.fs.TestDU.testDU(TestDU.java:87)
testDUSetInitialValue(org.apache.hadoop.fs.TestDU) Time elapsed: 6.084 sec
<<< FAILURE!
junit.framework.AssertionFailedError: Usage didn't get updated
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.assertTrue(Assert.java:22)
at junit.framework.TestCase.assertTrue(TestCase.java:192)
at org.apache.hadoop.fs.TestDU.testDUSetInitialValue(TestDU.java:133)
testAppendBlockCompression(org.apache.hadoop.io.TestSequenceFileAppend)
Time elapsed: 0.028 sec <<< ERROR!
java.io.IOException: not a gzip file
at
org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.processBasicHeader(BuiltInGzipDecompressor.java:49
6)
at
org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.executeHeaderState(BuiltInGzipDecompressor.java:25
7)
at
org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.decompress(BuiltInGzipDecompressor.java:186)
at
org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:111)
at
org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:105)
at
org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:92)
at java.io.DataInputStream.readByte(DataInputStream.java:265)
at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308)
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:329)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2612)
at
org.apache.hadoop.io.TestSequenceFileAppend.verify2Values(TestSequenceFileAppend.java:362)
at
org.apache.hadoop.io.TestSequenceFileAppend.testAppendBlockCompression(TestSequenceFileAppend.java:194)
testAppendSort(org.apache.hadoop.io.TestSequenceFileAppend) Time elapsed:
0.024 sec <<< ERROR!
java.io.IOException: not a gzip file
at
org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.processBasicHeader(BuiltInGzipDecompressor.java:49
6)
at
org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.executeHeaderState(BuiltInGzipDecompressor.java:25
7)
at
[jira] [Created] (HADOOP-14201) Fix some failing tests on windows
Steve Loughran created HADOOP-14201: --- Summary: Fix some failing tests on windows Key: HADOOP-14201 URL: https://issues.apache.org/jira/browse/HADOOP-14201 Project: Hadoop Common Issue Type: Task Components: test Affects Versions: 2.8.0 Environment: Windows Server 2012. Reporter: Steve Loughran Assignee: Steve Loughran Priority: Minor Some of the 2.8.0 tests are failing locally, without much in the way of diagnostics. They may be false alarms related to system, VM setup, performance, or they may be a sign of a problem. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14200) StatsDSink error reporting to use NetUtils for failure diagnostics
[
https://issues.apache.org/jira/browse/HADOOP-14200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931747#comment-15931747
]
Steve Loughran commented on HADOOP-14200:
-
the test logs probably have more detail,
1. The inner socket stuff isn't being translated
2. the rethrown message doesn't include the error of the inner exception in its
text
The combination means that the root cause is being lost, and even that lacks
diagnostics information
{code}
testPutMetrics(org.apache.hadoop.metrics2.impl.TestStatsDMetrics) Time
elapsed: 0.005 sec <<< ERROR!org.apache.hadoop.metrics2.MetricsException:
Error writing metric to StatsD
at java.net.TwoStacksPlainDatagramSocketImpl.send(Native Method)
at java.net.DatagramSocket.send(DatagramSocket.java:693)
at
org.apache.hadoop.metrics2.sink.StatsDSink$StatsD.write(StatsDSink.java:203)
at
org.apache.hadoop.metrics2.sink.StatsDSink.writeMetric(StatsDSink.java:151)
at
org.apache.hadoop.metrics2.sink.StatsDSink.putMetrics(StatsDSink.java:144)
at
org.apache.hadoop.metrics2.impl.TestStatsDMetrics.testPutMetrics(TestStatsDMetrics.java:74)
{code}
> StatsDSink error reporting to use NetUtils for failure diagnostics
> --
>
> Key: HADOOP-14200
> URL: https://issues.apache.org/jira/browse/HADOOP-14200
> Project: Hadoop Common
> Issue Type: Improvement
> Components: metrics
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>
> When statsd publisher fails, it doesn't include the hostname/port information
> needed to begin debugging the problem. That's what NetUtils is for
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14200) StatsDSink error reporting to use NetUtils for failure diagnostics
Steve Loughran created HADOOP-14200: --- Summary: StatsDSink error reporting to use NetUtils for failure diagnostics Key: HADOOP-14200 URL: https://issues.apache.org/jira/browse/HADOOP-14200 Project: Hadoop Common Issue Type: Improvement Components: metrics Affects Versions: 2.8.0 Reporter: Steve Loughran When statsd publisher fails, it doesn't include the hostname/port information needed to begin debugging the problem. That's what NetUtils is for -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13079) Add -q option to Ls to print ? instead of non-printable characters
[
https://issues.apache.org/jira/browse/HADOOP-13079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931741#comment-15931741
]
Steve Loughran commented on HADOOP-13079:
-
Test doesn't work on Windows I'm afraid; HADOOP-14199 covers it. Probably
simplest just to skip the test entirely there. Also looking at the test file,
it should be using {{GenericTestUtils.getTempPath()}} to get the temp dir
> Add -q option to Ls to print ? instead of non-printable characters
> --
>
> Key: HADOOP-13079
> URL: https://issues.apache.org/jira/browse/HADOOP-13079
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: HADOOP-13079.001.patch, HADOOP-13079.002.patch,
> HADOOP-13079.003.patch, HADOOP-13079.004.patch
>
>
> Add option {{-q}} to "hdfs dfs -ls" to print non-printable characters as "?".
> Non-printable characters are defined by
> [isprint(3)|http://linux.die.net/man/3/isprint] according to the current
> locale.
> Default to {{-q}} behavior on terminal; otherwise, print raw characters. See
> the difference in these 2 command lines:
> * {{hadoop fs -ls /dir}}
> * {{hadoop fs -ls /dir | od -c}}
> In C, {{isatty(STDOUT_FILENO)}} is used to find out whether the output is a
> terminal. Since Java doesn't have {{isatty}}, I will use JNI to call C
> {{isatty()}} because the closest test {{System.console() == null}} does not
> work in some cases.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14199) TestFsShellList.testList fails on windows: illegal filenames
[
https://issues.apache.org/jira/browse/HADOOP-14199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931736#comment-15931736
]
Steve Loughran commented on HADOOP-14199:
-
Stack trace. This is on the line
{code}
createFile(new Path(testRootDir, "abc\bd\tef"));
{code}
; we'll have to assume that the names in the path are not valid on NTFS.
{code}
TestFsShellListtestList(org.apache.hadoop.fs.TestFsShellList) Time elapsed:
0.095 sec <<< ERROR!org.apache.hadoop.io.nativeio.NativeIOException: The
filename, directory name, or volume label syntax is incorrect.
at
org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Native
Method)
at
org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileOutputStreamWithMode(NativeIO.java:556)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:231)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:221)
at
org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:319)
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:308)
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:339)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.(ChecksumFileSystem.java:399)
at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:462)
at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:441)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:928)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:806)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:795)
at
org.apache.hadoop.fs.TestFsShellList.createFile(TestFsShellList.java:57)
at
org.apache.hadoop.fs.TestFsShellList.testList(TestFsShellList.java:69)
{code}
> TestFsShellList.testList fails on windows: illegal filenames
>
>
> Key: HADOOP-14199
> URL: https://issues.apache.org/jira/browse/HADOOP-14199
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Affects Versions: 2.8.0
> Environment: win64
>Reporter: Steve Loughran
>Priority: Minor
>
> {{TestFsShellList.testList}} fails setting up the files to test against
> {code}
> org.apache.hadoop.io.nativeio.NativeIOException: The filename, directory
> name, or volume label syntax is incorrect.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14199) TestFsShellList.testList fails on windows: illegal filenames
Steve Loughran created HADOOP-14199:
---
Summary: TestFsShellList.testList fails on windows: illegal
filenames
Key: HADOOP-14199
URL: https://issues.apache.org/jira/browse/HADOOP-14199
Project: Hadoop Common
Issue Type: Bug
Components: test
Affects Versions: 2.8.0
Environment: win64
Reporter: Steve Loughran
Priority: Minor
{{TestFsShellList.testList}} fails setting up the files to test against
{code}
org.apache.hadoop.io.nativeio.NativeIOException: The filename, directory name,
or volume label syntax is incorrect.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13726) Enforce that FileSystem initializes only a single instance of the requested FileSystem.
[
https://issues.apache.org/jira/browse/HADOOP-13726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931610#comment-15931610
]
Manjunath Anand edited comment on HADOOP-13726 at 3/19/17 2:40 PM:
---
I looked at the code which is called by both
[LoadingCache#get|http://google.github.io/guava/releases/snapshot/api/docs/com/google/common/cache/LoadingCache.html#get-K-]
and
[Cache#get(key,Callable)|http://google.github.io/guava/releases/snapshot/api/docs/com/google/common/cache/Cache.html#get-K-java.util.concurrent.Callable-]
and they both call the same code so we should be good with the locking
granularity as mentioned by [~cnauroth] in his comments.
As per my understanding I dont see any issue with referring to state (uri and
conf) ,in the code in above comment, other than what is in the key unlike what
is mentioned in the
[Cache#get(key,Callable)|http://google.github.io/guava/releases/snapshot/api/docs/com/google/common/cache/Cache.html#get-K-java.util.concurrent.Callable-]
doc which says:- {quote} Or use LoadingCache.get(K), which lacks the ability
to refer to state other than that in the key. {quote}
The only concern I saw during testing of the code in the above comment versus
the {{computeIfAbsent}} code approach presented in earlier comments is that if
there are multiple threads trying to initialize concurrently same FileSystem
and if the thread which succeeded in getting the lock throws an exception
during FileSystem initialization, then all other threads waiting for the result
will get ExecutionException and wouldnot retry serially unlike the
{{computeIfAbsent}} wherein upon one thread throwing exception during
initialization, the other concurrent thread which was waiting would proceed
with the initialization which effectively means a cool auto retry feature with
{{computeIfAbsent}} approach during concurrent threads for same FileSystem
was (Author: manju_hadoop):
I looked at the code which is called by both
[LoadingCache#get|http://google.github.io/guava/releases/snapshot/api/docs/com/google/common/cache/LoadingCache.html#get-K-]
and
[Cache#get(key,Callable)|http://google.github.io/guava/releases/snapshot/api/docs/com/google/common/cache/Cache.html#get-K-java.util.concurrent.Callable-]
and they both call the same code so we should be good with the locking
granularity as mentioned by [~cnauroth] in his comments and also as per my
understanding I dont see any issue with referring to state (uri and conf) other
than what is in the key unlike what is mentioned in the
[Cache#get(key,Callable)|http://google.github.io/guava/releases/snapshot/api/docs/com/google/common/cache/Cache.html#get-K-java.util.concurrent.Callable-]
doc which says:- {quote} Or use LoadingCache.get(K), which lacks the ability
to refer to state other than that in the key. {quote}
The only concern I saw during testing of the code in the above comment versus
the {{computeIfAbsent}} code approach presented in earlier comments is that if
there are multiple threads trying to initialize concurrently same FileSystem
and if the thread which succeeded in getting the lock throws an exception
during FileSystem initialization, then all other threads waiting for the result
will get ExecutionException and wouldnot retry serially unlike the
{{computeIfAbsent}} wherein upon one thread throwing exception during
initialization, the other concurrent thread which was waiting would proceed
with the initialization which effectively means a cool auto retry feature with
{{computeIfAbsent}} approach during concurrent threads for same FileSystem
> Enforce that FileSystem initializes only a single instance of the requested
> FileSystem.
> ---
>
> Key: HADOOP-13726
> URL: https://issues.apache.org/jira/browse/HADOOP-13726
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Reporter: Chris Nauroth
>Assignee: Manjunath Anand
>
> The {{FileSystem}} cache is intended to guarantee reuse of instances by
> multiple call sites or multiple threads. The current implementation does
> provide this guarantee, but there is a brief race condition window during
> which multiple threads could perform redundant initialization. If the file
> system implementation has expensive initialization logic, then this is
> wasteful. This issue proposes to eliminate that race condition and guarantee
> initialization of only a single instance.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14135) Remove URI parameter in AWSCredentialProvider constructors
[
https://issues.apache.org/jira/browse/HADOOP-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14135:
Parent Issue: HADOOP-13204 (was: HADOOP-11694)
> Remove URI parameter in AWSCredentialProvider constructors
> --
>
> Key: HADOOP-14135
> URL: https://issues.apache.org/jira/browse/HADOOP-14135
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-14135.000.patch, HADOOP-14135.001.patch,
> HADOOP-14135.002.patch, HADOOP-14135.003.patch
>
>
> This was from comment in [HADOOP-13252].
> It looks like the URI parameter is not needed for our AWSCredentialProvider
> constructors. This was useful because we relied on URI parameter for
> retrieving user:pass. Now in binding URIs, we have
> {code}
> 279 S3xLoginHelper.Login creds = getAWSAccessKeys(binding, conf);
> 280 credentials.add(new BasicAWSCredentialsProvider(
> 281 creds.getUser(), creds.getPassword()));
> {code}
> This way, we only need configuration object (if necessary) for all
> AWSCredentialProvider implementations. The benefit is that, if we create
> AWSCredentialProvider list for DynamoDB, we don't have to pass down the
> associated file system URI. This might be useful to S3Guard tools.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13715) Add isErasureCoded() API to FileStatus class
[
https://issues.apache.org/jira/browse/HADOOP-13715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HADOOP-13715:
Status: Patch Available (was: Open)
> Add isErasureCoded() API to FileStatus class
>
>
> Key: HADOOP-13715
> URL: https://issues.apache.org/jira/browse/HADOOP-13715
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-13715.01.patch
>
>
> Per the discussion in
> [HDFS-10971|https://issues.apache.org/jira/browse/HDFS-10971?focusedCommentId=15567108=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15567108]
> I would like to add a new API {{isErasureCoded()}} to {{FileStatus}} so that
> tools and downstream applications can tell if it needs to treat a file
> differently.
> Hadoop tools that can benefit from this effort include: distcp and
> teragen/terasort.
> Downstream applications such as flume or hbase may also benefit from it.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13715) Add isErasureCoded() API to FileStatus class
[
https://issues.apache.org/jira/browse/HADOOP-13715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HADOOP-13715:
Attachment: HADOOP-13715.01.patch
[~drankye], [~rakeshr],
bq. I could see encryption feature has exposed APIs like, HdfsFileStatus has
#getFileEncryptionInfo() exposed and FileStatus contains #isEncrypted() flag,
to tell whether the underlying file or directory is encrypted or not. imho, EC
could expose APIs in similar lines.
Yes, I am extending {{FsPermissionExtension}} to include Erasure Coded status
just like in Encryption
[~andrew.wang],
bq. I just reviewed HDFS-6984 which redoes FileStatus to serialize via
protobuf, which will add a bitfield to FileStatus for flags like this. If we
also add a bitfield to HdfsFileStatus, this JIRA should become easier.
I hope HDFS-6984 will take care of folding in the newly added {{erasureBit}} to
HdfsFileStatus. If this patch
Attaching v01 patch to make FileStatus expose isErasureCoded() via a new bit
in FsPermissionExtension. To be consistent with encryption and acl bit
approach, this patch doesn't intend to change any on-disk formats.
[~andrew.wang], can you please take a look at the patch ?
> Add isErasureCoded() API to FileStatus class
>
>
> Key: HADOOP-13715
> URL: https://issues.apache.org/jira/browse/HADOOP-13715
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-13715.01.patch
>
>
> Per the discussion in
> [HDFS-10971|https://issues.apache.org/jira/browse/HDFS-10971?focusedCommentId=15567108=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15567108]
> I would like to add a new API {{isErasureCoded()}} to {{FileStatus}} so that
> tools and downstream applications can tell if it needs to treat a file
> differently.
> Hadoop tools that can benefit from this effort include: distcp and
> teragen/terasort.
> Downstream applications such as flume or hbase may also benefit from it.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13453) S3Guard: Instrument new functionality with Hadoop metrics.
[
https://issues.apache.org/jira/browse/HADOOP-13453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931613#comment-15931613
]
Hadoop QA commented on HADOOP-13453:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
34s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
27s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 12s{color} | {color:orange} hadoop-tools/hadoop-aws: The patch generated 19
new + 25 unchanged - 0 fixed = 44 total (was 25) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
36s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13453 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12859449/HADOOP-13453-HADOOP-13345-003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux bd705d5c15d7 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HADOOP-13345 / b54e1b2 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11851/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11851/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11851/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> S3Guard: Instrument new functionality with Hadoop metrics.
> --
>
> Key: HADOOP-13453
> URL:
[jira] [Comment Edited] (HADOOP-13726) Enforce that FileSystem initializes only a single instance of the requested FileSystem.
[
https://issues.apache.org/jira/browse/HADOOP-13726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931610#comment-15931610
]
Manjunath Anand edited comment on HADOOP-13726 at 3/19/17 8:14 AM:
---
I looked at the code which is called by both
[LoadingCache#get|http://google.github.io/guava/releases/snapshot/api/docs/com/google/common/cache/LoadingCache.html#get-K-]
and
[Cache#get(key,Callable)|http://google.github.io/guava/releases/snapshot/api/docs/com/google/common/cache/Cache.html#get-K-java.util.concurrent.Callable-]
and they both call the same code so we should be good with the locking
granularity as mentioned by [~cnauroth] in his comments and also as per my
understanding I dont see any issue with referring to state (uri and conf) other
than what is in the key unlike what is mentioned in the
[Cache#get(key,Callable)|http://google.github.io/guava/releases/snapshot/api/docs/com/google/common/cache/Cache.html#get-K-java.util.concurrent.Callable-]
doc which says:- {quote} Or use LoadingCache.get(K), which lacks the ability
to refer to state other than that in the key. {quote}
The only concern I saw during testing of the code in the above comment versus
the {{computeIfAbsent}} code approach presented in earlier comments is that if
there are multiple threads trying to initialize concurrently same FileSystem
and if the thread which succeeded in getting the lock throws an exception
during FileSystem initialization, then all other threads waiting for the result
will get ExecutionException and wouldnot retry serially unlike the
{{computeIfAbsent}} wherein upon one thread throwing exception during
initialization, the other concurrent thread which was waiting would proceed
with the initialization which effectively means a cool auto retry feature with
{{computeIfAbsent}} approach during concurrent threads for same FileSystem
was (Author: manju_hadoop):
I looked at the code which is called by both
[LoadingCache#get|http://google.github.io/guava/releases/snapshot/api/docs/com/google/common/cache/LoadingCache.html#get-K-]
and
[Cache#get(key,Callable)|http://google.github.io/guava/releases/snapshot/api/docs/com/google/common/cache/Cache.html#get-K-java.util.concurrent.Callable-]
and they both call the same code so we should be good with the locking
granularity as mentioned by [~cnauroth] in his comments and also as per my
understanding I dont see any issue with referring to state (uri and conf) other
than what is in the key unlike what is mentioned in the
[Cache#get(key,Callable)|http://google.github.io/guava/releases/snapshot/api/docs/com/google/common/cache/Cache.html#get-K-java.util.concurrent.Callable-]
doc which says:- {quote} Or use LoadingCache.get(K), which lacks the ability
to refer to state other than that in the key. {quote}
The only concern I saw during testing of the code in the above comment versus
the {{computeIfAbsent}} code approach presented in earlier comments is that if
there are multiple threads trying to initialize concurrently same FileSystem
and if the thread which succeeded in getting the lock throws an exception then
all other threads waiting for the result will get ExecutionException and
wouldnot retry serially unlike the {{computeIfAbsent}} wherein upon one thread
throwing exception during initialization, the other concurrent thread which was
waiting would proceed with the initialization which effectively means a cool
auto retry feature with {{computeIfAbsent}} approach during concurrent threads
for same FileSystem
> Enforce that FileSystem initializes only a single instance of the requested
> FileSystem.
> ---
>
> Key: HADOOP-13726
> URL: https://issues.apache.org/jira/browse/HADOOP-13726
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Reporter: Chris Nauroth
>Assignee: Manjunath Anand
>
> The {{FileSystem}} cache is intended to guarantee reuse of instances by
> multiple call sites or multiple threads. The current implementation does
> provide this guarantee, but there is a brief race condition window during
> which multiple threads could perform redundant initialization. If the file
> system implementation has expensive initialization logic, then this is
> wasteful. This issue proposes to eliminate that race condition and guarantee
> initialization of only a single instance.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13726) Enforce that FileSystem initializes only a single instance of the requested FileSystem.
[
https://issues.apache.org/jira/browse/HADOOP-13726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931610#comment-15931610
]
Manjunath Anand edited comment on HADOOP-13726 at 3/19/17 8:12 AM:
---
I looked at the code which is called by both
[LoadingCache#get|http://google.github.io/guava/releases/snapshot/api/docs/com/google/common/cache/LoadingCache.html#get-K-]
and
[Cache#get(key,Callable)|http://google.github.io/guava/releases/snapshot/api/docs/com/google/common/cache/Cache.html#get-K-java.util.concurrent.Callable-]
and they both call the same code so we should be good with the locking
granularity as mentioned by [~cnauroth] in his comments and also as per my
understanding I dont see any issue with referring to state (uri and conf) other
than what is in the key unlike what is mentioned in the
[Cache#get(key,Callable)|http://google.github.io/guava/releases/snapshot/api/docs/com/google/common/cache/Cache.html#get-K-java.util.concurrent.Callable-]
doc which says:- {quote} Or use LoadingCache.get(K), which lacks the ability
to refer to state other than that in the key. {quote}
The only concern I saw during testing of the code in the above comment versus
the {{computeIfAbsent}} code approach presented in earlier comments is that if
there are multiple threads trying to initialize concurrently same FileSystem
and if the thread which succeeded in getting the lock throws an exception then
all other threads waiting for the result will get ExecutionException and
wouldnot retry serially unlike the {{computeIfAbsent}} wherein upon one thread
throwing exception during initialization, the other concurrent thread which was
waiting would proceed with the initialization which effectively means a cool
auto retry feature with {{computeIfAbsent}} approach during concurrent threads
for same FileSystem
was (Author: manju_hadoop):
I looked at the code which is called by both
[LoadingCache#get|http://google.github.io/guava/releases/snapshot/api/docs/com/google/common/cache/LoadingCache.html#get-K-]
and
[Cache#get(key,Callable)|http://google.github.io/guava/releases/snapshot/api/docs/com/google/common/cache/Cache.html#get-K-java.util.concurrent.Callable-]
and they both call the same code so we should be good with the locking
granularity as mentioned by [~cnauroth] in his comments and also as per my
understanding I dont feel any issue with referring to state (uri and conf)
other than what is in the key unlike what is mentioned in the
[Cache#get(key,Callable)|http://google.github.io/guava/releases/snapshot/api/docs/com/google/common/cache/Cache.html#get-K-java.util.concurrent.Callable-]
doc {quote} Or use LoadingCache.get(K), which lacks the ability to refer to
state other than that in the key. {quote}
The only concern I saw during testing of the code in the above comment versus
the {{computeIfAbsent}} code approach presented in earlier comments is that if
there are multiple threads trying to initialize concurrently same FileSystem
and if the thread which succeeded in getting the lock throws an exception then
all other threads waiting for the result will get ExecutionException and
wouldnot retry serially unlike the {{computeIfAbsent}} wherein upon one thread
throwing exception during initialization, the other concurrent thread which was
waiting would proceed with the initialization which effectively means a cool
auto retry feature with {{computeIfAbsent}} approach during concurrent threads
for same FileSystem
> Enforce that FileSystem initializes only a single instance of the requested
> FileSystem.
> ---
>
> Key: HADOOP-13726
> URL: https://issues.apache.org/jira/browse/HADOOP-13726
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Reporter: Chris Nauroth
>Assignee: Manjunath Anand
>
> The {{FileSystem}} cache is intended to guarantee reuse of instances by
> multiple call sites or multiple threads. The current implementation does
> provide this guarantee, but there is a brief race condition window during
> which multiple threads could perform redundant initialization. If the file
> system implementation has expensive initialization logic, then this is
> wasteful. This issue proposes to eliminate that race condition and guarantee
> initialization of only a single instance.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13726) Enforce that FileSystem initializes only a single instance of the requested FileSystem.
[
https://issues.apache.org/jira/browse/HADOOP-13726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931610#comment-15931610
]
Manjunath Anand commented on HADOOP-13726:
--
I looked at the code which is called by both
[LoadingCache#get|http://google.github.io/guava/releases/snapshot/api/docs/com/google/common/cache/LoadingCache.html#get-K-]
and
[Cache#get(key,Callable)|http://google.github.io/guava/releases/snapshot/api/docs/com/google/common/cache/Cache.html#get-K-java.util.concurrent.Callable-]
and they both call the same code so we should be good with the locking
granularity as mentioned by [~cnauroth] in his comments and also as per my
understanding I dont feel any issue with referring to state (uri and conf)
other than what is in the key unlike what is mentioned in the
[Cache#get(key,Callable)|http://google.github.io/guava/releases/snapshot/api/docs/com/google/common/cache/Cache.html#get-K-java.util.concurrent.Callable-]
doc {quote} Or use LoadingCache.get(K), which lacks the ability to refer to
state other than that in the key. {quote}
The only concern I saw during testing of the code in the above comment versus
the {{computeIfAbsent}} code approach presented in earlier comments is that if
there are multiple threads trying to initialize concurrently same FileSystem
and if the thread which succeeded in getting the lock throws an exception then
all other threads waiting for the result will get ExecutionException and
wouldnot retry serially unlike the {{computeIfAbsent}} wherein upon one thread
throwing exception during initialization, the other concurrent thread which was
waiting would proceed with the initialization which effectively means a cool
auto retry feature with {{computeIfAbsent}} approach during concurrent threads
for same FileSystem
> Enforce that FileSystem initializes only a single instance of the requested
> FileSystem.
> ---
>
> Key: HADOOP-13726
> URL: https://issues.apache.org/jira/browse/HADOOP-13726
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Reporter: Chris Nauroth
>Assignee: Manjunath Anand
>
> The {{FileSystem}} cache is intended to guarantee reuse of instances by
> multiple call sites or multiple threads. The current implementation does
> provide this guarantee, but there is a brief race condition window during
> which multiple threads could perform redundant initialization. If the file
> system implementation has expensive initialization logic, then this is
> wasteful. This issue proposes to eliminate that race condition and guarantee
> initialization of only a single instance.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13453) S3Guard: Instrument new functionality with Hadoop metrics.
[ https://issues.apache.org/jira/browse/HADOOP-13453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931611#comment-15931611 ] Ai Deng commented on HADOOP-13453: -- I think maybe measure the number of path has been operated (put, get … ) in MetaStore could be interesting. The end user can see how big their S3 file system has been managed in S3Guard. > S3Guard: Instrument new functionality with Hadoop metrics. > -- > > Key: HADOOP-13453 > URL: https://issues.apache.org/jira/browse/HADOOP-13453 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Chris Nauroth >Assignee: Ai Deng > Attachments: HADOOP-13453-HADOOP-13345-001.patch, > HADOOP-13453-HADOOP-13345-002.patch, HADOOP-13453-HADOOP-13345-003.patch > > > Provide Hadoop metrics showing operational details of the S3Guard > implementation. > The metrics will be implemented in this ticket: > ● S3GuardRechecksNthPercentileLatency (MutableQuantiles) Percentile time > spent > in rechecks attempting to achieve consistency. Repeated for multiple > percentile values > of N. This metric is an indicator of the additional latency cost of running > S3A with > S3Guard. > ● S3GuardRechecksNumOps (MutableQuantiles) Number of times a consistency > recheck was required while attempting to achieve consistency. > ● S3GuardStoreNthPercentileLatency (MutableQuantiles) Percentile time > spent in > operations against the consistent store, including both write operations > during file system > mutations and read operations during file system consistency checks. Repeated > for > multiple percentile values of N. This metric is an indicator of latency to > the consistent > store implementation. > ● S3GuardConsistencyStoreNumOps (MutableQuantiles) Number of operations > against the consistent store, including both write operations during file > system mutations > and read operations during file system consistency checks. > ● S3GuardConsistencyStoreFailures (MutableCounterLong) Number of failures > during operations against the consistent store implementation. > ● S3GuardConsistencyStoreTimeouts (MutableCounterLong) Number of timeouts > during operations against the consistent store implementation. > ● S3GuardInconsistencies (MutableCounterLong) C ount of times S3Guard > failed to > achieve consistency, even after exhausting all rechecks. A high count may > indicate > unexpected outofband modification of the S3 bucket contents, such as by an > external > tool that does not make corresponding updates to the consistent store. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13453) S3Guard: Instrument new functionality with Hadoop metrics.
[ https://issues.apache.org/jira/browse/HADOOP-13453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931609#comment-15931609 ] Ai Deng commented on HADOOP-13453: -- Hi [~ste...@apache.org], I have added a new patch following your suggestion. If it is ok, we can discuss the metrics we want to add? I come out this list of operation and latency metrics for this ticket, can you check if I miss anything? Thank you. Status: S3GUARD_METADATASTORE_ENABLED S3GUARD_METADATASTORE_IS_AUTHORITATIVE Operations: S3GUARD_METADATASTORE_INITIALIZATION S3GUARD_METADATASTORE_DELETE_PATH S3GUARD_METADATASTORE_DELETE_PATH_LATENCY S3GUARD_METADATASTORE_DELETE_SUBTREE_PATCH S3GUARD_METADATASTORE_GET_PATH S3GUARD_METADATASTORE_GET_PATH_LATENCY S3GUARD_METADATASTORE_GET_CHILDREN_PATH S3GUARD_METADATASTORE_GET_CHILDREN_PATH_LATENCY S3GUARD_METADATASTORE_MOVE_PATH S3GUARD_METADATASTORE_PUT_PATH S3GUARD_METADATASTORE_PUT_PATH_LATENCY S3GUARD_METADATASTORE_CLOSE S3GUARD_METADATASTORE_DESTORY >From S3Guard: S3GUARD_METADATASTORE_MERGE_DIRECTORY For the failures: S3GUARD_METADATASTORE_DELETE_FAILURE S3GUARD_METADATASTORE_GET_FAILURE S3GUARD_METADATASTORE_PUT_FAILURE Etc: S3GUARD_METADATASTORE_PUT_RETRY_TIMES > S3Guard: Instrument new functionality with Hadoop metrics. > -- > > Key: HADOOP-13453 > URL: https://issues.apache.org/jira/browse/HADOOP-13453 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Chris Nauroth >Assignee: Ai Deng > Attachments: HADOOP-13453-HADOOP-13345-001.patch, > HADOOP-13453-HADOOP-13345-002.patch, HADOOP-13453-HADOOP-13345-003.patch > > > Provide Hadoop metrics showing operational details of the S3Guard > implementation. > The metrics will be implemented in this ticket: > ● S3GuardRechecksNthPercentileLatency (MutableQuantiles) Percentile time > spent > in rechecks attempting to achieve consistency. Repeated for multiple > percentile values > of N. This metric is an indicator of the additional latency cost of running > S3A with > S3Guard. > ● S3GuardRechecksNumOps (MutableQuantiles) Number of times a consistency > recheck was required while attempting to achieve consistency. > ● S3GuardStoreNthPercentileLatency (MutableQuantiles) Percentile time > spent in > operations against the consistent store, including both write operations > during file system > mutations and read operations during file system consistency checks. Repeated > for > multiple percentile values of N. This metric is an indicator of latency to > the consistent > store implementation. > ● S3GuardConsistencyStoreNumOps (MutableQuantiles) Number of operations > against the consistent store, including both write operations during file > system mutations > and read operations during file system consistency checks. > ● S3GuardConsistencyStoreFailures (MutableCounterLong) Number of failures > during operations against the consistent store implementation. > ● S3GuardConsistencyStoreTimeouts (MutableCounterLong) Number of timeouts > during operations against the consistent store implementation. > ● S3GuardInconsistencies (MutableCounterLong) C ount of times S3Guard > failed to > achieve consistency, even after exhausting all rechecks. A high count may > indicate > unexpected outofband modification of the S3 bucket contents, such as by an > external > tool that does not make corresponding updates to the consistent store. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13453) S3Guard: Instrument new functionality with Hadoop metrics.
[ https://issues.apache.org/jira/browse/HADOOP-13453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ai Deng updated HADOOP-13453: - Attachment: HADOOP-13453-HADOOP-13345-003.patch > S3Guard: Instrument new functionality with Hadoop metrics. > -- > > Key: HADOOP-13453 > URL: https://issues.apache.org/jira/browse/HADOOP-13453 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Chris Nauroth >Assignee: Ai Deng > Attachments: HADOOP-13453-HADOOP-13345-001.patch, > HADOOP-13453-HADOOP-13345-002.patch, HADOOP-13453-HADOOP-13345-003.patch > > > Provide Hadoop metrics showing operational details of the S3Guard > implementation. > The metrics will be implemented in this ticket: > ● S3GuardRechecksNthPercentileLatency (MutableQuantiles) Percentile time > spent > in rechecks attempting to achieve consistency. Repeated for multiple > percentile values > of N. This metric is an indicator of the additional latency cost of running > S3A with > S3Guard. > ● S3GuardRechecksNumOps (MutableQuantiles) Number of times a consistency > recheck was required while attempting to achieve consistency. > ● S3GuardStoreNthPercentileLatency (MutableQuantiles) Percentile time > spent in > operations against the consistent store, including both write operations > during file system > mutations and read operations during file system consistency checks. Repeated > for > multiple percentile values of N. This metric is an indicator of latency to > the consistent > store implementation. > ● S3GuardConsistencyStoreNumOps (MutableQuantiles) Number of operations > against the consistent store, including both write operations during file > system mutations > and read operations during file system consistency checks. > ● S3GuardConsistencyStoreFailures (MutableCounterLong) Number of failures > during operations against the consistent store implementation. > ● S3GuardConsistencyStoreTimeouts (MutableCounterLong) Number of timeouts > during operations against the consistent store implementation. > ● S3GuardInconsistencies (MutableCounterLong) C ount of times S3Guard > failed to > achieve consistency, even after exhausting all rechecks. A high count may > indicate > unexpected outofband modification of the S3 bucket contents, such as by an > external > tool that does not make corresponding updates to the consistent store. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14198) Should have a way to let PingInputStream to abort
Yongjun Zhang created HADOOP-14198:
--
Summary: Should have a way to let PingInputStream to abort
Key: HADOOP-14198
URL: https://issues.apache.org/jira/browse/HADOOP-14198
Project: Hadoop Common
Issue Type: Bug
Reporter: Yongjun Zhang
We observed a case that RPC call get stuck, since PingInputStream does the
following
{code}
/** This class sends a ping to the remote side when timeout on
* reading. If no failure is detected, it retries until at least
* a byte is read.
*/
private class PingInputStream extends FilterInputStream {
{code}
It seems that in this case no data is ever received, and it keeps pinging.
Should we ping forever here? Maybe we should introduce a config to stop the
ping after pinging for certain number of times, and report back timeout, let
the caller to retry the RPC?
Wonder if there is chance the RPC get dropped somehow by the server so no
response is ever received.
See
{code}
Thread 16127: (state = BLOCKED)
- sun.nio.ch.SocketChannelImpl.readerCleanup() @bci=6, line=279 (Compiled
frame)
- sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer) @bci=205, line=390
(Compiled frame)
-
org.apache.hadoop.net.SocketInputStream$Reader.performIO(java.nio.ByteBuffer)
@bci=5, line=57 (Compiled frame)
- org.apache.hadoop.net.SocketIOWithTimeout.doIO(java.nio.ByteBuffer, int)
@bci=35, line=142 (Compiled frame)
- org.apache.hadoop.net.SocketInputStream.read(java.nio.ByteBuffer) @bci=6,
line=161 (Compiled frame)
- org.apache.hadoop.net.SocketInputStream.read(byte[], int, int) @bci=7,
line=131 (Compiled frame)
- java.io.FilterInputStream.read(byte[], int, int) @bci=7, line=133 (Compiled
frame)
- java.io.FilterInputStream.read(byte[], int, int) @bci=7, line=133 (Compiled
frame)
- org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int,
int) @bci=4, line=521 (Compiled frame)
- java.io.BufferedInputStream.fill() @bci=214, line=246 (Compiled frame)
- java.io.BufferedInputStream.read() @bci=12, line=265 (Compiled frame)
- java.io.DataInputStream.readInt() @bci=4, line=387 (Compiled frame)
- org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse() @bci=19,
line=1081 (Compiled frame)
- org.apache.hadoop.ipc.Client$Connection.run() @bci=62, line=976 (Compiled
frame)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
