[jira] [Commented] (HADOOP-13192) org.apache.hadoop.util.LineReader match recordDelimiter has a bug
[
https://issues.apache.org/jira/browse/HADOOP-13192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338998#comment-15338998
]
Mingliang Liu commented on HADOOP-13192:
Hi [~zhudebin], the failing test is a known bug, see [HADOOP-12588]. This is an
intermittent failing unit test (UT) so you may not reproduce it consistently on
your local machine. I had a look at the patch and think it is an unrelated
failure UT.
So when contributing patches and review the pre-commit Jenkins report, you can
generally check if the failing tests are related. If unrelated, you can search
JIRA ticket that reports it and comment about it here. You can file another bug
for tracking the failing UTs if no one has reported it yet.
Thanks for the contribution.
> org.apache.hadoop.util.LineReader match recordDelimiter has a bug
> --
>
> Key: HADOOP-13192
> URL: https://issues.apache.org/jira/browse/HADOOP-13192
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Affects Versions: 2.6.2
>Reporter: binde
>Assignee: binde
> Attachments:
> 0001-HADOOP-13192-org.apache.hadoop.util.LineReader-match.patch,
> 0002-fix-bug-hadoop-1392-add-test-case-for-LineReader.patch
>
> Original Estimate: 5m
> Remaining Estimate: 5m
>
> org.apache.hadoop.util.LineReader.readCustomLine() has a bug,
> when line is bccc, recordDelimiter is aaab, the result should be a,ccc,
> show the code on line 310:
> for (; bufferPosn < bufferLength; ++bufferPosn) {
> if (buffer[bufferPosn] == recordDelimiterBytes[delPosn]) {
> delPosn++;
> if (delPosn >= recordDelimiterBytes.length) {
> bufferPosn++;
> break;
> }
> } else if (delPosn != 0) {
> bufferPosn--;
> delPosn = 0;
> }
> }
> shoud be :
> for (; bufferPosn < bufferLength; ++bufferPosn) {
> if (buffer[bufferPosn] == recordDelimiterBytes[delPosn]) {
> delPosn++;
> if (delPosn >= recordDelimiterBytes.length) {
> bufferPosn++;
> break;
> }
> } else if (delPosn != 0) {
> // - change here - start
> bufferPosn -= delPosn;
> // - change here - end
>
> delPosn = 0;
> }
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12893) Verify LICENSE.txt and NOTICE.txt
[ https://issues.apache.org/jira/browse/HADOOP-12893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-12893: --- Attachment: HADOOP-12893-addendum-branch-2.7.01.patch Thanks [~busbey] and [~ebadger] for the comments. Attaching a patch to add hadoop-build-tools as a dependency of hadoop-project. Is it correct? > Verify LICENSE.txt and NOTICE.txt > - > > Key: HADOOP-12893 > URL: https://issues.apache.org/jira/browse/HADOOP-12893 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Xiao Chen >Priority: Blocker > Fix For: 2.7.3, 2.6.5 > > Attachments: HADOOP-12893-addendum-branch-2.7.01.patch, > HADOOP-12893.002.patch, HADOOP-12893.003.patch, HADOOP-12893.004.patch, > HADOOP-12893.005.patch, HADOOP-12893.006.patch, HADOOP-12893.007.patch, > HADOOP-12893.008.patch, HADOOP-12893.009.patch, HADOOP-12893.01.patch, > HADOOP-12893.011.patch, HADOOP-12893.012.patch, HADOOP-12893.10.patch, > HADOOP-12893.branch-2.01.patch, HADOOP-12893.branch-2.6.01.patch, > HADOOP-12893.branch-2.7.01.patch, HADOOP-12893.branch-2.7.02.patch, > HADOOP-12893.branch-2.7.3.01.patch > > > We have many bundled dependencies in both the source and the binary artifacts > that are not in LICENSE.txt and NOTICE.txt. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13291) Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented
[
https://issues.apache.org/jira/browse/HADOOP-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338986#comment-15338986
]
Mingliang Liu commented on HADOOP-13291:
Thank you very much for your review and running test against S3,
[~ste...@apache.org].
> Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be
> correctly implemented
> ---
>
> Key: HADOOP-13291
> URL: https://issues.apache.org/jira/browse/HADOOP-13291
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-13291.000.patch, HADOOP-13291.001.patch,
> HADOOP-13291.002.patch
>
>
> To probe a stat in {{StorageStatistics}}, users can use the
> {{StorageStatistics#isTracked()}} API. Currently {{DFSOpsCountStatistics}}
> implements this function wrongly. {{S3AStorageStatistics}} borrowed the same
> idea and also has the same error.
> # The {{isTracked()}} is not correctly implemented. I believe this was an
> omission in code.
> # {{isTracked()}} checks a stat with operation symbol (instead of enum name).
> {{getLongStatistics()}} should return LongStatistics iterators with symbol as
> name, instead of the enum variable name. Or else,
> {{isTracked(getLongStatistics().next().getName());}} returns false. This will
> lead to confusion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13192) org.apache.hadoop.util.LineReader match recordDelimiter has a bug
[
https://issues.apache.org/jira/browse/HADOOP-13192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338981#comment-15338981
]
binde commented on HADOOP-13192:
I run the test case has no problem, why?
> org.apache.hadoop.util.LineReader match recordDelimiter has a bug
> --
>
> Key: HADOOP-13192
> URL: https://issues.apache.org/jira/browse/HADOOP-13192
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Affects Versions: 2.6.2
>Reporter: binde
>Assignee: binde
> Attachments:
> 0001-HADOOP-13192-org.apache.hadoop.util.LineReader-match.patch,
> 0002-fix-bug-hadoop-1392-add-test-case-for-LineReader.patch
>
> Original Estimate: 5m
> Remaining Estimate: 5m
>
> org.apache.hadoop.util.LineReader.readCustomLine() has a bug,
> when line is bccc, recordDelimiter is aaab, the result should be a,ccc,
> show the code on line 310:
> for (; bufferPosn < bufferLength; ++bufferPosn) {
> if (buffer[bufferPosn] == recordDelimiterBytes[delPosn]) {
> delPosn++;
> if (delPosn >= recordDelimiterBytes.length) {
> bufferPosn++;
> break;
> }
> } else if (delPosn != 0) {
> bufferPosn--;
> delPosn = 0;
> }
> }
> shoud be :
> for (; bufferPosn < bufferLength; ++bufferPosn) {
> if (buffer[bufferPosn] == recordDelimiterBytes[delPosn]) {
> delPosn++;
> if (delPosn >= recordDelimiterBytes.length) {
> bufferPosn++;
> break;
> }
> } else if (delPosn != 0) {
> // - change here - start
> bufferPosn -= delPosn;
> // - change here - end
>
> delPosn = 0;
> }
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13287) TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains '+'.
[
https://issues.apache.org/jira/browse/HADOOP-13287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338809#comment-15338809
]
Hadoop QA commented on HADOOP-13287:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
13s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:e2f6409 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12811731/HADOOP-13287.002.patch
|
| JIRA Issue | HADOOP-13287 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux c50fe11da9b9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 0319d73 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9830/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9830/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains
> '+'.
> ---
>
> Key: HADOOP-13287
> URL: https://issues.apache.org/jira/browse/HADOOP-13287
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3, test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HADOOP-13287.001.patch, HADOOP-13287.002.patch
>
>
> HADOOP-3733 fixed accessing S3A with credentials on the command line for an
> AWS
[jira] [Updated] (HADOOP-13287) TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains '+'.
[
https://issues.apache.org/jira/browse/HADOOP-13287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-13287:
---
Attachment: HADOOP-13287.002.patch
Thanks for testing, Steve. Could you please try again with patch 002? That
assertion needed to be updated to reflect the fact that we're now
double-encoding the input, for symmetry with the double-decoding done in the
main code. I also added a similar assertion for '+'. I retested against
US-west-2 using my secret key with a '+'.
> TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains
> '+'.
> ---
>
> Key: HADOOP-13287
> URL: https://issues.apache.org/jira/browse/HADOOP-13287
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3, test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HADOOP-13287.001.patch, HADOOP-13287.002.patch
>
>
> HADOOP-3733 fixed accessing S3A with credentials on the command line for an
> AWS secret key containing a '/'. The patch added a new test suite:
> {{TestS3ACredentialsInURL}}. One of the tests fails if your AWS secret key
> contains a '+'.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13287) TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains '+'.
[
https://issues.apache.org/jira/browse/HADOOP-13287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338802#comment-15338802
]
Chris Nauroth edited comment on HADOOP-13287 at 6/19/16 9:48 PM:
-
Thanks for testing, Steve. Could you please try again with patch 002? That
assertion needed to be updated to reflect the fact that we're now
double-encoding the input, for symmetry with the double-decoding done in the
main code. I also added a similar assertion for '\+'. I retested against
US-west-2 using my secret key with a '\+'.
was (Author: cnauroth):
Thanks for testing, Steve. Could you please try again with patch 002? That
assertion needed to be updated to reflect the fact that we're now
double-encoding the input, for symmetry with the double-decoding done in the
main code. I also added a similar assertion for '+'. I retested against
US-west-2 using my secret key with a '+'.
> TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains
> '+'.
> ---
>
> Key: HADOOP-13287
> URL: https://issues.apache.org/jira/browse/HADOOP-13287
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3, test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HADOOP-13287.001.patch, HADOOP-13287.002.patch
>
>
> HADOOP-3733 fixed accessing S3A with credentials on the command line for an
> AWS secret key containing a '/'. The patch added a new test suite:
> {{TestS3ACredentialsInURL}}. One of the tests fails if your AWS secret key
> contains a '+'.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10051) winutil.exe is not included in hadoop bin tarball
[ https://issues.apache.org/jira/browse/HADOOP-10051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338791#comment-15338791 ] Romain Manni-Bucau commented on HADOOP-10051: - +1 the github issue being more github than the signing there. What is blocking to put the binaries in the repo to allow a mvn packaging? We do it for tomee and while it is limited to few files it is acceptable and easy enough I think. > winutil.exe is not included in hadoop bin tarball > - > > Key: HADOOP-10051 > URL: https://issues.apache.org/jira/browse/HADOOP-10051 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Affects Versions: 2.2.0, 2.4.0, 2.5.0 >Reporter: Tsuyoshi Ozawa > > I don't have Windows environment, but one user who tried 2.2.0 release > on Windows reported that released tar ball doesn't contain > "winutil.exe" and cannot run any commands. I confirmed that winutil.exe is > not included in 2.2.0 bin tarball surely. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12804) Read Proxy Password from Credential Providers in S3 FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-12804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338788#comment-15338788 ] Steve Loughran commented on HADOOP-12804: - as we're consistently requesting for all s3 patches: which infrastructure did you run the full TestS3A* test suites against? > Read Proxy Password from Credential Providers in S3 FileSystem > -- > > Key: HADOOP-12804 > URL: https://issues.apache.org/jira/browse/HADOOP-12804 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Larry McCay >Assignee: Larry McCay >Priority: Minor > Attachments: HADOOP-12804-001.patch, HADOOP-12804-003.patch, > HADOOP-12804-branch-2-002.patch, HADOOP-12804-branch-2-003.patch > > > HADOOP-12548 added credential provider support for the AWS credentials to > S3FileSystem. This JIRA is for considering the use of the credential > providers for the proxy password as well. > Instead of adding the proxy password to the config file directly and in clear > text, we could provision it in addition to the AWS credentials into a > credential provider and keep it out of clear text. > In terms of usage, it could be added to the same credential store as the AWS > credentials or potentially to a more universally available path - since it is > the same for everyone. This would however require multiple providers to be > configured in the provider.path property and more open file permissions on > the store itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10051) winutil.exe is not included in hadoop bin tarball
[ https://issues.apache.org/jira/browse/HADOOP-10051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338787#comment-15338787 ] Steve Loughran commented on HADOOP-10051: - I did sign the JARs, with the same gpg that's listed as my hadoop committer credentials; it was built off the ASF commit ID, and on a dedicated VM that I use for build and test of Hadoop stuff. You can trust it as much as you can any other binary you come from me, and I'm sure your build already passes through code I've done. The main issues with github is durability; how long can you trust it to be there. What we are discussing is getting rid of winutils entirely, move to having a JAR containing the native libs inside, libs which are then unzipped depending on the platform...the way snappy does. That way: a JAR in the package or up on maven. Volunteers to help implement/test that welcome. > winutil.exe is not included in hadoop bin tarball > - > > Key: HADOOP-10051 > URL: https://issues.apache.org/jira/browse/HADOOP-10051 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Affects Versions: 2.2.0, 2.4.0, 2.5.0 >Reporter: Tsuyoshi Ozawa > > I don't have Windows environment, but one user who tried 2.2.0 release > on Windows reported that released tar ball doesn't contain > "winutil.exe" and cannot run any commands. I confirmed that winutil.exe is > not included in 2.2.0 bin tarball surely. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12949) Add HTrace to the s3a connector
[ https://issues.apache.org/jira/browse/HADOOP-12949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338785#comment-15338785 ] Steve Loughran commented on HADOOP-12949: - + we'll want to have the htrace context ID go all the way down to s3 by way of the HADOOP-13122 UA header. That lets your storage infra provider know which queries are causing problems, and, if this goes via a proxy capable of reading the HTTP Requests, lets them sample and correlate with network load > Add HTrace to the s3a connector > --- > > Key: HADOOP-12949 > URL: https://issues.apache.org/jira/browse/HADOOP-12949 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Reporter: Madhawa Gunasekara >Assignee: Madhawa Gunasekara > > Hi All, > s3, GCS, WASB, and other cloud blob stores are becoming increasingly > important in Hadoop. But we don't have distributed tracing for these yet. It > would be interesting to add distributed tracing here. It would enable > collecting really interesting data like probability distributions of PUT and > GET requests to s3 and their impact on MR jobs, etc. > I would like to implement this feature, Please shed some light on this > Thanks, > Madhawa -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10051) winutil.exe is not included in hadoop bin tarball
[ https://issues.apache.org/jira/browse/HADOOP-10051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338673#comment-15338673 ] Romain Manni-Bucau commented on HADOOP-10051: - [~ste...@apache.org] that's what I'm doing (patching beam to build on windows ATM) but would be saner and better to rely on an ASF or worse case central (like maven one) binary and not a github one > winutil.exe is not included in hadoop bin tarball > - > > Key: HADOOP-10051 > URL: https://issues.apache.org/jira/browse/HADOOP-10051 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Affects Versions: 2.2.0, 2.4.0, 2.5.0 >Reporter: Tsuyoshi Ozawa > > I don't have Windows environment, but one user who tried 2.2.0 release > on Windows reported that released tar ball doesn't contain > "winutil.exe" and cannot run any commands. I confirmed that winutil.exe is > not included in 2.2.0 bin tarball surely. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13283) Support reset operation for new global storage statistics and per FS storage stats
[
https://issues.apache.org/jira/browse/HADOOP-13283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338666#comment-15338666
]
Steve Loughran edited comment on HADOOP-13283 at 6/19/16 6:19 PM:
--
for the tests i was doing, i was just thinking of having a "snapshot" operation
which took the enum, built a list of the "before" Values, could do the same
with the next set, and so let me compare them. That's where I'm doing explicit
diffs of state (see {{S3ATestUtils.MetricDiff}} for the feature on a single
metric, with integrated assertions)...I think production code might prefer the
reset, just to avoid having to keep the original metric snapshot around
was (Author: ste...@apache.org):
for the tests i was doing, i was just thinking of having a "snapshot" operation
which took the enum, built a list of the "before" Values, could do the same
with the next set, and so let me compare them. That's where I'm doing explicit
diffs of state (see {{S3ATestUtils.MetricDiff for the feature on a single
metric, with integrated assertions}}...I think production code might prefer the
reset, just to avoid having to keep the original metric snapshot around
> Support reset operation for new global storage statistics and per FS storage
> stats
> --
>
> Key: HADOOP-13283
> URL: https://issues.apache.org/jira/browse/HADOOP-13283
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
>
> Applications may reuse the file system object across jobs and its storage
> statistics should be reset. Specially the {{FileSystem.Statistics}} supports
> reset and [HADOOP-13032] needs to keep that use case valid.
> This jira is for supporting reset operations for storage statistics.
> Thanks [~hitesh] for reporting this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13283) Support reset operation for new global storage statistics and per FS storage stats
[
https://issues.apache.org/jira/browse/HADOOP-13283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338666#comment-15338666
]
Steve Loughran commented on HADOOP-13283:
-
for the tests i was doing, i was just thinking of having a "snapshot" operation
which took the enum, built a list of the "before" Values, could do the same
with the next set, and so let me compare them. That's where I'm doing explicit
diffs of state (see {{S3ATestUtils.MetricDiff for the feature on a single
metric, with integrated assertions}}...I think production code might prefer the
reset, just to avoid having to keep the original metric snapshot around
> Support reset operation for new global storage statistics and per FS storage
> stats
> --
>
> Key: HADOOP-13283
> URL: https://issues.apache.org/jira/browse/HADOOP-13283
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
>
> Applications may reuse the file system object across jobs and its storage
> statistics should be reset. Specially the {{FileSystem.Statistics}} supports
> reset and [HADOOP-13032] needs to keep that use case valid.
> This jira is for supporting reset operations for storage statistics.
> Thanks [~hitesh] for reporting this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10051) winutil.exe is not included in hadoop bin tarball
[ https://issues.apache.org/jira/browse/HADOOP-10051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338665#comment-15338665 ] Steve Loughran commented on HADOOP-10051: - you can pick up a copy here https://github.com/steveloughran/winutils see http://wiki.apache.org/hadoop/WindowsProblems > winutil.exe is not included in hadoop bin tarball > - > > Key: HADOOP-10051 > URL: https://issues.apache.org/jira/browse/HADOOP-10051 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Affects Versions: 2.2.0, 2.4.0, 2.5.0 >Reporter: Tsuyoshi Ozawa > > I don't have Windows environment, but one user who tried 2.2.0 release > on Windows reported that released tar ball doesn't contain > "winutil.exe" and cannot run any commands. I confirmed that winutil.exe is > not included in 2.2.0 bin tarball surely. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10051) winutil.exe is not included in hadoop bin tarball
[ https://issues.apache.org/jira/browse/HADOOP-10051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338655#comment-15338655 ] Romain Manni-Bucau commented on HADOOP-10051: - still in 2.7 > winutil.exe is not included in hadoop bin tarball > - > > Key: HADOOP-10051 > URL: https://issues.apache.org/jira/browse/HADOOP-10051 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Affects Versions: 2.2.0, 2.4.0, 2.5.0 >Reporter: Tsuyoshi Ozawa > > I don't have Windows environment, but one user who tried 2.2.0 release > on Windows reported that released tar ball doesn't contain > "winutil.exe" and cannot run any commands. I confirmed that winutil.exe is > not included in 2.2.0 bin tarball surely. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13286) add a scale test to do gunzip and linecount
[ https://issues.apache.org/jira/browse/HADOOP-13286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338542#comment-15338542 ] Steve Loughran commented on HADOOP-13286: - My key goal here was to force through something which absolutely models a real use; this is essentially the same test which downstream tests demonstrated problems with the HADOOP-13203 test. A key thing is that I don't know how the gzip codec reads data (blocks? stream?) —and how it will continue to read data in future II'd like to keep it for that reason, pulling something else instead if you think it duplicates. > add a scale test to do gunzip and linecount > --- > > Key: HADOOP-13286 > URL: https://issues.apache.org/jira/browse/HADOOP-13286 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13286-branch-2-001.patch > > > the HADOOP-13203 patch proposal showed that there were performance problems > downstream which weren't surfacing in the current scale tests. > Trying to decompress the .gz test file and then go through it with LineReader > models a basic use case: parse a .csv.gz data source. > Add this, with metric printing -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13291) Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented
[
https://issues.apache.org/jira/browse/HADOOP-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338540#comment-15338540
]
Steve Loughran commented on HADOOP-13291:
-
+1
tested against s3 ireland; all well
> Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be
> correctly implemented
> ---
>
> Key: HADOOP-13291
> URL: https://issues.apache.org/jira/browse/HADOOP-13291
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-13291.000.patch, HADOOP-13291.001.patch,
> HADOOP-13291.002.patch
>
>
> To probe a stat in {{StorageStatistics}}, users can use the
> {{StorageStatistics#isTracked()}} API. Currently {{DFSOpsCountStatistics}}
> implements this function wrongly. {{S3AStorageStatistics}} borrowed the same
> idea and also has the same error.
> # The {{isTracked()}} is not correctly implemented. I believe this was an
> omission in code.
> # {{isTracked()}} checks a stat with operation symbol (instead of enum name).
> {{getLongStatistics()}} should return LongStatistics iterators with symbol as
> name, instead of the enum variable name. Or else,
> {{isTracked(getLongStatistics().next().getName());}} returns false. This will
> lead to confusion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13287) TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains '+'.
[
https://issues.apache.org/jira/browse/HADOOP-13287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338534#comment-15338534
]
Steve Loughran commented on HADOOP-13287:
-
-1. I'm afrad
I'm getting failures in a test, when I use a secret with a / in it
{code}
testInstantiateFromURL(org.apache.hadoop.fs.s3a.TestS3ACredentialsInURL) Time
elapsed: 0.146 sec <<< FAILURE!
java.lang.AssertionError: test URI encodes the / symbol
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.fs.s3a.TestS3ACredentialsInURL.testInstantiateFromURL(TestS3ACredentialsInURL.java:74)
{code}
Revert the patch and the test runs
The encoded string now contains "%252F"
> TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains
> '+'.
> ---
>
> Key: HADOOP-13287
> URL: https://issues.apache.org/jira/browse/HADOOP-13287
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3, test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HADOOP-13287.001.patch
>
>
> HADOOP-3733 fixed accessing S3A with credentials on the command line for an
> AWS secret key containing a '/'. The patch added a new test suite:
> {{TestS3ACredentialsInURL}}. One of the tests fails if your AWS secret key
> contains a '+'.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13291) Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented
[
https://issues.apache.org/jira/browse/HADOOP-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338528#comment-15338528
]
Steve Loughran commented on HADOOP-13291:
-
given this touches s3 code, It's going to need a validation run of the s3
tests...i'll do that after the run of chris's HADOOP-13287 patch completes
> Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be
> correctly implemented
> ---
>
> Key: HADOOP-13291
> URL: https://issues.apache.org/jira/browse/HADOOP-13291
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 2.8.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-13291.000.patch, HADOOP-13291.001.patch,
> HADOOP-13291.002.patch
>
>
> To probe a stat in {{StorageStatistics}}, users can use the
> {{StorageStatistics#isTracked()}} API. Currently {{DFSOpsCountStatistics}}
> implements this function wrongly. {{S3AStorageStatistics}} borrowed the same
> idea and also has the same error.
> # The {{isTracked()}} is not correctly implemented. I believe this was an
> omission in code.
> # {{isTracked()}} checks a stat with operation symbol (instead of enum name).
> {{getLongStatistics()}} should return LongStatistics iterators with symbol as
> name, instead of the enum variable name. Or else,
> {{isTracked(getLongStatistics().next().getName());}} returns false. This will
> lead to confusion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
