[jira] [Commented] (HADOOP-15446) WASB: PageBlobInputStream.skip breaks HBASE replication
[
https://issues.apache.org/jira/browse/HADOOP-15446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128251#comment-17128251
]
Andras Bokor commented on HADOOP-15446:
---
Git greppers!
2 commits belongs to this ticket:
{noformat}
HADOOP-15446. WASB: PageBlobInputStream.skip breaks HBASE replication.
HADOOP-15446. ABFS: tune imports & javadocs; stabilise tests.
{noformat}
The second one actually belongs to HADOOP-15546.
> WASB: PageBlobInputStream.skip breaks HBASE replication
> ---
>
> Key: HADOOP-15446
> URL: https://issues.apache.org/jira/browse/HADOOP-15446
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.9.0, 3.0.2
>Reporter: Thomas Marqardt
>Assignee: Thomas Marqardt
>Priority: Major
> Fix For: 2.10.0, 3.2.0, 3.1.1, 2.9.2
>
> Attachments: HADOOP-15446-001.patch, HADOOP-15446-002.patch,
> HADOOP-15446-003.patch, HADOOP-15446-branch-2.001.patch
>
>
> Page Blobs are primarily used by HBASE. HBASE replication, which apparently
> has not been used with WASB until recently, performs non-sequential reads on
> log files using PageBlobInputStream. There are bugs in this stream
> implementation which prevent skip and seek from working properly, and
> eventually the stream state becomes corrupt and unusable.
> I believe this bug affects all releases of WASB/HADOOP. It appears to be a
> day-0 bug in PageBlobInputStream. There were similar bugs opened in the past
> (HADOOP-15042) but the issue was not properly fixed, and no test coverage was
> added.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15446) WASB: PageBlobInputStream.skip breaks HBASE replication
[ https://issues.apache.org/jira/browse/HADOOP-15446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624930#comment-16624930 ] Hudson commented on HADOOP-15446: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15037 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15037/]) HADOOP-15446. ABFS: tune imports & javadocs; stabilise tests. (tmarq: rev ce03a93f78c4d97ccb48a3906fcd77ad0ac756be) * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/HttpHeaderConfigurations.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/diagnostics/TestConfigurationValidators.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/Abfss.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractAppendTest.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemCreate.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemBackCompat.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/annotations/ConfigurationValidationAnnotations.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemFileStatus.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/exceptions/AzureBlobFileSystemException.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemFlush.java * (edit) hadoop-tools/hadoop-azure/src/test/resources/log4j.properties * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/contract/ITestAbfsFileSystemContractSecureDistCp.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestFileSystemRegistration.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/contract/ITestAbfsFileSystemContractSetTimes.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ReadBuffer.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractConcatTest.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestFileSystemProperties.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/exceptions/InvalidUriException.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemE2EScale.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/contract/ITestAbfsFileSystemContractDistCp.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsReadWriteAndSeek.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemE2E.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/diagnostics/IntegerConfigurationBasicValidator.java * (edit) hadoop-tools/hadoop-azure/src/test/resources/azure-bfs-test.xml * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/ITestAzureNativeContractAppend.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/diagnostics/LongConfigurationBasicValidator.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemMkDir.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemInitAndCreate.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/diagnostics/ConfigurationValidator.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/contract/AbfsFileSystemContract.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/contract/ITestAbfsFileSystemContractCreate.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRename.java * (edit) hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/contract/ITestAbfsFileSystemContractConcat.java * (edit) hadoop-tools/hadoop-azure/src/site/markdown/testing_azure.md * (edit)
[jira] [Commented] (HADOOP-15446) WASB: PageBlobInputStream.skip breaks HBASE replication
[ https://issues.apache.org/jira/browse/HADOOP-15446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470848#comment-16470848 ] Arun Suresh commented on HADOOP-15446: -- Committed this to branch-2 and branch-2.9 as well (I didn't see a difference in the code between branch-2 and trunk - so, instead of applying the provided patch, I just cherry-picked the trunk commit and ran the tests - which passed) > WASB: PageBlobInputStream.skip breaks HBASE replication > --- > > Key: HADOOP-15446 > URL: https://issues.apache.org/jira/browse/HADOOP-15446 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 2.9.0, 3.0.2 >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt >Priority: Major > Fix For: 2.10.0, 3.2.0 > > Attachments: HADOOP-15446-001.patch, HADOOP-15446-002.patch, > HADOOP-15446-003.patch, HADOOP-15446-branch-2.001.patch > > > Page Blobs are primarily used by HBASE. HBASE replication, which apparently > has not been used with WASB until recently, performs non-sequential reads on > log files using PageBlobInputStream. There are bugs in this stream > implementation which prevent skip and seek from working properly, and > eventually the stream state becomes corrupt and unusable. > I believe this bug affects all releases of WASB/HADOOP. It appears to be a > day-0 bug in PageBlobInputStream. There were similar bugs opened in the past > (HADOOP-15042) but the issue was not properly fixed, and no test coverage was > added. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15446) WASB: PageBlobInputStream.skip breaks HBASE replication
[
https://issues.apache.org/jira/browse/HADOOP-15446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466862#comment-16466862
]
genericqa commented on HADOOP-15446:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
15s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 12s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated
20 new + 3 unchanged - 0 fixed = 23 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
3s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:f667ef1 |
| JIRA Issue | HADOOP-15446 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12922393/HADOOP-15446-branch-2.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 0b0a6a021c83 3.13.0-137-generic #186-Ubuntu SMP Mon Dec 4
19:09:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / 5679920 |
| maven | version: Apache Maven 3.3.9
(bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00) |
| Default Java | 1.7.0_171 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14600/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14600/testReport/ |
| Max. process+thread count | 190 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14600/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> WASB: PageBlobInputStream.skip breaks HBASE replication
> ---
>
> Key: HADOOP-15446
> URL: https://issues.apache.org/jira/browse/HADOOP-15446
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.9.0, 3.0.2
>
[jira] [Commented] (HADOOP-15446) WASB: PageBlobInputStream.skip breaks HBASE replication
[ https://issues.apache.org/jira/browse/HADOOP-15446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466846#comment-16466846 ] Thomas Marquardt commented on HADOOP-15446: --- FYI, the same changes apply to both branch-2 and trunk. > WASB: PageBlobInputStream.skip breaks HBASE replication > --- > > Key: HADOOP-15446 > URL: https://issues.apache.org/jira/browse/HADOOP-15446 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 2.9.0, 3.0.2 >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt >Priority: Major > Attachments: HADOOP-15446-001.patch, HADOOP-15446-002.patch, > HADOOP-15446-003.patch, HADOOP-15446-branch-2.001.patch > > > Page Blobs are primarily used by HBASE. HBASE replication, which apparently > has not been used with WASB until recently, performs non-sequential reads on > log files using PageBlobInputStream. There are bugs in this stream > implementation which prevent skip and seek from working properly, and > eventually the stream state becomes corrupt and unusable. > I believe this bug affects all releases of WASB/HADOOP. It appears to be a > day-0 bug in PageBlobInputStream. There were similar bugs opened in the past > (HADOOP-15042) but the issue was not properly fixed, and no test coverage was > added. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15446) WASB: PageBlobInputStream.skip breaks HBASE replication
[ https://issues.apache.org/jira/browse/HADOOP-15446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466843#comment-16466843 ] Thomas Marquardt commented on HADOOP-15446: --- For branch-2, I've attached HADOOP-15446-branch-2.001.patch. All tests are passing against my storage account: *$ mvn test -Dtest=ITestPageBlobInputStream#** [INFO] Running org.apache.hadoop.fs.azure.ITestPageBlobInputStream [INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.212 s - in org.apache.hadoop.fs.azure.ITestPageBlobInputStream *$ mvn -T 1C clean verify* [INFO] Results: [INFO] [WARNING] Tests run: 233, Failures: 0, Errors: 0, Skipped: 4 [INFO] Results: [INFO] [WARNING] Tests run: 570, Failures: 0, Errors: 0, Skipped: 12 > WASB: PageBlobInputStream.skip breaks HBASE replication > --- > > Key: HADOOP-15446 > URL: https://issues.apache.org/jira/browse/HADOOP-15446 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 2.9.0, 3.0.2 >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt >Priority: Major > Attachments: HADOOP-15446-001.patch, HADOOP-15446-002.patch, > HADOOP-15446-003.patch, HADOOP-15446-branch-2.001.patch > > > Page Blobs are primarily used by HBASE. HBASE replication, which apparently > has not been used with WASB until recently, performs non-sequential reads on > log files using PageBlobInputStream. There are bugs in this stream > implementation which prevent skip and seek from working properly, and > eventually the stream state becomes corrupt and unusable. > I believe this bug affects all releases of WASB/HADOOP. It appears to be a > day-0 bug in PageBlobInputStream. There were similar bugs opened in the past > (HADOOP-15042) but the issue was not properly fixed, and no test coverage was > added. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15446) WASB: PageBlobInputStream.skip breaks HBASE replication
[ https://issues.apache.org/jira/browse/HADOOP-15446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466557#comment-16466557 ] Thomas Marquardt commented on HADOOP-15446: --- Yes, I will submit a branch-2 patch later today. > WASB: PageBlobInputStream.skip breaks HBASE replication > --- > > Key: HADOOP-15446 > URL: https://issues.apache.org/jira/browse/HADOOP-15446 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 2.9.0, 3.0.2 >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt >Priority: Major > Attachments: HADOOP-15446-001.patch, HADOOP-15446-002.patch, > HADOOP-15446-003.patch > > > Page Blobs are primarily used by HBASE. HBASE replication, which apparently > has not been used with WASB until recently, performs non-sequential reads on > log files using PageBlobInputStream. There are bugs in this stream > implementation which prevent skip and seek from working properly, and > eventually the stream state becomes corrupt and unusable. > I believe this bug affects all releases of WASB/HADOOP. It appears to be a > day-0 bug in PageBlobInputStream. There were similar bugs opened in the past > (HADOOP-15042) but the issue was not properly fixed, and no test coverage was > added. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15446) WASB: PageBlobInputStream.skip breaks HBASE replication
[ https://issues.apache.org/jira/browse/HADOOP-15446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466250#comment-16466250 ] Duo Xu commented on HADOOP-15446: - [~tmarquardt] & [~ste...@apache.org] Could we backport this to branch-2? Thanks! > WASB: PageBlobInputStream.skip breaks HBASE replication > --- > > Key: HADOOP-15446 > URL: https://issues.apache.org/jira/browse/HADOOP-15446 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 2.9.0, 3.0.2 >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt >Priority: Major > Attachments: HADOOP-15446-001.patch, HADOOP-15446-002.patch, > HADOOP-15446-003.patch > > > Page Blobs are primarily used by HBASE. HBASE replication, which apparently > has not been used with WASB until recently, performs non-sequential reads on > log files using PageBlobInputStream. There are bugs in this stream > implementation which prevent skip and seek from working properly, and > eventually the stream state becomes corrupt and unusable. > I believe this bug affects all releases of WASB/HADOOP. It appears to be a > day-0 bug in PageBlobInputStream. There were similar bugs opened in the past > (HADOOP-15042) but the issue was not properly fixed, and no test coverage was > added. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15446) WASB: PageBlobInputStream.skip breaks HBASE replication
[ https://issues.apache.org/jira/browse/HADOOP-15446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465799#comment-16465799 ] Hudson commented on HADOOP-15446: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14132 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14132/]) HADOOP-15446. WASB: PageBlobInputStream.skip breaks HBASE replication. (stevel: rev 5b11b9fd413470e134ecdc7c50468f8c7b39fa50) * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobInputStream.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/ITestPageBlobInputStream.java > WASB: PageBlobInputStream.skip breaks HBASE replication > --- > > Key: HADOOP-15446 > URL: https://issues.apache.org/jira/browse/HADOOP-15446 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 2.9.0, 3.0.2 >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt >Priority: Major > Attachments: HADOOP-15446-001.patch, HADOOP-15446-002.patch, > HADOOP-15446-003.patch > > > Page Blobs are primarily used by HBASE. HBASE replication, which apparently > has not been used with WASB until recently, performs non-sequential reads on > log files using PageBlobInputStream. There are bugs in this stream > implementation which prevent skip and seek from working properly, and > eventually the stream state becomes corrupt and unusable. > I believe this bug affects all releases of WASB/HADOOP. It appears to be a > day-0 bug in PageBlobInputStream. There were similar bugs opened in the past > (HADOOP-15042) but the issue was not properly fixed, and no test coverage was > added. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15446) WASB: PageBlobInputStream.skip breaks HBASE replication
[
https://issues.apache.org/jira/browse/HADOOP-15446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465783#comment-16465783
]
Steve Loughran commented on HADOOP-15446:
-
+1, committed to branch 3.1 & trunk. If you want backporting to branch-2, run
the tests, tell me how it went, & I'll backport.
I did see one failure in my own test run, I'm assuming unrelated and just a
function of network distance.
{code}
[ERROR] Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 26.213
s <<< FAILURE! - in org.apache.hadoop.fs.azure.TestClientThrottlingAnalyzer
[ERROR]
testManySuccessAndErrorsAndWaiting(org.apache.hadoop.fs.azure.TestClientThrottlingAnalyzer)
Time elapsed: 1.123 s <<< FAILURE!
java.lang.AssertionError: The actual value 9 is not within the expected range:
[5.60, 8.40].
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.fs.azure.TestClientThrottlingAnalyzer.fuzzyValidate(TestClientThrottlingAnalyzer.java:46)
at
org.apache.hadoop.fs.azure.TestClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting(TestClientThrottlingAnalyzer.java:168)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
> WASB: PageBlobInputStream.skip breaks HBASE replication
> ---
>
> Key: HADOOP-15446
> URL: https://issues.apache.org/jira/browse/HADOOP-15446
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.9.0, 3.0.2
>Reporter: Thomas Marquardt
>Assignee: Thomas Marquardt
>Priority: Major
> Attachments: HADOOP-15446-001.patch, HADOOP-15446-002.patch,
> HADOOP-15446-003.patch
>
>
> Page Blobs are primarily used by HBASE. HBASE replication, which apparently
> has not been used with WASB until recently, performs non-sequential reads on
> log files using PageBlobInputStream. There are bugs in this stream
> implementation which prevent skip and seek from working properly, and
> eventually the stream state becomes corrupt and unusable.
> I believe this bug affects all releases of WASB/HADOOP. It appears to be a
> day-0 bug in PageBlobInputStream. There were similar bugs opened in the past
> (HADOOP-15042) but the issue was not properly fixed, and no test coverage was
> added.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15446) WASB: PageBlobInputStream.skip breaks HBASE replication
[
https://issues.apache.org/jira/browse/HADOOP-15446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464538#comment-16464538
]
genericqa commented on HADOOP-15446:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
49s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 31m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 44s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated
20 new + 4 unchanged - 0 fixed = 24 total (was 4) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 0s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
3s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 81m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HADOOP-15446 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12922113/HADOOP-15446-003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 4e71b61fd9a0 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6a69239 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14584/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14584/testReport/ |
| Max. process+thread count | 380 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14584/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
[jira] [Commented] (HADOOP-15446) WASB: PageBlobInputStream.skip breaks HBASE replication
[ https://issues.apache.org/jira/browse/HADOOP-15446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464483#comment-16464483 ] Thomas Marquardt commented on HADOOP-15446: --- Hi Steve, I have attached HADOOP-15446-003.patch. I renamed the test class ITestPageBlobInputStream, since tests that connect to Azure are supposed to begin with ITest. However, since i) these are not scale tests and ii) scale tests are not required to be run before committing changes (according to testing_azure.md), I have not marked them as scale tests. These tests cover functional behavior, not performance or scalability, and should be run before every commit in my opinion. The slowest of these tests takes me 3 seconds to run, and all of the tests in ITestPageBlobInputStream takes me between 7 and 9 seconds to run. I really do not consider them to be scale tests by any means. I know it takes longer for you over the internet to Ireland, but hopefully these new tests can still be run in under 60 seconds. The file that they create is about 6 MB, so it is tiny in the big data world. Let me know if I'm missing something. Thanks, Thomas > WASB: PageBlobInputStream.skip breaks HBASE replication > --- > > Key: HADOOP-15446 > URL: https://issues.apache.org/jira/browse/HADOOP-15446 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 2.9.0, 3.0.2 >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt >Priority: Major > Attachments: HADOOP-15446-001.patch, HADOOP-15446-002.patch, > HADOOP-15446-003.patch > > > Page Blobs are primarily used by HBASE. HBASE replication, which apparently > has not been used with WASB until recently, performs non-sequential reads on > log files using PageBlobInputStream. There are bugs in this stream > implementation which prevent skip and seek from working properly, and > eventually the stream state becomes corrupt and unusable. > I believe this bug affects all releases of WASB/HADOOP. It appears to be a > day-0 bug in PageBlobInputStream. There were similar bugs opened in the past > (HADOOP-15042) but the issue was not properly fixed, and no test coverage was > added. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15446) WASB: PageBlobInputStream.skip breaks HBASE replication
[ https://issues.apache.org/jira/browse/HADOOP-15446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464352#comment-16464352 ] Steve Loughran commented on HADOOP-15446: - bq. I don't experience the latency you do, and didn't notice that I forgot to remove the latency check. no, I don't expect you do :) All the checkstyle warnings can be ignored, as they are legit changes. Given the test is doing real IO, can you * stick it under org.apache.hadoop.fs.azure.integration * make a subclass of org.apache.hadoop.fs.azure.integration.AbstractAzureScaleTest, which ensures the test only runs with a -Dscale option. I'll enable this when I do my premerge test run. * give it the prefix ITest. other than that, LGTM thanks > WASB: PageBlobInputStream.skip breaks HBASE replication > --- > > Key: HADOOP-15446 > URL: https://issues.apache.org/jira/browse/HADOOP-15446 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 2.9.0, 3.0.2 >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt >Priority: Major > Attachments: HADOOP-15446-001.patch, HADOOP-15446-002.patch > > > Page Blobs are primarily used by HBASE. HBASE replication, which apparently > has not been used with WASB until recently, performs non-sequential reads on > log files using PageBlobInputStream. There are bugs in this stream > implementation which prevent skip and seek from working properly, and > eventually the stream state becomes corrupt and unusable. > I believe this bug affects all releases of WASB/HADOOP. It appears to be a > day-0 bug in PageBlobInputStream. There were similar bugs opened in the past > (HADOOP-15042) but the issue was not properly fixed, and no test coverage was > added. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15446) WASB: PageBlobInputStream.skip breaks HBASE replication
[
https://issues.apache.org/jira/browse/HADOOP-15446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464164#comment-16464164
]
genericqa commented on HADOOP-15446:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 14s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated
20 new + 4 unchanged - 0 fixed = 24 total (was 4) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 19s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
5s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HADOOP-15446 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12922079/HADOOP-15446-002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 22cbd392cc77 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6795f80 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14581/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14581/testReport/ |
| Max. process+thread count | 341 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14581/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
[jira] [Commented] (HADOOP-15446) WASB: PageBlobInputStream.skip breaks HBASE replication
[ https://issues.apache.org/jira/browse/HADOOP-15446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464138#comment-16464138 ] Thomas Marquardt commented on HADOOP-15446: --- I've attached HADOOP-15446-002.patch with the test fix. Sorry about that Steve. I copied the tests from TestBlockBlobInputStream because it covers random access well, but did not intend to include the performance checks. For Page Blob, it is necessary to read the last page to determine the length of the data, so the network I/O is expected. Each page has a 16-bit header which stores the length of the data in the page, and all but the last page is guaranteed to be full with data. Since my storage account and Hadoop development machine are in the same region, I don't experience the latency you do, and didn't notice that I forgot to remove the latency check. With HADOOP-15446-002.patch, all WASB test cases are passing against my Azure storage account: *$ mvn -T 1C -Dparallel-tests clean verify* Tests run: 252, Failures: 0, Errors: 0, Skipped: 11 Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 Tests run: 418, Failures: 0, Errors: 0, Skipped: 55 Tests run: 126, Failures: 0, Errors: 0, Skipped: 10 > WASB: PageBlobInputStream.skip breaks HBASE replication > --- > > Key: HADOOP-15446 > URL: https://issues.apache.org/jira/browse/HADOOP-15446 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 2.9.0, 3.0.2 >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt >Priority: Major > Attachments: HADOOP-15446-001.patch, HADOOP-15446-002.patch > > > Page Blobs are primarily used by HBASE. HBASE replication, which apparently > has not been used with WASB until recently, performs non-sequential reads on > log files using PageBlobInputStream. There are bugs in this stream > implementation which prevent skip and seek from working properly, and > eventually the stream state becomes corrupt and unusable. > I believe this bug affects all releases of WASB/HADOOP. It appears to be a > day-0 bug in PageBlobInputStream. There were similar bugs opened in the past > (HADOOP-15042) but the issue was not properly fixed, and no test coverage was > added. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15446) WASB: PageBlobInputStream.skip breaks HBASE replication
[
https://issues.apache.org/jira/browse/HADOOP-15446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463787#comment-16463787
]
Steve Loughran commented on HADOOP-15446:
-
I ran this against azure ireland and got two test failures
{code}
[ERROR] Tests run: 11, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 40.141
s <<< FAILURE! - in org.apache.hadoop.fs.azure.TestPageBlobInputStream
[ERROR]
test_0307_SkipBounds(org.apache.hadoop.fs.azure.TestPageBlobInputStream) Time
elapsed: 0.773 s <<< FAILURE!
java.lang.AssertionError: There should not be any network I/O
(elapsedTimeMs=36).
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.fs.azure.TestPageBlobInputStream.test_0307_SkipBounds(TestPageBlobInputStream.java:363)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
[ERROR]
test_0309_SeekBounds(org.apache.hadoop.fs.azure.TestPageBlobInputStream) Time
elapsed: 0.755 s <<< FAILURE!
java.lang.AssertionError: There should not be any network I/O
(elapsedTimeMs=36).
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.fs.azure.TestPageBlobInputStream.test_0309_SeekBounds(TestPageBlobInputStream.java:414)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
> WASB: PageBlobInputStream.skip breaks HBASE replication
> ---
>
> Key: HADOOP-15446
> URL: https://issues.apache.org/jira/browse/HADOOP-15446
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.9.0, 3.0.2
>Reporter: Thomas Marquardt
>Assignee: Thomas Marquardt
>Priority: Major
> Attachments: HADOOP-15446-001.patch
>
>
> Page Blobs are primarily used by HBASE. HBASE replication, which apparently
> has not been used with WASB until recently, performs non-sequential reads on
> log files using PageBlobInputStream. There are bugs in this stream
> implementation which prevent skip and seek from working properly, and
> eventually the stream state becomes corrupt and unusable.
> I believe this bug affects all releases of WASB/HADOOP. It appears to be a
> day-0 bug in PageBlobInputStream. There were similar bugs opened in the past
> (HADOOP-15042) but the issue was not properly fixed, and no test coverage was
> added.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15446) WASB: PageBlobInputStream.skip breaks HBASE replication
[
https://issues.apache.org/jira/browse/HADOOP-15446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463275#comment-16463275
]
genericqa commented on HADOOP-15446:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 30m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 7s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 17s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated
23 new + 4 unchanged - 0 fixed = 27 total (was 4) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 56s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
9s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HADOOP-15446 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12921867/HADOOP-15446-001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 5e959d34f1f8 3.13.0-137-generic #186-Ubuntu SMP Mon Dec 4
19:09:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a3b416f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14578/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14578/testReport/ |
| Max. process+thread count | 302 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14578/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
[jira] [Commented] (HADOOP-15446) WASB: PageBlobInputStream.skip breaks HBASE replication
[ https://issues.apache.org/jira/browse/HADOOP-15446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463261#comment-16463261 ] Shane Mainali commented on HADOOP-15446: +1, looks good to me. Thanks [~tmarquardt]! > WASB: PageBlobInputStream.skip breaks HBASE replication > --- > > Key: HADOOP-15446 > URL: https://issues.apache.org/jira/browse/HADOOP-15446 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 2.9.0, 3.0.2 >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt >Priority: Major > Attachments: HADOOP-15446-001.patch > > > Page Blobs are primarily used by HBASE. HBASE replication, which apparently > has not been used with WASB until recently, performs non-sequential reads on > log files using PageBlobInputStream. There are bugs in this stream > implementation which prevent skip and seek from working properly, and > eventually the stream state becomes corrupt and unusable. > I believe this bug affects all releases of WASB/HADOOP. It appears to be a > day-0 bug in PageBlobInputStream. There were similar bugs opened in the past > (HADOOP-15042) but the issue was not properly fixed, and no test coverage was > added. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
