[jira] [Updated] (HADOOP-14722) Azure: BlockBlobInputStream position incorrect after seek
[ https://issues.apache.org/jira/browse/HADOOP-14722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14722: Resolution: Fixed Fix Version/s: 3.0.0-beta1 2.9.0 Status: Resolved (was: Patch Available) +1 committed to branch-2 & trunk, with clean build and a run of the new test on each branch to verify. Thanks. > Azure: BlockBlobInputStream position incorrect after seek > - > > Key: HADOOP-14722 > URL: https://issues.apache.org/jira/browse/HADOOP-14722 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HADOOP-14722-001.patch, HADOOP-14722-002.patch, > HADOOP-14722-003.patch, HADOOP-14722-005.patch, HADOOP-14722-006.patch > > > The seek, skip, and getPos methods of BlockBlobInputStream do not correctly > account for the stream's internal buffer. This results in invalid stream > positions. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14722) Azure: BlockBlobInputStream position incorrect after seek
[ https://issues.apache.org/jira/browse/HADOOP-14722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marquardt updated HADOOP-14722: -- Attachment: HADOOP-14722-006.patch Attaching HADOOP-14722-006.patch. Thanks for the feedback Steve! I took your 005 patch and 1) added additional comments, 2) updated *BlockBlobInputStream.seek()* so the offset is positive for forward seek and negative for reverse seek, 3) changed the test name *BasicReadTestV2* to *BasicReadTest* since it validates V1 and V2 are identical, and 4) changed the access modifier of *TestBlockBlobInputStream.verifyConsistentReads()* from protected to private. I reviewed the de-duplication change you made to test_0201_RandomReadTest and it looks perfect. All hadoop-tools/hadoop-azure *tests are passing* against my tmarql3 storage account. I set "fs.azure.secure.mode" to false due to the HADOOP-14715 issue we are still working on. Here are the test results: *Tests run: 774, Failures: 0, Errors: 0, Skipped: 155* > Azure: BlockBlobInputStream position incorrect after seek > - > > Key: HADOOP-14722 > URL: https://issues.apache.org/jira/browse/HADOOP-14722 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt > Attachments: HADOOP-14722-001.patch, HADOOP-14722-002.patch, > HADOOP-14722-003.patch, HADOOP-14722-005.patch, HADOOP-14722-006.patch > > > The seek, skip, and getPos methods of BlockBlobInputStream do not correctly > account for the stream's internal buffer. This results in invalid stream > positions. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14722) Azure: BlockBlobInputStream position incorrect after seek
[ https://issues.apache.org/jira/browse/HADOOP-14722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14722: Status: Open (was: Patch Available) > Azure: BlockBlobInputStream position incorrect after seek > - > > Key: HADOOP-14722 > URL: https://issues.apache.org/jira/browse/HADOOP-14722 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt > Attachments: HADOOP-14722-001.patch, HADOOP-14722-002.patch, > HADOOP-14722-003.patch, HADOOP-14722-005.patch > > > The seek, skip, and getPos methods of BlockBlobInputStream do not correctly > account for the stream's internal buffer. This results in invalid stream > positions. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14722) Azure: BlockBlobInputStream position incorrect after seek
[
https://issues.apache.org/jira/browse/HADOOP-14722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14722:
Attachment: HADOOP-14722-005.patch
I think it's OK, though I had a hard time reading through skip(). I've added
some comments there.
I also found it a bit odd reading through seek(), as the value of {{offset}} is
negative for forward seeks; this is the opposite of other examples
{{S3AInputStream.seekInStream()}}, {{SwiftNativeInputStream,seek()}}. It'd
really be good for future maintenance if this class's seek was consistent
*unless there's a really good reason not to*,
This has just made clear to me we've no FS contract tests for the semantics of
{{InputStream.skip()}}. I'll do it in the newly created HADOOP-14736 if/when I
get round to it.
Interestingly, {{InputStream.skip()}} is pretty vaguely defined; the base class
returns 0 for a -ve offset, and again, on any seek past EOF.
{{BlockBlobInputStream}} isn't following that, but neither does the only
implementation of skip() which is tested, the CryptoStream. Nobody has
complained so far.
h3. {{TestBlockBlobInputStream.test_0201_RandomReadTest}}
There's a lot of duplication in this test. Factored it out into a helper method
for a bit of clarity.
Also: noticed the {{assertEquals()}} args in {{test_0200_BasicReadTestV2}} were
the wrong way round: fixed.
Overall, I'm happy, as long as you are confident that the tests verify the
problem has been fixed. I would like that {{offset}} field inverted for
consistency with the others (or a justification for not doing so), and a full
test run through of my modified patch. I've already tested
{{TestBlockBlobInputStream}} against Azure Ireland.
> Azure: BlockBlobInputStream position incorrect after seek
> -
>
> Key: HADOOP-14722
> URL: https://issues.apache.org/jira/browse/HADOOP-14722
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Thomas Marquardt
>Assignee: Thomas Marquardt
> Attachments: HADOOP-14722-001.patch, HADOOP-14722-002.patch,
> HADOOP-14722-003.patch, HADOOP-14722-005.patch
>
>
> The seek, skip, and getPos methods of BlockBlobInputStream do not correctly
> account for the stream's internal buffer. This results in invalid stream
> positions.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14722) Azure: BlockBlobInputStream position incorrect after seek
[ https://issues.apache.org/jira/browse/HADOOP-14722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14722: Status: Patch Available (was: Open) > Azure: BlockBlobInputStream position incorrect after seek > - > > Key: HADOOP-14722 > URL: https://issues.apache.org/jira/browse/HADOOP-14722 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt > Attachments: HADOOP-14722-001.patch, HADOOP-14722-002.patch, > HADOOP-14722-003.patch, HADOOP-14722-005.patch > > > The seek, skip, and getPos methods of BlockBlobInputStream do not correctly > account for the stream's internal buffer. This results in invalid stream > positions. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14722) Azure: BlockBlobInputStream position incorrect after seek
[ https://issues.apache.org/jira/browse/HADOOP-14722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marquardt updated HADOOP-14722: -- Attachment: HADOOP-14722-003.patch Attaching patch HADOOP-14722-003.patch. This addresses the feedback, fixes an issue with the BlockBlobInputStream.skip implementation, and adds additional test coverage so all the code paths are exercised for seek and skip. All hadoop-tools/hadoop-azure *tests are passing* with this patch, except for TestWasbRemoteCallHelper which is a known issue tracked by HADOOP-14715. I tested against my endpoint tmarql3. > Azure: BlockBlobInputStream position incorrect after seek > - > > Key: HADOOP-14722 > URL: https://issues.apache.org/jira/browse/HADOOP-14722 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt > Attachments: HADOOP-14722-001.patch, HADOOP-14722-002.patch, > HADOOP-14722-003.patch > > > The seek, skip, and getPos methods of BlockBlobInputStream do not correctly > account for the stream's internal buffer. This results in invalid stream > positions. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14722) Azure: BlockBlobInputStream position incorrect after seek
[ https://issues.apache.org/jira/browse/HADOOP-14722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marquardt updated HADOOP-14722: -- Attachment: HADOOP-14722-002.patch Attaching HADOOP-14722-002.patch. This addresses the findbugs warning. > Azure: BlockBlobInputStream position incorrect after seek > - > > Key: HADOOP-14722 > URL: https://issues.apache.org/jira/browse/HADOOP-14722 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt > Attachments: HADOOP-14722-001.patch, HADOOP-14722-002.patch > > > The seek, skip, and getPos methods of BlockBlobInputStream do not correctly > account for the stream's internal buffer. This results in invalid stream > positions. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14722) Azure: BlockBlobInputStream position incorrect after seek
[ https://issues.apache.org/jira/browse/HADOOP-14722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marquardt updated HADOOP-14722: -- Status: Patch Available (was: Open) > Azure: BlockBlobInputStream position incorrect after seek > - > > Key: HADOOP-14722 > URL: https://issues.apache.org/jira/browse/HADOOP-14722 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt > Attachments: HADOOP-14722-001.patch > > > The seek, skip, and getPos methods of BlockBlobInputStream do not correctly > account for the stream's internal buffer. This results in invalid stream > positions. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14722) Azure: BlockBlobInputStream position incorrect after seek
[ https://issues.apache.org/jira/browse/HADOOP-14722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marquardt updated HADOOP-14722: -- Attachment: HADOOP-14722-001.patch Attaching HADOOP-14722-001.patch. The fix includes a new test case (*TestBlockBlobInputStream.test_0201_RandomReadTest*) which fails without the fix and passes with the fix. I ran *hadoop-tools/hadoop-azure* tests against my endpoint tmarql3. All tests are passing except TestWasbRemoteCallHelper which is tracked by HADOOP-14715 and unrelated to the patch. Tests run: 765, Failures: 1, Errors: 0, Skipped 70 > Azure: BlockBlobInputStream position incorrect after seek > - > > Key: HADOOP-14722 > URL: https://issues.apache.org/jira/browse/HADOOP-14722 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt > Attachments: HADOOP-14722-001.patch > > > The seek, skip, and getPos methods of BlockBlobInputStream do not correctly > account for the stream's internal buffer. This results in invalid stream > positions. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
