[
https://issues.apache.org/jira/browse/BEAM-4421?focusedWorklogId=108206&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-108206
]
ASF GitHub Bot logged work on BEAM-4421:
----------------------------------------
Author: ASF GitHub Bot
Created on: 01/Jun/18 21:39
Start Date: 01/Jun/18 21:39
Worklog Time Spent: 10m
Work Description: iemejia closed pull request #5533: [BEAM-4421] Fix for
issue with reading s3 files using ParquetIO
URL: https://github.com/apache/beam/pull/5533
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/build_rules.gradle b/build_rules.gradle
index da30637a81a..e6ee81ec833 100644
--- a/build_rules.gradle
+++ b/build_rules.gradle
@@ -1058,6 +1058,11 @@ ext.provideIntegrationTestingDependencies = {
testCompile project(path: ":beam-sdks-java-io-hadoop-file-system",
configuration: 'shadowTest')
shadowTest library.java.hadoop_client
}
+
+ /* include dependencies required by AWS S3 */
+ if (filesystem?.equalsIgnoreCase('s3')) {
+ testCompile project(path: ":beam-sdks-java-io-amazon-web-services",
configuration: 'shadowTest')
+ }
}
}
}
diff --git
a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3ReadableSeekableByteChannel.java
b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3ReadableSeekableByteChannel.java
index f0cf1e073ef..e1e71910507 100644
---
a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3ReadableSeekableByteChannel.java
+++
b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3ReadableSeekableByteChannel.java
@@ -131,9 +131,11 @@ public SeekableByteChannel position(long newPosition)
throws IOException {
return this;
}
- // The position has changed, so close the object to induce a re-open on
the next call to read()
+ // The position has changed, so close and destroy the object to induce a
re-creation on the next
+ // call to read()
if (s3Object != null) {
s3Object.close();
+ s3Object = null;
}
position = newPosition;
return this;
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 108206)
Time Spent: 0.5h (was: 20m)
> Cannot read s3 files using ParquetIO
> ------------------------------------
>
> Key: BEAM-4421
> URL: https://issues.apache.org/jira/browse/BEAM-4421
> Project: Beam
> Issue Type: Bug
> Components: io-java-parquet
> Reporter: Łukasz Gajowy
> Assignee: Alexey Romanenko
> Priority: Major
> Fix For: 2.5.0
>
> Attachments: errorlog.txt
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> For s3 the read doesn't work and throws an IOException. Please see the
> enclosed logs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
