[
https://issues.apache.org/jira/browse/BEAM-6697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777499#comment-16777499
]
Chamikara Jayalath commented on BEAM-6697:
------------------------------------------
I confirmed that the test passes with the library upgrade.
./gradlew integrationTest -p sdks/java/io/file-based-io-tests
-DintegrationTestPipelineOptions='["--project=apache-beam-testing",
"--gcpTempLocation=gs://chamikara-beam-test/temp",
"--tempRoot=gs://chamikara-beam-test/temproot", "--numberOfRecords=100000000",
"--filenamePrefix=gs://chamikara-beam-test/temproot",
"--runner=TestDataflowRunner"]' -DintegrationTestRunner=dataflow
--tests=org.apache.beam.sdk.io.parquet.ParquetIOIT
[https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-02-25_17_48_02-4265881975334840619?project=apache-beam-testing&folder&organizationId]
Will send out a PR.
> ParquetIO Performance test is failing on (GCS filesystem)
> ---------------------------------------------------------
>
> Key: BEAM-6697
> URL: https://issues.apache.org/jira/browse/BEAM-6697
> Project: Beam
> Issue Type: New Feature
> Components: io-java-parquet, test-failures
> Reporter: Lukasz Gajowy
> Assignee: Chamikara Jayalath
> Priority: Blocker
> Fix For: 2.11.0
>
>
> Relevant failure logs:
> {code:java}
> Caused by: java.lang.RuntimeException:
> org.apache.beam.sdk.io.parquet.ParquetIO$ReadFiles$BeamParquetInputFile@2de8303e
> is not a Parquet file (too small length: -1)
> at
> org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:514)
> at
> org.apache.parquet.hadoop.ParquetFileReader.<init>(ParquetFileReader.java:689)
> at
> org.apache.parquet.hadoop.ParquetFileReader.open(ParquetFileReader.java:595)
> at
> org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:152)
> at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135)
> at
> org.apache.beam.sdk.io.parquet.ParquetIO$ReadFiles$ReadFn.processElement(ParquetIO.java:221){code}
>
> Full logs can be found here:
> [https://builds.apache.org/view/A-D/view/Beam/view/PerformanceTests/job/beam_PerformanceTests_ParquetIOIT/|https://builds.apache.org/view/A-D/view/Beam/view/PerformanceTests/job/beam_PerformanceTests_ParquetIOIT/1096/console]
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
