[
https://issues.apache.org/jira/browse/AVRO-3266?focusedWorklogId=778019&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-778019
]
ASF GitHub Bot logged work on AVRO-3266:
----------------------------------------
Author: ASF GitHub Bot
Created on: 03/Jun/22 10:20
Start Date: 03/Jun/22 10:20
Worklog Time Spent: 10m
Work Description: eejbyfeldt commented on PR #1618:
URL: https://github.com/apache/avro/pull/1618#issuecomment-1145818671
> Looks reasonable good to me as well, but personally I prefer PR #1431
because it doesn't use reflection (there's no need; the `PathOutputComitter`
interface is generic enough).
>
> On the other hand: I don't use Hadoop, so maybe I'm missing something.
As discussed in #1431 that approach does not work for hadoop 2.x and since
hadoop 2 is still not EOL (Actually made a release last week:
https://hadoop.apache.org/release/2.10.2.html) this approach was suggest as a
way to be compatible with both versions. `PathOutputComitter` was is only
introduced into hadoop 3 in
https://github.com/apache/hadoop/commit/11390c2d111910b01d9c4d3e39dee49babae272f
But if the need to support hadoop 2 has changed since this was originally
suggested, then I also agree that the approach in #1431 is better.
Issue Time Tracking
-------------------
Worklog Id: (was: 778019)
Time Spent: 2h 50m (was: 2h 40m)
> Output stream incompatible with MagicS3GuardCommitter
> -----------------------------------------------------
>
> Key: AVRO-3266
> URL: https://issues.apache.org/jira/browse/AVRO-3266
> Project: Apache Avro
> Issue Type: Bug
> Components: java
> Reporter: Michiel de Jong
> Assignee: Emil Ejbyfeldt
> Priority: Minor
> Labels: pull-request-available
> Fix For: 1.11.1
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Avro's output stream can not be used in combination with the
> MagicS3GuardCommitter
> {code:java}
> Error: java.lang.ClassCastException: class
> org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter cannot be cast to
> class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter{code}
> The reason for this problem is that
> AvroOutputFormatBase.getAvroFileOutputStream tries to cast the outputcommiter
> to a FileOutputCommitter.
> It can be solved by casting to a PathOutputCommitter instead (which is a
> superclass of both the FileOutputCommitter and the MagicS3GuardCommitter)
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
