[
https://issues.apache.org/jira/browse/AVRO-3266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17561568#comment-17561568
]
ASF subversion and git services commented on AVRO-3266:
-------------------------------------------------------
Commit 8caab7bd4a47e2c8e048f2c42478bdb22b18c993 in avro's branch
refs/heads/branch-1.11 from Emil Ejbyfeldt
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=8caab7bd4 ]
AVRO-3266: Work with hadoop 3.x PathOutputComitters (#1618)
* AVRO-3266: Work with hadoop 3.x PathOutputComitters
In hadoop 3.x the abstract class PathOutputCommitter defines the method
`getWorkPath()`, but in hadoop 2.x it only defined on
FileOutputCommitter. So to be compatible with both hadoop 2.x and 3.x
and support committers that does only implements PathOutputComitter and
not FileOutputCommitter we make the call to getWorkPath using
reflection.
* AVRO-3266: Add committer class name to exception
* Add comment on better fix when hadoop2 is dropped
Co-authored-by: Ryan Skraba <[email protected]>
> Output stream incompatible with MagicS3GuardCommitter
> -----------------------------------------------------
>
> Key: AVRO-3266
> URL: https://issues.apache.org/jira/browse/AVRO-3266
> Project: Apache Avro
> Issue Type: Bug
> Components: java
> Reporter: Michiel de Jong
> Assignee: Emil Ejbyfeldt
> Priority: Minor
> Labels: pull-request-available
> Fix For: 1.11.1
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> Avro's output stream can not be used in combination with the
> MagicS3GuardCommitter
> {code:java}
> Error: java.lang.ClassCastException: class
> org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter cannot be cast to
> class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter{code}
> The reason for this problem is that
> AvroOutputFormatBase.getAvroFileOutputStream tries to cast the outputcommiter
> to a FileOutputCommitter.
> It can be solved by casting to a PathOutputCommitter instead (which is a
> superclass of both the FileOutputCommitter and the MagicS3GuardCommitter)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
