[
https://issues.apache.org/jira/browse/NIFI-12923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Balázs Gerner updated NIFI-12923:
---------------------------------
Description:
The goal of this ticket is to extend the PutHDFS processor with the ability to
append avro records. The processor already provides an option to set 'append'
as conflict resolution strategy, but that does not work correctly in case of
avro files, because the serialized avro file cannot be deserialized again
(because the binary content is invalid).
Some notes about the implementation:
* The user needs to explicitly select avro as file format and append as
conflict resolution mode to enable 'avro append' mode, otherwise regular append
mode will work just as before. There is no auto detection of mimetype for the
incoming flowfile.
* The records of the incoming flowfile and the ones in the existing avro file
need to conform to the same avro schema, otherwise the append operation fails
with incompatible schema.
* The 'avro append' mode should only work when compression type is set to
'none', if any other compression type is selected in 'avro append' mode the
user should get a validation error.
The changes will have to be added to *support/nifi-1.x* branch also.
was:
The goal of this ticket is to extend the PutHDFS processor with the ability to
append avro records. The processor already provides an option to set 'append'
as conflict resolution strategy, but that does not work correctly in case of
avro files, because the serialized avro file cannot be deserialized again
(because the binary content is invalid).
Some notes about the implementation:
* The user needs to explicitly select avro as file format and append as
conflict resolution mode to enable 'avro append' mode, otherwise regular append
mode will work just as before. There is no auto detection of mimetype for the
incoming flowfile.
* The records of the incoming flowfile and the ones in the existing avro file
need to conform to the same avro schema, otherwise the append operation fails
with incompatible schema.
* The 'avro append' mode should only work when compression type is set to
'none', if any other compression type is selected in 'avro append' mode the
user should get a validation error.
> PutHDFS to support appending avro data
> --------------------------------------
>
> Key: NIFI-12923
> URL: https://issues.apache.org/jira/browse/NIFI-12923
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: Balázs Gerner
> Assignee: Balázs Gerner
> Priority: Major
>
> The goal of this ticket is to extend the PutHDFS processor with the ability
> to append avro records. The processor already provides an option to set
> 'append' as conflict resolution strategy, but that does not work correctly in
> case of avro files, because the serialized avro file cannot be deserialized
> again (because the binary content is invalid).
> Some notes about the implementation:
> * The user needs to explicitly select avro as file format and append as
> conflict resolution mode to enable 'avro append' mode, otherwise regular
> append mode will work just as before. There is no auto detection of mimetype
> for the incoming flowfile.
> * The records of the incoming flowfile and the ones in the existing avro
> file need to conform to the same avro schema, otherwise the append operation
> fails with incompatible schema.
> * The 'avro append' mode should only work when compression type is set to
> 'none', if any other compression type is selected in 'avro append' mode the
> user should get a validation error.
> The changes will have to be added to *support/nifi-1.x* branch also.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
