Balázs Gerner created NIFI-12923:
------------------------------------
Summary: PutHDFS to support appending avro data
Key: NIFI-12923
URL: https://issues.apache.org/jira/browse/NIFI-12923
Project: Apache NiFi
Issue Type: Improvement
Reporter: Balázs Gerner
Assignee: Balázs Gerner
The goal of this ticket is to extend the PutHDFS processor with the ability to
append avro records. The processor already provides an option to set 'append'
as conflict resolution strategy, but that does not work correctly in case of
avro files, because the serialized avro file cannot be deserialized again
(because the binary content is invalid).
Some notes about the implementation:
* The user needs to explicitly select avro as file format and append as
conflict resolution mode to enable 'avro append' mode, otherwise regular append
mode will work just as before. There is no auto detection of mimetype for the
incoming flowfile.
* The records of the incoming flowfile and the ones in the existing avro file
need to conform to the same avro schema, otherwise the append operation fails
with incompatible schema.
* The 'avro append' mode should only work when compression type is set to
'none', if any other compression type is selected in 'avro append' mode the
user should get a validation error.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
