[
https://issues.apache.org/jira/browse/NIFI-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16019796#comment-16019796
]
Joseph Witt commented on NIFI-3759:
-----------------------------------
[~my2ndhead] Got ya. You're definitely right that smaller files should be
discouraged because they lead to inefficiencies compared to files more closely
aligned to the size. This is, as i understand it, both a space consideration
and an efficiency of data access consideration. Given the slow rate of the
flows this would affect, however, I'm not sure how problematic this will be in
practice given the efficiency of access would not matter as much (space I can
understand).
To me the hesitation here is that appending to a file is often not safe due to
competing producers (how do we avoid more than one system writing to that
file), conflicting format/schema (like in the case of avro), etc.. We can of
course still do it and just document these things. Just feels a bit off so
want to make sure we're eyes wide open on this.
> Enable Avro append for Put HDFS
> -------------------------------
>
> Key: NIFI-3759
> URL: https://issues.apache.org/jira/browse/NIFI-3759
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Affects Versions: 1.1.0, 1.1.1
> Reporter: Jonas Hartwig
> Priority: Minor
>
> It would be nice where Nifi already supports working with AVRO to enable hdfs
> append working correctly with avro files coming from MergeContent processor.
> I suggest making a similar choice in as in MergeContent to be able to choose
> a file format which applies file format specific logic when required.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
