[
https://issues.apache.org/jira/browse/NIFI-12923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17839171#comment-17839171
]
Joe Witt commented on NIFI-12923:
---------------------------------
[~balazsgerner][~mattyb149] I am reverting the commit for this test change as
the two tests are unstable and need review/resolution.
{noformat}
[ERROR] Tests run: 20, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.478
s <<< FAILURE! -- in org.apache.nifi.processors.hadoop.PutHDFSTest
[ERROR]
org.apache.nifi.processors.hadoop.PutHDFSTest.testPutFileWithAppendAvroModeNewFileCreated
-- Time elapsed: 0.005 s <<< ERROR!
java.io.FileNotFoundException: src/test/resources/testdata-avro/input.avro (No
such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:213)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:152)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:106)
at
org.apache.nifi.processors.hadoop.PutHDFSTest.testPutFileWithAppendAvroModeNewFileCreated(PutHDFSTest.java:281)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
[ERROR]
org.apache.nifi.processors.hadoop.PutHDFSTest.testPutFileWithAppendAvroModeWhenTargetFileAlreadyExists
-- Time elapsed: 0.009 s <<< ERROR!
java.io.FileNotFoundException: src/test/resources/testdata-avro/input.avro (No
such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:213)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:152)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:106)
at
org.apache.nifi.processors.hadoop.PutHDFSTest.testPutFileWithAppendAvroModeWhenTargetFileAlreadyExists(PutHDFSTest.java:308)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
{noformat}
> PutHDFS to support appending avro data
> --------------------------------------
>
> Key: NIFI-12923
> URL: https://issues.apache.org/jira/browse/NIFI-12923
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: Balázs Gerner
> Assignee: Balázs Gerner
> Priority: Major
> Fix For: 2.0.0-M3, 1.26.0
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> The goal of this ticket is to extend the PutHDFS processor with the ability
> to append avro records. The processor already provides an option to set
> 'append' as conflict resolution strategy, but that does not work correctly in
> case of avro files, because the serialized avro file cannot be deserialized
> again (because the binary content is invalid).
> Some notes about the implementation:
> * The user needs to explicitly select avro as file format and append as
> conflict resolution mode to enable 'avro append' mode, otherwise regular
> append mode will work just as before. There is no auto detection of mimetype
> for the incoming flowfile.
> * The records of the incoming flowfile and the ones in the existing avro
> file need to conform to the same avro schema, otherwise the append operation
> fails with incompatible schema.
> * The 'avro append' mode should only work when compression type is set to
> 'none', if any other compression type is selected in 'avro append' mode the
> user should get a validation error.
> The changes will have to be added to *support/nifi-1.x* branch also.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
