[
https://issues.apache.org/jira/browse/NIFI-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15328084#comment-15328084
]
Daniel Cave commented on NIFI-1935:
-----------------------------------
1) Thursday it was NiFi-1987 that broke a fresh master build on Windows due to
the deletion of a Cloudera repository on org.sonatype on 6/7 that was a deep
sub-dependency in nifi-spark-receivers (it was deep under avro:avro.mapred
1.7). This mysteriously fixed itself and was not repeatable on 6/10 on a fresh
Linux build (0.x or master).
2) On our 0.x Linux server (with NiFi-1933/1934/1935 merged in, which had been
developed against master since they started before the 0.x/master split) I was
seeing old issues such as AttributesToJson failing to convert schema as
attribute due to not being a string (InferAvroSchema writing it as avro.binary
to attribute), and EvaluateJsonPath not properly interpretting the avro.binary,
and other issues. After trying the fresh 0.x version without the master based
changes being merged in the attribute type changed to string and everything
worked again.
3) Flow is OOB GetFile->SplitJSON->InverAvroSchema->ConvertAvroSchema->PutFile
(attached). Today, on fresh Linux server with fresh 0.x and master builds,
with a higher throughput I was seeing larger strings passing through
ConvertJsonToAvro in 0.x ok but in master they were changing formed and were
being stripped of whitespaces. At lower throughput I can't repeat the issue,
and since our update to fix the filenames in SplitJson can't be added in (due
to issue 2), I can't revalidate since all the filenames are the same.
Basically, I need to know what branch to use as a true master branch so that I
can have builds and make needed changes and have consistent results. Having
essentially two masters is causing gremlins that are affecting my customer
solutions.
> Added ConvertDynamicJsonToAvro processor
> ----------------------------------------
>
> Key: NIFI-1935
> URL: https://issues.apache.org/jira/browse/NIFI-1935
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
> Affects Versions: 1.0.0, 0.7.0
> Reporter: Daniel Cave
> Assignee: Alex Halldin
> Priority: Minor
> Fix For: 1.0.0, 0.7.0
>
> Attachments:
> 0001-NIFI-1935-Added-ConvertDynamicJSONToAvro.java.-Added.patch,
> GetFileSplitInferConvertAvro.xml
>
>
> ConvertJsonToAvro required a predefined Avro schema to convert JSON and
> required the presence of all field on the incoming JSON.
> ConvertDynamicJsonToAvro functions similarly, however it now accepts the JSON
> and schema as incoming flowfiles and creates the Avro dynamically.
> This processor requires the InferAvroSchema processor in its upstream flow so
> that it can use the original and schema flowfiles as input. These two
> flowfiles will have the unique attribute inferredAvroId set on them by
> InferAvroSchema so that they can be properly matched in
> ConvertDynamicJsonToAvro.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
