[GitHub] nifi issue #3057: NIFI-5667: Add nested record support for PutORC

2018-10-15 Thread bbende
Github user bbende commented on the issue: https://github.com/apache/nifi/pull/3057 Looks good to me, was able to verify the functionality, going to merge ---

[GitHub] nifi issue #3057: NIFI-5667: Add nested record support for PutORC

2018-10-12 Thread mattyb149
Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/3057 Updated the rest of the backticks, thanks @VikingK and @bbende for your reviews! ---

[GitHub] nifi issue #3057: NIFI-5667: Add nested record support for PutORC

2018-10-12 Thread VikingK
Github user VikingK commented on the issue: https://github.com/apache/nifi/pull/3057 @mattyb149 I checked out the new backtick ddl patch you applied. Its seems it only backticks the complex data structures, the "top" level ones are for example not ticked. For example ProductId,Ite

[GitHub] nifi issue #3057: NIFI-5667: Add nested record support for PutORC

2018-10-11 Thread VikingK
Github user VikingK commented on the issue: https://github.com/apache/nifi/pull/3057 @mattyb149 asome work, I was gonna suggest the backtick feature since its a pain when downstream systems sends all kinds of weird names, like Timestamp etc. My solution right now is an ugly groovy scr

[GitHub] nifi issue #3057: NIFI-5667: Add nested record support for PutORC

2018-10-11 Thread mattyb149
Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/3057 I found the issue and pushed a new commit with the fix. Also your test data exercised another part of the code I hadn't thought of, your "as" field is a keyword in Hive so when I used the generated

[GitHub] nifi issue #3057: NIFI-5667: Add nested record support for PutORC

2018-10-11 Thread bbende
Github user bbende commented on the issue: https://github.com/apache/nifi/pull/3057 Ok using your test.avro going into PutORC with an AvroReader that uses embedded schema I can get the error you showed earlier. I also tested ConvertRecord using the AvroReader and JsonWrtier a

[GitHub] nifi issue #3057: NIFI-5667: Add nested record support for PutORC

2018-10-11 Thread VikingK
Github user VikingK commented on the issue: https://github.com/apache/nifi/pull/3057 Maybe it's because I am missing a name: on the array level that's causing the Jason to pick 'array'. I'll test later tonight. ---

[GitHub] nifi issue #3057: NIFI-5667: Add nested record support for PutORC

2018-10-11 Thread VikingK
Github user VikingK commented on the issue: https://github.com/apache/nifi/pull/3057 @bbende wierd, I tried it again and I got the same error, here is the output from avro tools and I also attached my test.avro message ``` java -jar avro-tools-1.8.2.jar getschema test.avro

[GitHub] nifi issue #3057: NIFI-5667: Add nested record support for PutORC

2018-10-11 Thread bbende
Github user bbende commented on the issue: https://github.com/apache/nifi/pull/3057 @VikingK in your schema it has OItems defined as an array, but then in the JSON OItems is not an array, its an object with a field called array. So running with that schema and example JSON I get:

[GitHub] nifi issue #3057: NIFI-5667: Add nested record support for PutORC

2018-10-10 Thread VikingK
Github user VikingK commented on the issue: https://github.com/apache/nifi/pull/3057 Sorry for the late answer got bogged down. Avro: ``` { "name": "IOlist", "namespace": "analytics.models.its", "type": "record", "fields": [ { "

[GitHub] nifi issue #3057: NIFI-5667: Add nested record support for PutORC

2018-10-10 Thread mattyb149
Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/3057 Can you share the schema of the data, and possibly a JSON export of your Avro file? I couldn't reproduce this with an array of ints, and @bbende ran successfully with an array of records. ---

[GitHub] nifi issue #3057: NIFI-5667: Add nested record support for PutORC

2018-10-10 Thread VikingK
Github user VikingK commented on the issue: https://github.com/apache/nifi/pull/3057 Hi, I have been verifying the pull request and I came across this error for one of my Avro messages, I am currently working to figure out which part of the message caused this. ``` 201