Hi Matt, Thank you for the quick response. I was not aware of the hive3 nar. Is there a list of these NARs that are released but not included in the default distribution? A quick search didn't turn up anything.
Thanks, Ben On Thu, Mar 26, 2020 at 12:36 PM Matt Burgess <[email protected]> wrote: > Ben, > > NiFi doesn't currently have components that support Hive 2, although > some work has been done on that front [1]. However there are Hive 3 > components including a PutORC processor that is kind of a mashup > between the old Hive 1-based ConvertAvroToORC and PutHDFS/PutParquet, > it does the conversion from any format (using a supplied Reader) and > writes it out to the specified location. The choice was made to have > the conversion and writing done at the same time as you can't do much > with ORC in NiFi (transformations, e.g.). > > PutORC is part of the Hive 3 NAR, which due to size constraints is not > included with the Apache NiFi distribution. However we do build and > publish the NAR for each release, so you can download it separately > and drop it into your lib/ folder. You can find the Hive 3 NAR for the > latest release at [2]. > > If you have a use case for which PutORC is not sufficient, please > explain and/or vote for the Jira case that covers adding ORC > reader/writer services [3]. > > Regards, > Matt > > [1] https://issues.apache.org/jira/browse/NIFI-6456 > [2] > https://repository.apache.org/content/repositories/releases/org/apache/nifi/nifi-hive3-nar/1.11.4/ > [3] https://issues.apache.org/jira/browse/NIFI-4186 > > On Thu, Mar 26, 2020 at 12:23 PM Benjamin Garrett > <[email protected]> wrote: > > > > Hi, > > > > Regarding ConvertAvroToORC, I found this link which describes > incompatibilities with later version of hive. > https://community.cloudera.com/t5/Support-Questions/hive-2-3-return-NULLS-using-NIFI-convertAvroToORC/td-p/175279 > > > > Are there plans to support newer versions of hive with orc output from > Nifi? > > > > I couldn't find any jira ticket related to this topic. I did find this > open ticket regarding the RecordPath/RecordReader pattern and orc: > https://issues.apache.org/jira/browse/NIFI-4186 > > > > Thank you! > > Ben >
