GitHub user bbende opened a pull request:
https://github.com/apache/nifi/pull/1712
NIFI-3724 - Add Put/Fetch Parquet Processors
This PR adds a new nifi-parquet-bundle with PutParquet and FetchParquet
processors. These work similar to PutHDFS and FetchHDFS, but instead read and
write Records.
While working on this I needed to reuse portions of the record
reader/writer code, and thus refactored some of the project structure which
caused many files to move around.
Summary of changes:
- Created nifi-parquet-bundle
- Created nifi-commons/nifi-record to hold domain/API related to records
- Created nifi-nar-bundles/nifi-extension-utils as a place for utility code
specific to extensions
- Moved nifi-commons/nifi-processor-utils under nifi-extension-utils
- Moved nifi-commons/nifi-hadoop-utils under nifi-extension-utils
- Create nifi-extension-utils/nifi-record-utils for utility code related
records
To test the Parquet processors you can create a core-site.xml with a local
file system and read/write parquet to local directories:
```
<configuration>
<property>
<name>fs.defaultFS</name>
<value>file:///</value>
</property>
</configuration>
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bbende/nifi parquet-bundle
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/1712.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1712
----
commit a35e5957f5ff8c47df5352b7b1a5ef494fed8633
Author: Bryan Bende <bbe...@apache.org>
Date: 2017-04-12T22:25:31Z
NIFI-3724 - Initial commit of Parquet bundle with PutParquet and
FetchParquet
- Creating nifi-records-utils to share utility code from record services
- Refactoring Parquet tests to use MockRecorderParser and MockRecordWriter
- Refactoring AbstractPutHDFSRecord to use schema access strategy
- Adding custom validate to AbstractPutHDFSRecord and adding handling of
UNION types when writing Records as Avro
- Refactoring project structure to get CS API references out of
nifi-commons, introducing nifi-extension-utils under nifi-nar-bundles
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
