[
https://issues.apache.org/jira/browse/NIFI-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15456739#comment-15456739
]
ASF GitHub Bot commented on NIFI-2631:
--------------------------------------
Github user jvwing commented on the issue:
https://github.com/apache/nifi/pull/917
Build with contrib-check passes OK. The TravisCI failure was from memory
("GC overhead limit exceeded"), not code issues. All of my manual testing went
well, both with and without versioning. The code quality looks very good.
One small thing -- the integration test
ITListS3::testGetPropertyDescriptors fails due to the new property descriptors.
Would you please update that test? Also, what would you think about a test or
two for the versioning feature? Even if you copy/pasted one of the existing
integration tests and added `runner.setProperty(ListS3.USE_VERSIONS, "true");`,
running it against the non-version-enabled bucket, it would still exercise the
S3VersionBucketLister code paths.
> ListS3 improvements: "Use versions" and "Commit mode"
> -----------------------------------------------------
>
> Key: NIFI-2631
> URL: https://issues.apache.org/jira/browse/NIFI-2631
> Project: Apache NiFi
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Joseph Gresock
> Assignee: Joseph Gresock
> Priority: Minor
> Fix For: 1.1.0, 0.8.0
>
>
> Our team needs to be able to list individual versions in S3. We also ran
> into a use case where a bucket with many objects (over 1 million in our case)
> seemed to cause ListS3 to run forever. The S3 list command finished in a few
> minutes, but we believe it was taking a very long time for NiFi to commit all
> the flow files at once.
> To handle this use case, we added a Commit Mode property to ListS3 that
> allows you specify that you want to commit "Per page" vs. "Once". This has
> proven to correctly emit the flow files as the S3 paging progresses.
> We also implemented support for S3 List Versions, which includes the
> "s3.version" and "s3.isLatest" attributes if applicable. The "s3.version"
> attribute can in turn be used in the FetchS3 processor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
