[
https://issues.apache.org/jira/browse/NIFI-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15125343#comment-15125343
]
ASF GitHub Bot commented on NIFI-1107:
--------------------------------------
Github user trkurc commented on a diff in the pull request:
https://github.com/apache/nifi/pull/192#discussion_r51360885
--- Diff:
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/PutS3Object.java
---
@@ -118,6 +210,75 @@ protected PropertyDescriptor
getSupportedDynamicPropertyDescriptor(final String
.build();
}
+ protected File getPersistenceFile() {
+ return new File(PERSISTENCE_ROOT + getIdentifier());
+ }
+
+ protected synchronized MultipartState getLocalState(final String
s3ObjectKey) throws IOException {
+ // get local state if it exists
+ MultipartState currState = null;
+ final File persistenceFile = getPersistenceFile();
+ if (persistenceFile.exists()) {
+ try (final FileInputStream fis = new
FileInputStream(persistenceFile)) {
+ final Properties props = new Properties();
+ props.load(fis);
+ if (props.containsKey(s3ObjectKey)) {
+ final String localSerialState =
props.getProperty(s3ObjectKey);
+ if (localSerialState != null) {
+ currState = new MultipartState(localSerialState);
+ getLogger().info("Local state for {} loaded with
uploadId {} and {} partETags",
+ new Object[]{s3ObjectKey,
currState.getUploadId(), currState.getPartETags().size()});
+ }
+ }
+ } catch (IOException ioe) {
+ getLogger().warn("Failed to recover local state for {} due
to {}. Assuming no local state and " +
+ "restarting upload.", new Object[]{s3ObjectKey,
ioe.getMessage()});
+ }
+ }
+ return currState;
+ }
+
+ protected synchronized void persistLocalState(final String
s3ObjectKey, final MultipartState currState) throws IOException {
+ final String currStateStr = (currState == null) ? null :
currState.toString();
+ final File persistenceFile = getPersistenceFile();
+ final File parentDir = persistenceFile.getParentFile();
+ if (!parentDir.exists() && !parentDir.mkdirs()) {
+ throw new IOException("Persistence directory (" +
parentDir.getAbsolutePath() + ") does not exist and " +
+ "could not be created.");
+ }
+ final Properties props = new Properties();
+ if (persistenceFile.exists()) {
+ try (final FileInputStream fis = new
FileInputStream(persistenceFile)) {
+ props.load(fis);
+ }
+ }
+ if (currStateStr != null) {
+ props.setProperty(s3ObjectKey, currStateStr);
+ } else {
+ props.remove(s3ObjectKey);
+ }
+
+ if (props.size() > 0) {
+ try (final FileOutputStream fos = new
FileOutputStream(persistenceFile)) {
+ props.store(fos, null);
+ } catch (IOException ioe) {
+ getLogger().error("Could not store state {} due to {}.",
+ new Object[]{persistenceFile.getAbsolutePath(),
ioe.getMessage()});
+ }
+ } else {
+ try {
+ Files.delete(persistenceFile.toPath());
+ } catch (IOException ioe) {
+ getLogger().error("Could not remove state file {} due to
{}.",
+ new Object[]{persistenceFile.getAbsolutePath(),
ioe.getMessage()});
--- End diff --
So, on a clean flow, just writing small files, I get these error messages,
which aren't particularly helpful and show up on the processor bulletin,
despite sending to success. Not ideal behavior
```
2016-01-31 08:59:38,680 ERROR [Timer-Driven Process Thread-4]
o.a.nifi.processors.aws.s3.PutS3Object
PutS3Object[id=9460c94b-72b4-4c39-89a6-2d96f2e1ae81] Could not remove state
file
C:\development\nifi\nifi-assembly\target\nifi-0.4.2-SNAPSHOT-bin\nifi-0.4.2-SNAPSHOT\conf\state\9460c94b-72b4-4c39-89a6-2d96f2e1ae81
due to conf\state\9460c94b-72b4-4c39-89a6-2d96f2e1ae81.
```
> Create new PutS3ObjectMultipart processor
> -----------------------------------------
>
> Key: NIFI-1107
> URL: https://issues.apache.org/jira/browse/NIFI-1107
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
> Reporter: Joe Skora
> Assignee: Joe Skora
> Labels: s3
> Fix For: 0.5.0
>
>
> A new `PutS3ObjectMultipart` processor using the AWS S3 API to upload files
> larger than those supported by `PutS3Object` which has a [5GB
> limit|http://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html]
> limit.
> To support S3 compatible endpoints this will also add an `Endpoint Override
> URL` property to `AbstractAWSProcessor` to set the service
> [endpoint|http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/AmazonWebServiceClient.html#setEndpoint(java.lang.String)]
> to override the endpoint URL normally selected based on the the Amazon
> region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
