Github user trkurc commented on a diff in the pull request:
https://github.com/apache/nifi/pull/192#discussion_r51360885
--- Diff:
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/PutS3Object.java
---
@@ -118,6 +210,75 @@ protected PropertyDescriptor
getSupportedDynamicPropertyDescriptor(final String
.build();
}
+ protected File getPersistenceFile() {
+ return new File(PERSISTENCE_ROOT + getIdentifier());
+ }
+
+ protected synchronized MultipartState getLocalState(final String
s3ObjectKey) throws IOException {
+ // get local state if it exists
+ MultipartState currState = null;
+ final File persistenceFile = getPersistenceFile();
+ if (persistenceFile.exists()) {
+ try (final FileInputStream fis = new
FileInputStream(persistenceFile)) {
+ final Properties props = new Properties();
+ props.load(fis);
+ if (props.containsKey(s3ObjectKey)) {
+ final String localSerialState =
props.getProperty(s3ObjectKey);
+ if (localSerialState != null) {
+ currState = new MultipartState(localSerialState);
+ getLogger().info("Local state for {} loaded with
uploadId {} and {} partETags",
+ new Object[]{s3ObjectKey,
currState.getUploadId(), currState.getPartETags().size()});
+ }
+ }
+ } catch (IOException ioe) {
+ getLogger().warn("Failed to recover local state for {} due
to {}. Assuming no local state and " +
+ "restarting upload.", new Object[]{s3ObjectKey,
ioe.getMessage()});
+ }
+ }
+ return currState;
+ }
+
+ protected synchronized void persistLocalState(final String
s3ObjectKey, final MultipartState currState) throws IOException {
+ final String currStateStr = (currState == null) ? null :
currState.toString();
+ final File persistenceFile = getPersistenceFile();
+ final File parentDir = persistenceFile.getParentFile();
+ if (!parentDir.exists() && !parentDir.mkdirs()) {
+ throw new IOException("Persistence directory (" +
parentDir.getAbsolutePath() + ") does not exist and " +
+ "could not be created.");
+ }
+ final Properties props = new Properties();
+ if (persistenceFile.exists()) {
+ try (final FileInputStream fis = new
FileInputStream(persistenceFile)) {
+ props.load(fis);
+ }
+ }
+ if (currStateStr != null) {
+ props.setProperty(s3ObjectKey, currStateStr);
+ } else {
+ props.remove(s3ObjectKey);
+ }
+
+ if (props.size() > 0) {
+ try (final FileOutputStream fos = new
FileOutputStream(persistenceFile)) {
+ props.store(fos, null);
+ } catch (IOException ioe) {
+ getLogger().error("Could not store state {} due to {}.",
+ new Object[]{persistenceFile.getAbsolutePath(),
ioe.getMessage()});
+ }
+ } else {
+ try {
+ Files.delete(persistenceFile.toPath());
+ } catch (IOException ioe) {
+ getLogger().error("Could not remove state file {} due to
{}.",
+ new Object[]{persistenceFile.getAbsolutePath(),
ioe.getMessage()});
--- End diff --
So, on a clean flow, just writing small files, I get these error messages,
which aren't particularly helpful and show up on the processor bulletin,
despite sending to success. Not ideal behavior
```
2016-01-31 08:59:38,680 ERROR [Timer-Driven Process Thread-4]
o.a.nifi.processors.aws.s3.PutS3Object
PutS3Object[id=9460c94b-72b4-4c39-89a6-2d96f2e1ae81] Could not remove state
file
C:\development\nifi\nifi-assembly\target\nifi-0.4.2-SNAPSHOT-bin\nifi-0.4.2-SNAPSHOT\conf\state\9460c94b-72b4-4c39-89a6-2d96f2e1ae81
due to conf\state\9460c94b-72b4-4c39-89a6-2d96f2e1ae81.
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
