lordgamez commented on code in PR #1586:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1586#discussion_r1232009977
##########
extensions/aws/processors/PutS3Object.cpp:
##########
@@ -77,7 +78,31 @@ void PutS3Object::onSchedule(const
std::shared_ptr<core::ProcessContext> &contex
use_virtual_addressing_ = !*use_path_style_access;
}
+ context->getProperty(MultipartThreshold.getName(), multipart_threshold_);
+ if (multipart_threshold_ > getMaxUploadSize() || multipart_threshold_ <
getMinPartSize()) {
+ throw Exception(PROCESS_SCHEDULE_EXCEPTION, "Multipart Threshold is not
between the valid 5MB and 5GB range!");
+ }
+ logger_->log_debug("PutS3Object: Multipart Threshold %" PRIu64,
multipart_threshold_);
+ context->getProperty(MultipartPartSize.getName(), multipart_size_);
+ if (multipart_size_ > getMaxUploadSize() || multipart_size_ <
getMinPartSize()) {
+ throw Exception(PROCESS_SCHEDULE_EXCEPTION, "Multipart Part Size is not
between the valid 5MB and 5GB range!");
+ }
+ logger_->log_debug("PutS3Object: Multipart Size %" PRIu64, multipart_size_);
+
+
+ multipart_upload_ageoff_interval_ =
minifi::utils::getRequiredPropertyOrThrow<core::TimePeriodValue>(*context,
MultipartUploadAgeOffInterval.getName()).getMilliseconds();
+ logger_->log_debug("PutS3Object: Multipart Upload Ageoff Interval %" PRIu64
" ms", multipart_upload_ageoff_interval_.count());
+
+ multipart_upload_max_age_threshold_ =
minifi::utils::getRequiredPropertyOrThrow<core::TimePeriodValue>(*context,
MultipartUploadMaxAgeThreshold.getName()).getMilliseconds();
+ logger_->log_debug("PutS3Object: Multipart Upload Max Age Threshold %"
PRIu64 " ms", multipart_upload_max_age_threshold_.count());
+
fillUserMetadata(context);
+
+ std::string multipart_temp_dir;
+ context->getProperty(TemporaryDirectoryMultipartState.getName(),
multipart_temp_dir);
+
+
+ s3_wrapper_.initailizeMultipartUploadStateStorage(multipart_temp_dir,
getUUIDStr());
Review Comment:
I agree that it would be optimal to use the state manager. In its current
form however as a single transaction is tied to a single onTrigger session it
will probably take a larger effort to figure out how should commit and rollback
work with multiple states in a single session. Because of this I wouldn't make
it part of this PR.
IMO we shouldn't give up continuing failed multipart uploads after a restart
as it could be a useful feature for the user if for any reason the system is
restarted, the process is killed or maybe the MiNiFi service is restarted. I
would also like to make it configurable to avoid any directory permission
issues or just to give the user the option to decide where MiNiFi should be
allowed to write on the system. Because of this I would go with option 3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
