GitHub user Susmit07 added a comment to the discussion: S3 multipart upload for parquet
Several advantages of using S3.multipartUpload & S3.putObject ------------------------ - Both the APIs support all or nothing, no dangling file visible if Process restarts - For large files we can use Multipart Upload, and can resume upload if we can keep track of the last upload eTag - Support for any file type. - AvroParquetWriter can also work but its confined to Parquet files only, and need hadoop-aws library dependency to uplaod to s3. (added overhead) GitHub link: https://github.com/apache/pekko-connectors/discussions/870#discussioncomment-10996668 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
