OK thanks for the notice on the cost point. I will check the cost calculations.
This already does have SNS enabled for another solution to this problem, but
I'm trying to use the minimal amount of different software components at this
stage of the pipeline. My prefered approach would have been t
Word of caution. Streaming from S3 is really cost prohibitive as the only
way to detect new files is to continuously spam the S3 List API.
On Tue, Sep 1, 2020 at 4:50 PM Jörn Franke wrote:
> Why don’t you get an S3 notification on SQS and do the actions from there?
>
> You will probably need to
Why don’t you get an S3 notification on SQS and do the actions from there?
You will probably need to write the content of the files to a no sql database .
Alternatively send the s3 notification to Kafka and read flink from there.
https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo
Hi,
I have a S3 bucket that is continuously written to by millions of devices.
These upload small compressed archives.
What I want to do is treat the tar gzipped (.tgz) files as a streaming source
and process each archive. The archive contains three files that each might need
to be processed.
