nickrvieira opened a new issue, #9398: URL: https://github.com/apache/hudi/issues/9398
**Issues** I'm finding a bit of trouble in double-checking if this is the expected behavior or even if I'm missing parametrizations for non-continuous pipelines (run-once) with both S3EventsSource + IncrSource The setup is the same as the [documentation](https://hudi.apache.org/blog/2021/08/23/s3-events-source/) suggests, with the removal of the `--continuous` flag. `S3EventsSource` - SQS 1. If we have an `S3EventsSource` for an SQS queue with a small `hoodie.deltastreamer.s3.source.queue.max.messages.per.batch`- it runs once and then stops, but it only runs for a single batch. Is there a way to make DeltaStreamer run for the entire queue, without having to set an enormous `max.messages.per.batch`? `S3EventsHoodieIncrSource` 1. For a first run, if the `hoodie.deltastreamer.source.hoodieincr.read_latest_on_missing_ckpt` is set to `true` - then it did read the last **changes** committed in the source event table, but not the entire metadata table. This is a bit confusing as the source (S3EventsSource) job might have run more times than the downstream S3EventsHoodieIncrSource job so it failed to capture all events/s3 objects registered in the metadata source table. What is the expected behavior there? 2. Documentation mentions a fan-out behavior for the queue but is the only way to filter bucket paths here is by using [Transformers](https://hudi.apache.org/docs/next/transforms/) and applying a filter for each path/key? (so it would be possible to have a fan-out with a single queue/S3EventsSource job) **Environment Description** * Hudi version : 0.12.1 * Spark version : 3.3.1 * Hive version : 3.1.3 * Hadoop version : 3.3.3 * SQS JAR version: 1.12.22 * Storage (HDFS/S3/GCS..) : S3 * Running on Docker? (yes/no) : no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
