steveloughran commented on issue #24934: [SPARK-28124] [SS] SQS source for Structured Streaming URL: https://github.com/apache/spark/pull/24934#issuecomment-517852694 FWIW, thinking about this, an interesting way to do this (for S3A only), is for us to provide a way there to subscribe to SQS events on an s3a instance, e.g ```scala val fs = bucketPath.getFileSystem(conf).asInstanceOf[S3AFileSystem] val sqs = fs.subscribeToS3Events() ``` Good: * makes it trivial to implement the subscriber * lets s3a code handle all the existing proxy/connection/auth setup grief * lets us right some tests for it in S3AFS ITest suites so you can be confident we don't break things Con * depends on a new release of Hadoop * doesn't work with AWS EMRFS until they implement the same API. * potentially less flexible for changes/fixes ``` I think for *me*, it'd work, but for you, not worth the complexity. And unless I can see a way to provide an event stream API which could be used across filesystems (e.g. implement an abfds and gcs equivalents; hdfs already has fadvise), you don't gain by having it added there. Sorry
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
