DmitryGrb commented on a change in pull request #97:
URL: https://github.com/apache/bahir/pull/97#discussion_r463750572
##########
File path:
sql-streaming-sqs/src/main/scala/org/apache/spark/sql/streaming/sqs/SqsClient.scala
##########
@@ -131,13 +131,24 @@ class SqsClient(sourceOptions: SqsSourceOptions,
}
}
+ private def extractS3Message(parsedBody: JValue): JValue = {
+ implicit val formats = DefaultFormats
+ sourceOptions.messageWrapper match {
+ case sourceOptions.S3MessageWrapper.None => parsedBody
+ case sourceOptions.S3MessageWrapper.SNS => parse((parsedBody \
"Message").extract[String])
Review comment:
Right now if we fail to parse SQS we just remove these messages from
SQS. I think what would make sense is try to refetch them again. And agree,
this should be done in another PR. I can raise a ticket and make these changes.
Thoughts/Objections?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
