jdattani commented on issue #4086: URL: https://github.com/apache/hudi/issues/4086#issuecomment-998953492
> @jdattani : Do you know if this happens w/ datasource or deltastreamer directly? I could not reproduce this issue unless its very use-case specific. But general errors, my console or driver logs does print the errors. (spark-shell and spark-submit) This happened for us w/ datasource. Haven't checked through deltastreamer. Was able to build hudi after enabling additional logging in the snippet originally shared (ws.getErrors) and found the exception to be "ArrayIndexOutOfBounds". It seems to be a specific use-case we have where we were getting an array field input as a string value that we were trying to cast to another datatype. This array could have variable number of elements for each row. And that somehow was causing an error at the time of writing to the hudi table. However, the issue here seems to be in error propagation. Unless I'm missing something, I was expecting the job to fail instead of silently logging "UPSERT failed with errors" and continuing to finish Glue job execution successfully. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
