Re: s3a staging committer(directory committer )not writing data to s3 bucket (final output directory) in spark3

Hi Shiva, *This works on 3.0.1 on prem* but not on Google dataproc with spark 3.1.1-RC2 These are the jar files used for structured streaming All added under $SPARK_HOME/jars on all nodes spark-sql-kafka-0-10_2.12-3.0.1.jar kafka-clients-2.7.0.jar spark-token-provider-kafka-0-10_2.12-3.0.1.jar

Re: s3a staging committer(directory committer )not writing data to s3 bucket (final output directory) in spark3

Hi Mich Talebzadeh, Could you please share the spark configuration used to run the job? you mentioned it works on 3.0.1 I will check if I am also using the same configuration or not. Regards, Shiva -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

Re: s3a staging committer(directory committer )not writing data to s3 bucket (final output directory) in spark3

Hi, We also have an issue with data not being displayed in Google Cloud DataProc 2 that uses Spark 3.1.1. It works on 3.0.1 on Prem but not on 3.1.1 on Google Data Proc (offered as a service). It may be related to Spark version It is concerning. HTH LinkedIn *

Re: s3a staging committer(directory committer )not writing data to s3 bucket (final output directory) in spark3

Hi Mich Talebzadeh, Thanks for your reply, the issue is seen in spark 3.0.0 and with spark 2.4.5 it works without any problem. Regards, Shiva -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe

Re: s3a staging committer(directory committer )not writing data to s3 bucket (final output directory) in spark3

Hi, What exact version of spark is it? HTH LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * *Disclaimer:* Use it at your own risk. Any and all responsibility for

Re: s3a staging committer(directory committer )not writing data to s3 bucket (final output directory) in spark3

Any suggestions or help is greatly appreciated! -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

s3a staging committer(directory committer )not writing data to s3 bucket (final output directory) in spark3

Hi, I'm running spark3 on Kubernetes and using S3A staging committer (directory committer) to write data to s3 bucket. The same set up works fine with spark2 but with spark3 the final data (writing in parquet format) is not visible in s3 bucket and when read operation is performed on that parquet

s3a staging committer (directory committer) not writing data to s3 bucket (final output directory) in spark3

Hi, I'm running spark3 on Kubernetes and using S3A staging committer (directory committer) to write data to s3 bucket. The same set up works fine with spark 2.4.5 but with spark3 the final data (writing in parquet format) is not visible in s3 bucket and when read operation is performed on that