This is where the exception occurs:
myAppDes.coalesce(1)
.write
.format("com.databricks.spark.redshift")
.option("url", redshiftURL)
.option("dbtable", redshiftTableName)
.option("forward_spark_s3_credentials", "true")
.option("tempdir", "s3a://zest-hevo-datalake/temp/data")
.mode(SaveMode.Append)
.save()I have attached the stack trace with the email. My build.sbt looks like: version := "1.0" scalaVersion := "2.11.8" val sparkVersion = "2.3.0" val hadoopVersion = "3.1.2" resolvers ++= Seq( "apache-snapshots" at "http://repository.apache.org/snapshots/";, "redshift" at "https://s3.amazonaws.com/redshift-maven-repository/release ", "redshift" at " http://redshift-maven-repository.s3-website-us-east-1.amazonaws.com/release ", "jitpack" at "https://jitpack.io"; ) libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % sparkVersion, "org.apache.spark" %% "spark-sql" % sparkVersion, "org.apache.spark" %% "spark-hive" % sparkVersion, "org.apache.hadoop" % "hadoop-aws" % hadoopVersion, "org.apache.hadoop" % "hadoop-common" % hadoopVersion, "org.apache.hadoop" % "hadoop-mapreduce-client-core" % hadoopVersion, "com.amazon.redshift" % "redshift-jdbc42-no-awssdk" % "1.2.15.1025", "com.databricks" %% "spark-redshift" % "3.0.0-preview1", "com.amazon.redshift" % "redshift-jdbc42" % "1.2.1.1001" ) dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.8" dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.8" dependencyOverrides += "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.8.8" I have checked and I can read data from the same Redshift Instance. I also have created table in redshift. (I also tried it without creating table but got the same error)
exception
Description: Binary data
--------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
