Hi, I am using spark 1.4.1 and trying to copy all rows from a table in one MySQL Database to a Amazon RDS table using spark SQL.
Some columns in the source table are defined as DECIMAL type and are nullable. Others are not. When I run my spark job, val writeData = sqlContext.read.format("jdbc").option("url", >> sourceUrl).option("driver", "com.mysql.jdbc.Driver").option("dbtable", >> table).option("user", sourceUsername).option("password", >> sourcePassword).load() > > > > > > writeData.write.format("com.databricks.spark.redshift").option("url", >> String.format(targetUrl, targetUsername, targetPassword)).option("dbtable", >> table).option("tempdir", redshiftTempDir+table).mode("append").save() > > it fails with the following exception Can't translate null value for field > StructField(weight,DecimalType(5,2),false) Any insights about this exception would be very helpful.