Tim Robertson created SPARK-45855: ------------------------------------- Summary: Unable to set codec for Hive CTAS Key: SPARK-45855 URL: https://issues.apache.org/jira/browse/SPARK-45855 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 3.4.0 Environment: Spark 3.4.0 Stackable.tech release 23.7.0 which runs spark on K8s. Reporter: Tim Robertson
Hi, We've discovered code that worked in Spark 3.3.0 doesn't in 3.4.0. I can't find anything in the release notes to indicate why, so I wonder if this is a bug. Thank you for looking. Here we're using our own custom codec, but we noticed we can't set gzip either. {{ SparkConf conf = spark.sparkContext().conf();}} {{ conf.set("hive.exec.compress.output", "true");}} {{ conf.set("mapred.output.compression.codec", D2Codec.class.getName()); }} {{ spark.sql("CREATE TABLE b AS SELECT id FROM a");}} Any advice is appreciated and I can help run tests. We run Spark on K8S using the stackable.tech distribution. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org