Hi, I think you have a typo :
root |-- ceated: string (nullable = true) and then: withColumnRenamed("created","Date Calculated"). On Mon, 4 Jan 2021 at 19:12, Lalwani, Jayesh <jlalw...@amazon.com.invalid> wrote: > You don’t have a column named “created”. The column name is “ceated”, > without the “r” > > > > *From: *Mich Talebzadeh <mich.talebza...@gmail.com> > *Date: *Monday, January 4, 2021 at 1:06 PM > *To: *"user @spark" <user@spark.apache.org> > *Subject: *[EXTERNAL] Spark DF does not rename the column > > > > *CAUTION*: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > Hi, > > > > version 2.4.3 > > > > I don't know the cause of this. > > > > This renaming of DF columns used to work fine. I did couple of changes to > spark/Scala code not relevant to this table and it refuses to rename the > columns for a table!. > > > > val summaryACC = HiveContext.table("summaryACC") > > > > summaryACC.printSchema() > > > > root > > |-- ceated: string (nullable = true) > > |-- hashtag: string (nullable = true) > > |-- paid: float (nullable = true) > > |-- received: float (nullable = true) > > > > summaryACC. > > orderBy(desc("paid"),desc("received")). > > withColumnRenamed("created","Date Calculated"). > > withColumnRenamed("hashtag","Who"). > > withColumn(("received"),format_number(col("received"),2)). > > withColumn(("paid"),format_number(col("paid"),2)). > > withColumnRenamed("paid","paid out/GBP"). > > withColumnRenamed("received","paid in/GBP"). > > withColumn("paid in/GBP",when(col("paid in/GBP") === > "0.00","--").otherwise(col("paid in/GBP"))). > > withColumn("paid out/GBP",when(col("paid out/GBP") === > "0.00","--").otherwise(col("paid out/GBP"))). > > select("Date Calculated","Who","paid in/GBP","paid > out/GBP").show(1000,false) > > > > and this is the error > > > > org.apache.spark.sql.AnalysisException: cannot resolve '`Date Calculated`' > given input columns: [alayer.summaryacc.ceated, Who, paid out/GBP, paid > in/GBP];; > > > > This used to work before! > > > > +----------------------------+------------------+-----------+------------+ > > |Date Calculated |Who |paid in/GBP|paid out/GBP| > > +----------------------------+------------------+-----------+------------+ > > |Mon Jan 04 14:22:17 GMT 2021|paypal |579.98 |1,526.86 | > > > > Appreciate any ideas. > > > > Thanks, Mich > > >