Mendelson, Assaf
Cc: user
Subject: Re: DataFrame select non-existing column
The problem is that I do not know which data frames has the pass.mobile column.
I just list a HDFS directory which contain the parquet files and some files has
the column and some don't. I really don't want
structure and work on that (then you can do withColumn("mobile",...)
> instead of "pass.mobile") but this would change the schema.
>
>
> -Original Message-
> From: Kristoffer Sjögren [mailto:sto...@gmail.com]
> Sent: Saturday, November 19, 2016 4:57 PM
> To: Mendel
om]
Sent: Saturday, November 19, 2016 4:57 PM
To: Mendelson, Assaf
Cc: user
Subject: Re: DataFrame select non-existing column
Thanks. Here's my code example [1] and the printSchema() output [2].
This code still fails with the following message: "No such struct field mobile
in auction,
ng like:
>
> df.withColumn("newColName",pyspark.sql.functions.lit(None))
>
> Assaf.
> -Original Message-
> From: Kristoffer Sjögren [mailto:sto...@gmail.com]
> Sent: Friday, November 18, 2016 9:19 PM
> To: Mendelson, Assaf
> Cc: user
> Subject: Re
null (or some
> literal) as a preprocessing.
>
> -Original Message-
> From: Kristoffer Sjögren [mailto:sto...@gmail.com]
> Sent: Friday, November 18, 2016 4:32 PM
> To: user
> Subject: DataFrame select non-existing column
>
> Hi
>
> We have evolved a Data
ffer Sjögren [mailto:sto...@gmail.com]
> > Sent: Friday, November 18, 2016 4:32 PM
> > To: user
> > Subject: DataFrame select non-existing column
> >
> > Hi
> >
> > We have evolved a DataFrame by adding a few columns but cannot write
> select statements on the
) as a preprocessing.
>
> -Original Message-
> From: Kristoffer Sjögren [mailto:sto...@gmail.com]
> Sent: Friday, November 18, 2016 4:32 PM
> To: user
> Subject: DataFrame select non-existing column
>
> Hi
>
> We have evolved a DataFrame by adding a few columns but
You can always add the columns to old dataframes giving them null (or some
literal) as a preprocessing.
-Original Message-
From: Kristoffer Sjögren [mailto:sto...@gmail.com]
Sent: Friday, November 18, 2016 4:32 PM
To: user
Subject: DataFrame select non-existing column
Hi
We have
Hi
We have evolved a DataFrame by adding a few columns but cannot write
select statements on these columns for older data that doesn't have
them since they fail with a AnalysisException with message "No such
struct field".
We also tried dropping columns but this doesn't work for nested columns.
