Chuck Connell created SPARK-37187:
-------------------------------------
Summary: pyspark.pandas fails to create a histogram of one column
from a large DataFrame
Key: SPARK-37187
URL: https://issues.apache.org/jira/browse/SPARK-37187
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 3.2.0
Reporter: Chuck Connell
When trying to create a histogram from one column of a large DataFrame,
pyspark.pandas fails. So this line
{quote}DF.plot.hist(column="FullVaxPer100", bins=20) # there are many other
columns
{quote}
yields this error
{quote}cannot resolve 'least(min(EndDate), min(EndDeaths), min(`STATE-COUNTY`),
min(StartDate), min(StartDeaths), min(POPESTIMATE2020), min(ST_ABBR),
min(VaxStartDate), min(Series_Complete_Yes_Start),
min(Administered_Dose1_Recip_Start), min(VaxEndDate),
min(Series_Complete_Yes_End), min(Administered_Dose1_Recip_End), min(Deaths),
min(Series_Complete_Yes_Mid), min(Administered_Dose1_Recip_Mid),
min(FullVaxPer100), min(OnePlusVaxPer100), min(DeathsPer100k))' due to data
type mismatch: The expressions should all have the same type, got
LEAST(timestamp, bigint, string, timestamp, bigint, bigint, string, timestamp,
bigint, bigint, timestamp, bigint, bigint, bigint, double, double, double,
double, double).;
{quote}
The odd thing is that pyspark.pandas seems to be operating on all the columns
when only one is needed.
As a workaround, you can first create a one-column DataFrame that selects just
the field you want, then make a histogram of that. But the command above should
work also.
I can supply the complete program and datasets that demonstrate the error.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
