Jason Altekruse created SPARK-58453:
---------------------------------------
Summary: Improve show() output to include row count if not
truncated
Key: SPARK-58453
URL: https://issues.apache.org/jira/browse/SPARK-58453
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.2.0
Reporter: Jason Altekruse
Very often row counts are a useful integrity check for a query result, or a
mental reference for which result set you are thinking about when running a
series of exploratory queries.
Many query clients will report row counts for these reasons. When I am querying
spark datasets in an environment like a python notebook or the spark shell cli,
I only get a report as part of the output of a call to show() if the result set
was truncated. Nothing is reported with total row count isn't truncated. Seeing
the truncation reported often leads to setting the options in the show() call
to something large like n=1000 in pyspark or numRows=1000 in scala environments.
This can get me to the state where it will stop reporting that a truncation
occurred, but if I want to know the exact number of rows that were returned I
need to run another query, or write an additional line of code to compute the
dataframe, print the number of rows and then call show() to see the data. I am
currently using spark for a use case where I'm trying to make a SQL interface
available to semi-technical users, but trying to limit the amount of python or
internal details of spark they need to know, like why they would need to call
cache() to resolve the dataframe.
It would be nice if this could just be a one-liner, and/or go as far as to make
reporting row count the default behavior of show().
{code:java}
date_error_df = spark.sql("select * from my_table where try_to_date(column) is
not null")
print(date_error_df.cache().count())
date_error_df.show(n=100, truncate=False); {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
