[
https://issues.apache.org/jira/browse/SPARK-38483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18129858#comment-18129858
]
Preet Mehta commented on SPARK-38483:
-------------------------------------
Hi, I'd like to work on this issue. Before writing a PR, I want to confirm the
minimal approach matches what maintainers have in mind, since the code layout
has changed since this was reported (Column is now a dispatch stub in
`pyspark/sql/column.py` with concrete implementations in `classic/column.py`
and `connect/column.py`).
What I have in mind, following the reporter's minimal design:
1. Track a private `Column._alias`: set on the column returned by `alias()`
when a single alias is given (`name()` delegates to / is `alias()` in both
implementations, so it is covered). Multiple aliases leave it unset. No change
to the public behavior of `alias()`/`name()`.
2. Add a private `Column._name` property that returns, in order:
- `_alias`, if set;
- the field name, if the column came from struct-field access via attribute
access (`df.s.a`) or `getField("a")` — those methods would record the field
name on the returned column, since a selected struct field takes the field
name, whereas its string form is e.g. `s['a']`;
- otherwise, the string form of the underlying expression, i.e. what
`Column.__repr__` renders today (Classic: `_jc.toString()`; Connect: the
expression repr).
I am aware of the limitation Hyukjin raised earlier in this issue: the real
output name is only known after the analyzer resolves the column, so `_name`
can only be a best-effort syntactic name — it can differ from the resolved name
under joins, shadowing, case-sensitivity rules, and similar. The proposal is to
document that limitation plainly and cover the common cases (plain, aliased,
struct field, expression) with tests in `test_column.py`'s shared mixin so both
Classic and Connect are covered.
Two questions before I proceed:
1. Is this minimal syntactic approach acceptable, with the limitation
documented — or would you rather see this wait for something resolution-aware?
2. For struct fields, is returning the bare field name for attribute/`getField`
access the behavior you want, or should `_name` stick to the string form
everywhere for consistency?
Happy to adjust the scope based on your feedback. If this direction looks
right, please assign the issue to me (or I can self-assign if I have the
permission) and I will open a PR.
Thanks,
Preet Mehta
> Column name or alias as an attribute of the PySpark Column class
> ----------------------------------------------------------------
>
> Key: SPARK-38483
> URL: https://issues.apache.org/jira/browse/SPARK-38483
> Project: Spark
> Issue Type: New Feature
> Components: PySpark
> Affects Versions: 3.2.1
> Reporter: Brian Schaefer
> Priority: Minor
> Labels: pull-request-available, starter
>
> Having the name of a column as an attribute of PySpark {{Column}} class
> instances can enable some convenient patterns, for example:
> Applying a function to a column and aliasing with the original name:
> {code:java}
> values = F.col("values")
> # repeating the column name as an alias
> distinct_values = F.array_distinct(values).alias("values")
> # re-using the existing column name
> distinct_values = F.array_distinct(values).alias(values._name){code}
> Checking the column name inside a custom function and applying conditional
> logic on the name:
> {code:java}
> def custom_function(col: Column) -> Column:
> if col._name == "my_column":
> return col.astype("int")
> return col.astype("string"){code}
> The proposal in this issue is to add a property {{Column.\_name}} that
> obtains the name or alias of a column in a similar way as currently done in
> the {{Column.\_\_repr\_\_}} method:
> [https://github.com/apache/spark/blob/master/python/pyspark/sql/column.py#L1062.]
> The choice of {{_name}} intentionally avoids collision with the existing
> {{Column.name}} method, which is an alias for {{{}Column.alias{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
