pralabhkumar commented on a change in pull request #35191:
URL: https://github.com/apache/spark/pull/35191#discussion_r799222433
##########
File path: python/pyspark/pandas/series.py
##########
@@ -5228,22 +5228,128 @@ def asof(self, where: Union[Any, List]) ->
Union[Scalar, "Series"]:
where = [where]
index_scol = self._internal.index_spark_columns[0]
index_type = self._internal.spark_type_for(index_scol)
+
+ # e.g where = [10, 20]
+ # In the comments below , will explain how the dataframe will look
after transformations.
+ # e.g pd.Series([2, 1, np.nan, 4], index=[10, 20, 30, 40],
name="Koalas")
+
+ column_prefix_constant = "col_"
cond = [
- F.max(F.when(index_scol <= SF.lit(index).cast(index_type),
self.spark.column))
- for index in where
+ F.when(
+ index_scol <= SF.lit(index).cast(index_type),
+ F.struct(
+ F.lit(column_prefix_constant + str(index) + "_" +
str(idx)).alias("identifier"),
+ self.spark.column.alias("col_value"),
+ ),
+ ).alias(column_prefix_constant + str(index) + "_" + str(idx))
Review comment:
Yes @itholic , this is working (since __index_level_0__) is sorted.
However , test case with psser.asof([25, 25]) is failing , ambiguous of
duplicate cols in psdf = ps.DataFrame(sdf) . Therefore , in order to pass
above test case ,
below is the change.
```python
cond = [
F.last(
F.when(index_scol <= SF.lit(index).cast(index_type),
self.spark.column),
ignorenulls=True,
).alias(column_prefix_constant + str(index) + "_" + str(idx))
for idx, index in enumerate(where)
]
```
Then
```python
with ps.option_context("compute.default_index_type", "distributed",
"compute.max_rows", 1):
psdf = ps.DataFrame(sdf) # type: DataFrame
df = pd.DataFrame(psdf.transpose().values, columns=[self.name],
index=where)
return df[df.columns[0]]
```
Please let me know , if its ok , i'll update the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
