Looks like what you want is to add a column that, when ordered by that
column, the current order of the dateframe is preserved.
All you need is the monotonically_increasing_id() function:
spark.range(0, 10, 1, 5).withColumn("row",
monotonically_increasing_id()).show()
+---+-----------+
| id| row|
+---+-----------+
| 0| 0|
| 1| 1|
| 2| 8589934592|
| 3| 8589934593|
| 4|17179869184|
| 5|17179869185|
| 6|25769803776|
| 7|25769803777|
| 8|34359738368|
| 9|34359738369|
+---+-----------+
Within a partition, all columns have consecutive row numbers, the start
of a new partition observes a jump in the row number. The example above
has 5 partitions with 2 rows each.
In case you need a global consecutive row number (not needed to preserve
current dataframe order as you want it), you can use the
Dataframe.with_row_numbers() method provided by the Spark-Extension
package:
https://github.com/G-Research/spark-extension/blob/master/ROW_NUMBER.md
import gresearch.spark
df.with_row_numbers().show()
+---+----------+
| id|row_number|
+---+----------+
| 1| 1|
| 2| 2|
| 2| 3|
| 3| 4|
+---+----------+
Cheers,
Enrico
Am 05.12.23 um 18:25 schrieb Som Lima:
want to maintain the order of the rows in the data frame in Pyspark.
Is there any way to achieve this for this function here we have the
row ID which will give numbering to each row. Currently, the below
function results in the rearrangement of the row in the data frame.
|def createRowIdColumn(new_column, position, start_value): row_count =
df.count() row_ids = spark.range(int(start_value), int(start_value) +
row_count, 1).toDF(new_column) window = Window.orderBy(lit(1))
df_row_ids = row_ids.withColumn("row_num", row_number().over(window) -
1) df_with_row_num = df.withColumn("row_num",
row_number().over(window) - 1) if position == "Last Column": result =
df_with_row_num.join(df_row_ids, on="row_num").drop("row_num") else:
result = df_row_ids.join(df_with_row_num,
on="row_num").drop("row_num") return result.orderBy(new_column) |
Please let me know the solution if we can achieve this requirement.
