Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/14568
Not super. I will try to explain why.
`monotonically_increasing_id` constructs an id based on an increasing
offset (the lower 33 bits) and the partition id (the upper 31 bits). The
example given is not super realistic because it only seems to contain the one
partition (id=0). In this case maximum id of the previous run would be usable
as the seed.
If you have more than one partition an issue emerges, for instance
`range(0, 9, 1, 3).select(monotonically_increasing_id()).show` yields the
following ids:
```
+-----------------------------+
|monotonically_increasing_id()|
+-----------------------------+
| 0|
| 1|
| 2|
| 3|
| 4|
| 8589934592|
| 8589934593|
| 8589934594|
| 8589934595|
| 8589934596|
+-----------------------------+
```
Which offset for the next run would you use in this case? Lets explore the
the options we have:
- Taking the maximum offset. This will probably lead to collisions between
ids.
- Calculating the maximum of the lower 33 bits. This relies heavily on the
exact implementation of `monotonically_increasing_id`.
What would you recommend an end user to do?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
