carloea2 commented on code in PR #8258:
URL: https://github.com/apache/texera/pull/8258#discussion_r4059712197
##########
amber/src/main/python/core/architecture/sendsemantics/range_based_shuffle_partitioner.py:
##########
@@ -51,6 +52,8 @@ def __init__(self, partitioning:
RangeBasedShufflePartitioning):
)
def get_receiver_index(self, column_val) -> int:
+ if isinstance(column_val, float) and math.isnan(column_val):
+ column_val = 0
Review Comment:
Added finite narrowing in df2c526dc5. The -0.5 regression failed before the
change and now passes. This overlaps #8256, which remains open.
##########
amber/src/main/python/core/architecture/sendsemantics/range_based_shuffle_partitioner.py:
##########
@@ -51,6 +52,8 @@ def __init__(self, partitioning:
RangeBasedShufflePartitioning):
)
def get_receiver_index(self, column_val) -> int:
Review Comment:
I reproduced this with channels S1 to A, S2 to A, and S1 to B: two
receivers, but key 9 produces index 2. #8246 covers the divisor change
separately. I have not demonstrated that topology through the planner.
##########
amber/src/test/python/core/architecture/sendsemantics/test_partitioners.py:
##########
@@ -331,6 +331,19 @@ def test_value_in_range_routes_by_quotient(self,
partitioner):
assert partitioner.get_receiver_index(8) == 2
assert partitioner.get_receiver_index(9) == 2
+ def test_nan_key_routes_like_scala_zero(self):
Review Comment:
Added both infinity cases alongside the fractional-key tests. All 41
partitioner tests pass.
##########
amber/src/test/python/core/architecture/sendsemantics/test_partitioners.py:
##########
@@ -331,6 +331,19 @@ def test_value_in_range_routes_by_quotient(self,
partitioner):
assert partitioner.get_receiver_index(8) == 2
assert partitioner.get_receiver_index(9) == 2
+ def test_nan_key_routes_like_scala_zero(self):
Review Comment:
Keeping feat because this adds previously missing support. No automatic
release backport is intended; the description now says so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
