Vamsi-klu commented on code in PR #58191:
URL: https://github.com/apache/spark/pull/58191#discussion_r3849361991
##########
python/pyspark/sql/datasource_internal.py:
##########
@@ -162,11 +162,10 @@ def getCache(self, start: dict, end: dict) ->
Optional[Iterator[Tuple]]:
it = chain(*entries)
return it
- def read(
- self,
- input_partition: SimpleInputPartition, # type: ignore[override]
- ) -> Iterator[Tuple]:
- return self.simple_reader.readBetweenOffsets(input_partition.start,
input_partition.end)
+ def read(self, partition: InputPartition) -> Iterator[Tuple]:
+ # partitions() only yields SimpleInputPartition. Cast keeps the ABC
override valid.
+ simple_partition = cast(SimpleInputPartition, partition)
Review Comment:
cast does not raise. I ran typing.cast(SimpleInputPartition,
InputPartition(1)) and got an InputPartition back. No exception, no message.
So the old revision was worse than isinstance. A bad partition failed later
on .start, or did not fail at all.
I stopped making SimpleInputPartition an InputPartition. partitions() now
returns InputPartition(SimpleInputPartition(start, end)). read() takes
InputPartition like the ABC. A wrong value raises DATA_SOURCE_TYPE_MISMATCH
with the type we actually got.
I do not want the old SimpleInputPartition parameter back. That only made
mypy happy by lying about the override.
cc @gaogaotiantian
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
