peterxcli commented on PR #5554: URL: https://github.com/apache/datafusion-comet/pull/5554#issuecomment-5543043113
@comphead pure spark: <img width="317" height="25" alt="image" src="https://github.com/user-attachments/assets/2f1ba2c2-6a57-42df-9ccf-9517444fd97e"; /> <img width="469" height="250" alt="image" src="https://github.com/user-attachments/assets/4d83b32b-4680-4daf-9fef-95fc22e8f5d3"; /> <img width="2173" height="530" alt="image" src="https://github.com/user-attachments/assets/21d3060c-870d-42ac-b6b6-2faf0575be2a"; /> looks like they have different shuffle read byte but I think it's because Spark uses row/serializer shuffle data; Comet uses compressed Arrow IPC, so identical bytes are neither expected nor desirable. Spark itself derives this counter from each fetched buffer’s physical size ([Spark source](https://github.com/apache/spark/blob/v4.1.2/core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala#L582-L589)) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
