Github user witgo commented on the issue:
https://github.com/apache/spark/pull/14995
This PR is Test only, it used to
1. verify code through CI
2. verify the effectiveness of the solution
includes two underlying API changes.
1. Replace ByteBuffer with ChunkedByteBuffer.
2. Replace ByteBuf with InputStream.
There should not be much debate about 1.(Master branch has done some of the
relevant changes), But @rxin has a different idea for 2.
We should reach a consensus on the above two underlying changes, and then
do the next step.
@srowen What do you think of the above two changes?
@opme Have you done more test on large scale data shuflle?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
