Github user CodingCat commented on the pull request:
https://github.com/apache/spark/pull/2851#issuecomment-74582634
Hi, @squito, thank you very much for your patient review
Here is the summary on what I have done on addressing your comments
1. regarding the duplication in RDDPage and BroadcastPage
Now, both of them inherit from InMemoryObjectPage, which defines the
common parts of these two classes, e.g. Worker Header, etc.
2. Serialization and Deserialization of SparkListenerBlockUpdate event
I implemented the serialization and deserialization methods
(https://github.com/apache/spark/pull/2851/files#diff-4f6ba18259eb4c31ac930e18f1ba6f88R99)
and
(https://github.com/apache/spark/pull/2851/files#diff-4f6ba18259eb4c31ac930e18f1ba6f88R480)
To prevent the large volume of events being logged, there is an event
filter existing in SparkListenerBus
(https://github.com/apache/spark/pull/2851/files#diff-01185688f339238fc6689baa6106df63R27).
Now this filter ensure that only the broadcast block update event is logged
(https://github.com/apache/spark/pull/2851/files#diff-fbe8f967070627c8dc155237e77c7314R130)
(actually this was implemented long time ago)
3. get rid of the mutable variable in BlockManagerMasterActor
Done
(https://github.com/apache/spark/pull/2851/files#diff-6da9360efd5e37dcd0edcc651db1e9cbR508)
4. shorten the compare() in BroadcastInfo
(https://github.com/apache/spark/pull/2851/files#diff-6da9360efd5e37dcd0edcc651db1e9cbR508)
Unfortunately, I probably cannot do that, since compare() returns Int,
but broadcastId is Long
5. tighten the methods in StorageUtils
Done
(https://github.com/apache/spark/pull/2851/files#diff-6da9360efd5e37dcd0edcc651db1e9cbR508)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
