milenkovicm commented on code in PR #1875:
URL:
https://github.com/apache/datafusion-ballista/pull/1875#discussion_r3428078400
##########
ballista/core/src/serde/scheduler/from_proto.rs:
##########
@@ -108,14 +108,19 @@ impl TryInto<PartitionLocation> for
protobuf::PartitionLocation {
)
})?
.into(),
- executor_meta: self
- .executor_meta
- .ok_or_else(|| {
+ partition_location_metadata: {
+ let m = self.executor_meta.ok_or_else(|| {
BallistaError::General(
"executor_meta in PartitionLocation is
missing".to_owned(),
)
- })?
- .into(),
+ })?;
+ Arc::new(PartitionLocationMetadata {
Review Comment:
my idea was to have single instance of partition location and then Arc-ed
behind shared pointers. At the moment, we have instance of executor metadata
per partition location which for scenarios where we have just a few executor
and many partitions may produce a lot of data (id, host ...). also when
serialised we would serialize executor metadata for each of partition, which is
hitting gprc issues. not sure if we can somehow reduce executor metadata
instance count and its serialization
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
