[
https://issues.apache.org/jira/browse/AURORA-1802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677066#comment-15677066
]
Stephan Erb commented on AURORA-1802:
-------------------------------------
TLDR: I tried some things to make the creation of {{AttributeAggregate}} faster
and none worked. I am now out of ideas.
Profiling indicated that we are slowed down by the many {{getHostAttribute}}
calls. I therefore hoped to speed up the creation by batching queries together
so that we fetch multiple host attributes per DB roundtrip.
First iteration with the idea to select all required host attributes in one go:
{code}
static AttributeAggregate create(
final Supplier<Iterable<IScheduledTask>> taskSupplier,
final AttributeStore attributeStore) {
final Function<Iterable<IScheduledTask>, Iterable<IAttribute>>
tasksToAttributes =
tasks -> {
FluentIterable<String> hosts = FluentIterable
.from(tasks)
.transform(Tasks::scheduledToSlaveHost);
ImmutableMap<String, IHostAttributes> hostAttributes =
Maps.uniqueIndex(
attributeStore.getHostAttributes(hosts.toSet()),
a -> a.getHost());
return hosts.transformAndConcat(host ->
hostAttributes.get(host).getAttributes());
};
return create(Suppliers.compose(tasksToAttributes, taskSupplier));
}
{code}
This was slow due to the enormous generated SQL query listing all hosts with
relevant tasks on it. As an alternative, I tried the following generalised
version with configurable batch sizes:
{code}
static AttributeAggregate create(
final Supplier<Iterable<IScheduledTask>> taskSupplier,
final AttributeStore attributeStore) {
final Function<Iterable<IScheduledTask>, Iterable<IAttribute>>
tasksToAttributes =
tasks -> {
FluentIterable<String> hosts = FluentIterable
.from(tasks)
.transform(Tasks::scheduledToSlaveHost);
ImmutableMap<String, IHostAttributes> hostAttributes = FluentIterable
.from(Iterables.partition(hosts.toSet(), 25))
.transformAndConcat(partition ->
attributeStore.getHostAttributes(partition))
.uniqueIndex(a -> a.getHost());
return hosts.transformAndConcat(host ->
hostAttributes.get(host).getAttributes());
};
return create(Suppliers.compose(tasksToAttributes, taskSupplier));
}
{code}
Interestingly enough, the larger the batch the slower the entire computation. I
interpret this as mybatis being bad at handling result sets larger than one
item. Could that be?
Does anyone have another idea what could be wrong in the setup?
> AttributeAggregate slows down scheduling of jobs with many instances
> --------------------------------------------------------------------
>
> Key: AURORA-1802
> URL: https://issues.apache.org/jira/browse/AURORA-1802
> Project: Aurora
> Issue Type: Bug
> Components: Scheduler
> Reporter: Stephan Erb
> Fix For: 0.17.0
>
>
> The current implementation of
> [{{AttributeAggregate}}|https://github.com/apache/aurora/blob/f559e930659e25b3d7cacb7b845ebda50d18d66a/src/main/java/org/apache/aurora/scheduler/filter/AttributeAggregate.java]
> slows down scheduling of jobs with many instances. Interestingly, this is
> currently not visible in our job scheduling benchmark results as it only
> affects the benchmark setup time but not the measured part.
> {{AttributeAggregate}} relies on {{Suppliers.memoize}} to ensure that it is
> only computed once and only when necessary. This has probably been done
> because the factory
> [{{AttributeAggregate.getJobActiveState}}|https://github.com/apache/aurora/blob/f559e930659e25b3d7cacb7b845ebda50d18d66a/src/main/java/org/apache/aurora/scheduler/filter/AttributeAggregate.java#L56-L91]
> is slow.
> After some recent changes to schedule multiple task instances per scheduling
> round the aggregate is computed in each scheduling round via the call
> [{{resourceRequest.getJobState().updateAttributeAggregate(...)}}
> |https://github.com/apache/aurora/blob/f559e930659e25b3d7cacb7b845ebda50d18d66a/src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java#L173]
> in {{TaskAssigner}}. This means the expensive factory is called once per
> scheduling round.
> h3. Potential improvements
> * the current factory implementation performs one {{fetchTasks}} query
> followed by {{n}} distinct {{getHostAttributes}} queries. This could be
> reduced to a single SQL query.
> * the aggregate makes heavy use of {{ImmutableMultiset}} even though it is
> not immutable any more. There is potential room for improvement here.
> * The aggregate uses suppliers to perform a lazy instantiation even though
> its current usage is not lazy any more. We can either make the implementation
> eager, or ensure that the expensive part is only run when absolutely
> necessary.
> h3. Proof of concept
> * 4 mins 23.407 secs -- total runtime of {{./gradlew jmh
> -Pbenchmarks='SchedulingBenchmarks.InsufficientResourcesSchedulingBenchmark'}}
> * 2 mins 40.308 secs -- total runtime of {{./gradlew jmh
> -Pbenchmarks='SchedulingBenchmarks.InsufficientResourcesSchedulingBenchmark'}}
> with [{{resourceRequest.getJobState().updateAttributeAggregate(...)}}
> |https://github.com/apache/aurora/blob/f559e930659e25b3d7cacb7b845ebda50d18d66a/src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java#L173]
> commented out. This works as the call is not necessary when only a single
> instance is scheduled per scheduling round, as done in the benchmarks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
