[
https://issues.apache.org/jira/browse/RYA-359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Dasch updated RYA-359:
---------------------------
Description:
In theory, this can be achieved in two different ways:
1) Update code to support concurrent PeriodicNotificationApplications in
separate YARN containers that split the workload for a single rya instance.
2) Update code to support the various Runnables executing in separate YARN
containers. This would likely mean replacing the blocking queue being used
with serialization and kafka, or possibly adopting the kafka streams library.
The second option has the benefit that it allows for finer grained control of
concurrency, though it does introduce an added serialization overhead.
Other goal for this effort should be to ensure graceful degradation of
performance in the event of a container failure.
was:
In theory, this can be achieved in two different ways:
1) Update code to support concurrent PeriodicNotificationApplications in
separate YARN containers that split the workload for a single rya instance.
2) Update code to support the various Runnables executing in separate YARN
containers. This would likely mean replacing the blocking queue being used
with serialization and kafka, or possibly adopting the kafka streams library.
The second option has the benefit that it allows for finer grained control of
concurrency, though it does introduce an added serialization overhead.
> Update PeriodicNotificationApplication to scale beyond one YARN container.
> --------------------------------------------------------------------------
>
> Key: RYA-359
> URL: https://issues.apache.org/jira/browse/RYA-359
> Project: Rya
> Issue Type: Improvement
> Reporter: Jeff Dasch
> Priority: Minor
>
> In theory, this can be achieved in two different ways:
> 1) Update code to support concurrent PeriodicNotificationApplications in
> separate YARN containers that split the workload for a single rya instance.
> 2) Update code to support the various Runnables executing in separate YARN
> containers. This would likely mean replacing the blocking queue being used
> with serialization and kafka, or possibly adopting the kafka streams library.
> The second option has the benefit that it allows for finer grained control of
> concurrency, though it does introduce an added serialization overhead.
> Other goal for this effort should be to ensure graceful degradation of
> performance in the event of a container failure.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
