All,
While working with Karthik, we have discovered that the way ECO handles
topology configuration will not work for all configuration types. To be
specific, setting individual component's ram will not work. We will also
have to keep in mind container size that contains the components. My
proposal is this:
Create a standardized way to allow for the configuring of component ram
size in the "config" section of the eco yaml file. This would be a list
of key value pairs that mapped the "id" of a component to an allocated ram
size in MB. An example is below:
config:
topology.workers: 2
topology.component.rammap:
- "some-id": 1234
- "other-id": 6789
However the above implementation may be unclear when it comes to
understanding what unit of measurement is implicitly specified and/or
expected. Or we could do something like below.
config:
topology.workers: 2
topology.component.rammap:
- spec:
id: "component-1"
size: 1234
type: MB // Megabytes
- spec:
id: "component-2"
size: 6789
type: GB // GigaBytes
- spec:
id: "component-3"
size: 123456789
type: B // Bytes
If a mapping is not specified for a component, we can just assume Heron's
defaults. We could then dynamically calculate the container size based off
of the number components and their corresponding allocated resources for
simplicity of use for the user, but still allow them to specify a custom
set of resources to a container like below
topology.container.disk: 1234
topology.container.ram: 3456
topology.container.cpu: 2
It may be best if I reused the ByteAmount object to calculate resource size
to remain consistent with the other Heron APIs. Any concerns or
improvements to this approach I am missing?
Please Advise,
Josh
