To All,
I think I made a mistake in my previous email
config:
topology.workers: 2
topology.component.rammap:
- "some-id": 1234
- "other-id": 6789
I think the yaml above is incorrect as well as other examples. I think we
would have to do something like below
config:
topology.workers: 2
topology.component.rammap:
- "some-id:1234"
- "other-id:6789"
Which would then product a list of strings that would match the way the
topology_component_rammap is set via other apis. The problem with this
approach is it would be easy for someone to make a mistake within the
formatting of the strings and would then cause us to have to validate the
format to fit the specs. I think the approach below would be better. I
would then just take the input, do some validation and conversion via the
ByteAmount class and generate a properly formatted string to fit the specs
of the topology_component_rammap values.
config:
topology.workers: 2
topology.component.rammap:
- id: "component-1"
size: 1234
type: MB // Megabytes
- id: "component-2"
size: 6789
type: GB // GigaBytes
- id: "component-3"
size: 123456789
type: B // Bytes
Hope I was clear with trying to explain things. Of course I will also be
creating the docs as well to explain usage.
-Josh
On Sun, Jan 21, 2018 at 8:21 AM, Josh Fischer <[email protected]> wrote:
> All,
>
> While working with Karthik, we have discovered that the way ECO handles
> topology configuration will not work for all configuration types. To be
> specific, setting individual component's ram will not work. We will also
> have to keep in mind container size that contains the components. My
> proposal is this:
>
> Create a standardized way to allow for the configuring of component ram
> size in the "config" section of the eco yaml file. This would be a list
> of key value pairs that mapped the "id" of a component to an allocated ram
> size in MB. An example is below:
>
> config:
> topology.workers: 2
> topology.component.rammap:
> - "some-id": 1234
> - "other-id": 6789
>
> However the above implementation may be unclear when it comes to
> understanding what unit of measurement is implicitly specified and/or
> expected. Or we could do something like below.
>
> config:
> topology.workers: 2
> topology.component.rammap:
> - spec:
> id: "component-1"
> size: 1234
> type: MB // Megabytes
> - spec:
> id: "component-2"
> size: 6789
> type: GB // GigaBytes
> - spec:
> id: "component-3"
> size: 123456789
> type: B // Bytes
>
>
> If a mapping is not specified for a component, we can just assume Heron's
> defaults. We could then dynamically calculate the container size based off
> of the number components and their corresponding allocated resources for
> simplicity of use for the user, but still allow them to specify a custom
> set of resources to a container like below
>
> topology.container.disk: 1234
> topology.container.ram: 3456
> topology.container.cpu: 2
>
>
> It may be best if I reused the ByteAmount object to calculate resource
> size to remain consistent with the other Heron APIs. Any concerns or
> improvements to this approach I am missing?
>
> Please Advise,
>
> Josh
>
