Hi Jen,
We're thinking about creating a convention to define gobblin properties. Given
we have a large number of and distributed properties, centralizing
configuration management doesn't scale. I have an idea, of keeping tracking of
configs ownership and hierarchy, that:
* Each top level configurable construct/class defines configuration:
`<camel-cased class name>.<config name>`, for example, a `HttpSource` may have
a config named `url`, whose full config name will be `httpSource.url`
* For sub level or nested construct, the convention will be
`<camel-cased class name>.<sub scope>.<config name>`, for example, a
`HttpSource` may uses a `HttpExtractor`, which has a config named `dataSize`,
whose full config name will be `httpSource.extractor.dataSize`
For rolling out this idea
* Newly defined construct should accept `Config` as a constructor parameter
instead of any kind of `State`
* Leave old construct untouched, but its new configurations will be defined
following this convention.
Zhixiong
________________________________
From: Jay Sen <[email protected]>
Sent: Saturday, June 22, 2019 9:17 PM
To: [email protected]
Subject: Gobblin properties
Hi Guys,
I noticed that the property usage and definition is bit confusing,
Ex; property to define path where all job configurations (.pull files) are
placed.
following are all ways for the same:
"job.config.path"
"jobconf.dir"
"jobconf.fullyQualifiedPath"
"gobblin.cluster.jobconf.fullyQualifiedPath"
"gobblin.cluster.job.conf.path"
Also question about the properties with "gobblin.cluster" prefix and
without it, does it get automatically prefixed somewhere base don the mode
?
and should we standardize these properties, like have getJobPath() in
configUtil to centralize the decision making ?
comments / thoughts ?
Thanks
Jay
