Thanks Pallavi for bring this up. I have been wanting namespace isolation for
properties for a fair while and this allows to build consensus and move forward.
>From what I can gather from the discussions on 1573, multiple issues are
>hinted at and proposed to be solved.
1. Allowing for properties being overidden at schedule time - directly being
passed to underlying scheduler, essentially a pass through for falcon
2. Allowing for default properties in falcon configured in entity definition to
be overriden at schedule time
3. Namespacing of properties such that there is no accidental clash in property
name and there is general improvement in maintainability.
To pass properties that override falcon entity “properties” and have them
passed through as is to the underlying scheduler would be a good start.
My personal preference would be for the following overlay scheme, where user
overrides gets preference and the resolution order is as follows.
——> FALCON SYSTEM <— —
(USER PROPERTIES)
> (PROCESS PROPERTIES)
> (FEED PROPERTIES)
> (CLUSTER PROPERTIES)
——> SCHEDULER SYSTEM (OOZIE) <— —
@Schedule / Instance Management
(RESOLVED PROPERTIES FROM ABOVE)
> (SCHEDULER DEFAULT PROPERTIES)
The properties in the entity definition today aren’t totally opaque (for ex.
queue-name, priority etc). We ought to segregate properties into various
classes such as System level Internal properties, Entity level internal
properties, Entity level properties, Entity level user properties.
System level internal properties are used by falcon and is application to the
entire instance and as such isn’t available for override. However these
properties can be re-read by the server if it is deemed to be a runtime
property. Tucking them under a namespace is desirable (for. ex falcon.system).
Properties with this namespace if sent for any entity or instance operation
should be ignored and appropriately warned.
Entity level internal property is what is used by falcon internally to exchange
information between different systems and these shouldn’t overridable either.
Namespace for these could be say falcon.internal. Properties with this
namespace if sent for any entity or instance operation should be ignored and
appropriately warned.
Entity level properties can be overriden by users. Seems very important to put
this in an exclusive namespace (for ex. falcon.entity.override). This is well
defined list of properties and by definition is not a first class property
element in the property bag of entity. For ex. priority, process-parallelism,
scheduler etc
Entity level user properties are the ones typically enlisted in the property
bag of the entity definition. Anything supplied by the user overrides or is
additionally added to the entity or instance operation. This I would prefer to
keep it without any namespace, as these property names make it ways into
workflow definition and possibly into user code and it would be preferable not
to carry the prefix all the way and muddy user code.
Optionally if we particularly feel the need to pass something to the scheduler
and scheduler only, we can set aside a namespace for it.
Regards
Srikanth Sundarrajan
> Date: Mon, 16 Nov 2015 10:02:57 +0530
> Subject: [DISCUSS] Namespace-ing properties and FALCON-1573
> From: [email protected]
> To: [email protected]
> CC: [email protected]
>
> Folks,
> With FALCON-1573, Daniel came up with a very good suggestion of allowing
> user-supplied properties at schedule time. He proposed 2 approaches
> <https://issues.apache.org/jira/browse/FALCON-1573?focusedCommentId=14982386&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14982386>
> -
>
> 1. Allow user to specify free form name-value pairs. Since user-supplied
> property names can clash with Falcon property names, give Falcon properties
> (generated when the bundle is built) priority over user-supplied ones, in
> case of clash.
> 2. Namespace user-supplied properties, so we can distinguish user and
> system properties. This will allow user to supply properties without any
> clash with system properties.
>
> Daniel has already uploaded patch for approach (1). Regarding this, wish to
> solicit your input on 2 questions:
>
> 1. Should we pursue approach (2) in FALCON-1573 itself or take it up
> separately?
> 2. When we eventually namespace the properties, what should the
> namespace format be?
>
> My 2 cents:
>
> Question (1) : I think we should pursue namespace approach in FALCON-1573
> itself. Two reasons for the same:
>
> - When user supplies a name-value pair and we silently ignore (we only
> log) it when it clashes with system property, it is not very intuitive to
> the user.
> - If we don't introduce it now, but, bring it in later, we'll have to
> deal with 2 kinds of properties, one that is namespace-d and one that
> isn't.
>
> Question (2) : I suggest we use falcon.system.* (or just falcon.*) for
> system properties and falcon.user.* for user-supplied properties.
>
> Apologies for the tad lengthy mail. Request you all to provide your inputs.
>
> Thanks and Regards,
> Pallavi
>
> --
> _____________________________________________________________
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
