Re: Use an explicit property to define type of Whisk entity

[Chetan Mehrotra]

Thanks Rodric for the background details. Continuing the discussion from Slack

> You'll have to augment the serialize entity type to add this property (but
> I would not add it to the actual types since it's redundant).

Ack. So would add the types implicitly in serialization logic

> The same applies for the rootns: it's a space/time tradeoff why it doesn't
> exist: it's cheap to recompute in the view vs the space the redundant
> information would occup.

Ack. For declarative indexes we would probably need to store rootns.

> I've previously explored consolidating the views as you suggest. One reason
> not to do that is that the different entity types change at different rates
> - by having different views for each type, it is potentially cheaper to
> recompute some views.

Ack and makes sense. Probably with versioned views we can make an
attempt and see if using single view provide any tangible benefit.

Would now work on PR to add the type support. For now focus would be
to just add type property in write flow only. So read flow would work
as is via current views for Couch db. Later would look into making use
of this on read side.

Chetan Mehrotra


On Fri, Dec 8, 2017 at 9:49 AM, Rodric Rabbah  wrote:
> Adding an entity type can work - it would simplify the views. It's mostly a
> historical footnote why it doesn't exist and why it's inferred.
> You'll have to augment the serialize entity type to add this property (but
> I would not add it to the actual types since it's redundant).
> This could benefit the unmarshalling that happens when reading entities
> from the data store. Brendan McAdams might also have some thoughts on this.
> The same applies for the rootns: it's a space/time tradeoff why it doesn't
> exist: it's cheap to recompute in the view vs the space the redundant
> information would occup.
>
> I've previously explored consolidating the views as you suggest. One reason
> not to do that is that the different entity types change at different rates
> - by having different views for each type, it is potentially cheaper to
> recompute some views.
>
> Changing views requires recomputing the indexes in live dbs before roll out
> - to this effect, we've recently added versioned views which allows
> deploying new views before they become active. So it's important to add a
> feature toggle in the code that's specific to the view versions.
>
> -r
>
>
>
>
> On Mon, Dec 4, 2017 at 6:11 AM, Chetan Mehrotra 
> wrote:
>
>> Hi everyone,
>>
>> Currently all whisk entities are stored in "whisks" collection in
>> couchdb. Most queries around these entities are of form [root
>> namespace, date] invoked against entity specific views.
>>
>> GET /local_whisks/_design/whisks.v2/_view/packages?descending=
>> true=["guest",0]=false=["guest","￰","￰"]=30
>> 200
>>
>> The views are computed by "inferring" the entity type [1] based on
>> document properties.This mode of querying works fine with couchdb
>> which provides "computed" views. However it poses problem with storage
>> system where query relies on declarative indexes.
>>
>> Proposal
>> 
>> Introduce 2 properties
>>
>> 1. 'entityType' - whose value is set to action/package/rile/trigger
>> etc at time of creation.
>> 2. 'rootns' - Root namespace
>>
>> These properties would be set at time of creation itself in
>> Controller. This would allow creating a declarative index for the
>> entities. For e.g. in Mongo terms we can have a single index
>>
>> db.whisks.createIndex({entityType:1, rootns:1, updated:1}
>>
>> This may also possibly allow us to have a single view in couchdb whose
>> structure would be
>>
>> [entityType, rootns, date] = {  } //docs properties would be
>> superset of properties included in views for
>> action/package/rile/trigger
>>
>> Thoughts?
>>
>> Chetan Mehrotra
>>
>> [1]
>> var isPackage = function (doc) {
>> return (doc.binding !== undefined)
>>  };
>> var isAction = function (doc) {
>>  return (doc.exec !== undefined)
>>  };
>>  var isTrigger = function (doc) {
>> return (doc.exec === undefined && doc.binding === undefined &&
>> doc.parameters !== undefined)
>>  };
>>  var isRule = function (doc) {
>> return (doc.trigger !== undefined)
>>  };
>>

Re: Use an explicit property to define type of Whisk entity

[Rodric Rabbah]

Adding an entity type can work - it would simplify the views. It's mostly a
historical footnote why it doesn't exist and why it's inferred.
You'll have to augment the serialize entity type to add this property (but
I would not add it to the actual types since it's redundant).
This could benefit the unmarshalling that happens when reading entities
from the data store. Brendan McAdams might also have some thoughts on this.
The same applies for the rootns: it's a space/time tradeoff why it doesn't
exist: it's cheap to recompute in the view vs the space the redundant
information would occup.

I've previously explored consolidating the views as you suggest. One reason
not to do that is that the different entity types change at different rates
- by having different views for each type, it is potentially cheaper to
recompute some views.

Changing views requires recomputing the indexes in live dbs before roll out
- to this effect, we've recently added versioned views which allows
deploying new views before they become active. So it's important to add a
feature toggle in the code that's specific to the view versions.

-r




On Mon, Dec 4, 2017 at 6:11 AM, Chetan Mehrotra 
wrote:

> Hi everyone,
>
> Currently all whisk entities are stored in "whisks" collection in
> couchdb. Most queries around these entities are of form [root
> namespace, date] invoked against entity specific views.
>
> GET /local_whisks/_design/whisks.v2/_view/packages?descending=
> true=["guest",0]=false=["guest","￰","￰"]=30
> 200
>
> The views are computed by "inferring" the entity type [1] based on
> document properties.This mode of querying works fine with couchdb
> which provides "computed" views. However it poses problem with storage
> system where query relies on declarative indexes.
>
> Proposal
> 
> Introduce 2 properties
>
> 1. 'entityType' - whose value is set to action/package/rile/trigger
> etc at time of creation.
> 2. 'rootns' - Root namespace
>
> These properties would be set at time of creation itself in
> Controller. This would allow creating a declarative index for the
> entities. For e.g. in Mongo terms we can have a single index
>
> db.whisks.createIndex({entityType:1, rootns:1, updated:1}
>
> This may also possibly allow us to have a single view in couchdb whose
> structure would be
>
> [entityType, rootns, date] = {  } //docs properties would be
> superset of properties included in views for
> action/package/rile/trigger
>
> Thoughts?
>
> Chetan Mehrotra
>
> [1]
> var isPackage = function (doc) {
> return (doc.binding !== undefined)
>  };
> var isAction = function (doc) {
>  return (doc.exec !== undefined)
>  };
>  var isTrigger = function (doc) {
> return (doc.exec === undefined && doc.binding === undefined &&
> doc.parameters !== undefined)
>  };
>  var isRule = function (doc) {
> return (doc.trigger !== undefined)
>  };
>

Re: Use an explicit property to define type of Whisk entity

[Chetan Mehrotra]

Would be helpful if some feedback can be provided here.

If this looks ok I can work on PR around this topic
Chetan Mehrotra


On Mon, Dec 4, 2017 at 4:41 PM, Chetan Mehrotra
 wrote:
> Hi everyone,
>
> Currently all whisk entities are stored in "whisks" collection in
> couchdb. Most queries around these entities are of form [root
> namespace, date] invoked against entity specific views.
>
> GET 
> /local_whisks/_design/whisks.v2/_view/packages?descending=true=["guest",0]=false=["guest","￰","￰"]=30
> 200
>
> The views are computed by "inferring" the entity type [1] based on
> document properties.This mode of querying works fine with couchdb
> which provides "computed" views. However it poses problem with storage
> system where query relies on declarative indexes.
>
> Proposal
> 
> Introduce 2 properties
>
> 1. 'entityType' - whose value is set to action/package/rile/trigger
> etc at time of creation.
> 2. 'rootns' - Root namespace
>
> These properties would be set at time of creation itself in
> Controller. This would allow creating a declarative index for the
> entities. For e.g. in Mongo terms we can have a single index
>
> db.whisks.createIndex({entityType:1, rootns:1, updated:1}
>
> This may also possibly allow us to have a single view in couchdb whose
> structure would be
>
> [entityType, rootns, date] = {  } //docs properties would be
> superset of properties included in views for
> action/package/rile/trigger
>
> Thoughts?
>
> Chetan Mehrotra
>
> [1]
> var isPackage = function (doc) {
> return (doc.binding !== undefined)
>  };
> var isAction = function (doc) {
>  return (doc.exec !== undefined)
>  };
>  var isTrigger = function (doc) {
> return (doc.exec === undefined && doc.binding === undefined &&
> doc.parameters !== undefined)
>  };
>  var isRule = function (doc) {
> return (doc.trigger !== undefined)
>  };